| tdc - Linux Traffic Control (tc) unit testing suite | |
| Author: Lucas Bates - lucasb@mojatatu.com | |
| tdc is a Python script to load tc unit tests from a separate JSON file and | |
| execute them inside a network namespace dedicated to the task. | |
| REQUIREMENTS | |
| ------------ | |
| * Minimum Python version of 3.4. Earlier 3.X versions may work but are not | |
| guaranteed. | |
| * The kernel must have network namespace support | |
| * The kernel must have veth support available, as a veth pair is created | |
| prior to running the tests. | |
| * All tc-related features being tested must be built in or available as | |
| modules. To check what is required in current setup run: | |
| ./tdc.py -c | |
| Note: | |
| In the current release, tdc run will abort due to a failure in setup or | |
| teardown commands - which includes not being able to run a test simply | |
| because the kernel did not support a specific feature. (This will be | |
| handled in a future version - the current workaround is to run the tests | |
| on specific test categories that your kernel supports) | |
| BEFORE YOU RUN | |
| -------------- | |
| The path to the tc executable that will be most commonly tested can be defined | |
| in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and | |
| define the path. | |
| If you need to test a different tc executable on the fly, you can do so by | |
| using the -p option when running tdc: | |
| ./tdc.py -p /path/to/tc | |
| RUNNING TDC | |
| ----------- | |
| To use tdc, root privileges are required. This is because the | |
| commands being tested must be run as root. The code that enforces | |
| execution by root uid has been moved into a plugin (see PLUGIN | |
| ARCHITECTURE, below). | |
| If nsPlugin is linked, all tests are executed inside a network | |
| namespace to prevent conflicts within the host. | |
| Running tdc without any arguments will run all tests. Refer to the section | |
| on command line arguments for more information, or run: | |
| ./tdc.py -h | |
| tdc will list the test names as they are being run, and print a summary in | |
| TAP (Test Anything Protocol) format when they are done. If tests fail, | |
| output captured from the failing test will be printed immediately following | |
| the failed test in the TAP output. | |
| OVERVIEW OF TDC EXECUTION | |
| ------------------------- | |
| One run of tests is considered a "test suite" (this will be refined in the | |
| future). A test suite has one or more test cases in it. | |
| A test case has four stages: | |
| - setup | |
| - execute | |
| - verify | |
| - teardown | |
| The setup and teardown stages can run zero or more commands. The setup | |
| stage does some setup if the test needs it. The teardown stage undoes | |
| the setup and returns the system to a "neutral" state so any other test | |
| can be run next. These two stages require any commands run to return | |
| success, but do not otherwise verify the results. | |
| The execute and verify stages each run one command. The execute stage | |
| tests the return code against one or more acceptable values. The | |
| verify stage checks the return code for success, and also compares | |
| the stdout with a regular expression. | |
| Each of the commands in any stage will run in a shell instance. | |
| USER-DEFINED CONSTANTS | |
| ---------------------- | |
| The tdc_config.py file contains multiple values that can be altered to suit | |
| your needs. Any value in the NAMES dictionary can be altered without affecting | |
| the tests to be run. These values are used in the tc commands that will be | |
| executed as part of the test. More will be added as test cases require. | |
| Example: | |
| $TC qdisc add dev $DEV1 ingress | |
| The NAMES values are used to substitute into the commands in the test cases. | |
| COMMAND LINE ARGUMENTS | |
| ---------------------- | |
| Run tdc.py -h to see the full list of available arguments. | |
| usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]] | |
| [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v] | |
| [-d DEVICE] [-n NS] [-V] | |
| Linux TC unit tests | |
| optional arguments: | |
| -h, --help show this help message and exit | |
| -p PATH, --path PATH The full path to the tc executable to use | |
| -v, --verbose Show the commands that are being run | |
| -d DEVICE, --device DEVICE | |
| Execute the test case in flower category | |
| selection: | |
| select which test cases: files plus directories; filtered by categories | |
| plus testids | |
| -D DIR [DIR ...], --directory DIR [DIR ...] | |
| Collect tests from the specified directory(ies) | |
| (default [tc-tests]) | |
| -f FILE [FILE ...], --file FILE [FILE ...] | |
| Run tests from the specified file(s) | |
| -c [CATG [CATG ...]], --category [CATG [CATG ...]] | |
| Run tests only from the specified category/ies, or if | |
| no category/ies is/are specified, list known | |
| categories. | |
| -e ID [ID ...], --execute ID [ID ...] | |
| Execute the specified test cases with specified IDs | |
| action: | |
| select action to perform on selected test cases | |
| -l, --list List all test cases, or those only within the | |
| specified category | |
| -s, --show Display the selected test cases | |
| -i, --id Generate ID numbers for new test cases | |
| netns: | |
| options for nsPlugin(run commands in net namespace) | |
| -n NS, --namespace NS | |
| Run commands in namespace NS | |
| valgrind: | |
| options for valgrindPlugin (run command under test under Valgrind) | |
| -V, --valgrind Run commands under valgrind | |
| PLUGIN ARCHITECTURE | |
| ------------------- | |
| There is now a plugin architecture, and some of the functionality that | |
| was in the tdc.py script has been moved into the plugins. | |
| The plugins are in the directory plugin-lib. The are executed from | |
| directory plugins. Put symbolic links from plugins to plugin-lib, | |
| and name them according to the order you want them to run. | |
| Example: | |
| bjb@bee:~/work/tc-testing$ ls -l plugins | |
| total 4 | |
| lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py | |
| lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py | |
| -rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py | |
| The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and | |
| must be called "SubPlugin" so tdc can find them. They are | |
| distinguished from each other in the python program by their module | |
| name. | |
| This base class supplies "hooks" to run extra functions. These hooks are as follows: | |
| pre- and post-suite | |
| pre- and post-case | |
| pre- and post-execute stage | |
| adjust-command (runs in all stages and receives the stage name) | |
| The pre-suite hook receives the number of tests and an array of test ids. | |
| This allows you to dump out the list of skipped tests in the event of a | |
| failure during setup or teardown stage. | |
| The pre-case hook receives the ordinal number and test id of the current test. | |
| The adjust-command hook receives the stage id (see list below) and the | |
| full command to be executed. This allows for last-minute adjustment | |
| of the command. | |
| The stages are identified by the following strings: | |
| - pre (pre-suite) | |
| - setup | |
| - command | |
| - verify | |
| - teardown | |
| - post (post-suite) | |
| To write a plugin, you need to inherit from TdcPlugin in | |
| TdcPlugin.py. To use the plugin, you have to put the | |
| implementation file in plugin-lib, and add a symbolic link to it from | |
| plugins. It will be detected at run time and invoked at the | |
| appropriate times. There are a few examples in the plugin-lib | |
| directory: | |
| - rootPlugin.py: | |
| implements the enforcement of running as root | |
| - nsPlugin.py: | |
| sets up a network namespace and runs all commands in that namespace | |
| - valgrindPlugin.py | |
| runs each command in the execute stage under valgrind, | |
| and checks for leaks. | |
| This plugin will output an extra test for each test in the test file, | |
| one is the existing output as to whether the test passed or failed, | |
| and the other is a test whether the command leaked memory or not. | |
| (This one is a preliminary version, it may not work quite right yet, | |
| but the overall template is there and it should only need tweaks.) | |
| ACKNOWLEDGEMENTS | |
| ---------------- | |
| Thanks to: | |
| Jamal Hadi Salim, for providing valuable test cases | |
| Keara Leibovitz, who wrote the CLI test driver that I used as a base for the | |
| first version of the tc testing suite. This work was presented at | |
| Netdev 1.2 Tokyo in October 2016. | |
| Samir Hussain, for providing help while I dove into Python for the first time | |
| and being a second eye for this code. |