Test Definition Format ====================== Any binary that can run on Ubuntu, can be considered a Test Case as far as UTAH is concerned. If the right control files are added and the test suite is structured in directories neatly, UTAH will run it. Directory Structure ------------------- This is what a UTAH test suite looks like:: testsuite1/ tslist.run (or tslist.auto) ts_control testcase1/ tc_control Makefile test.c testcase2/ tc_control test.py The utah-client package includes a utility called phoenix_ that helps create a skeleton layout. .. _phoenix: phoenix.html master.run ---------- A runlist is a collection of test cases that we want to run in one go. The syntax for runlists is:: - testsuites: - name: testsuite1 fetch_method: git fetch_location: repo - name: testsuite2 fetch_method: bzr fetch_location: lp:utah/dev/ include_tests: # optionally include specific tests - t1 - t2 - t3... exclude_tests: # optionally exclude specific tests - st4 - st5 The only required fields are ``name``, ``fetch_method`` and ``fetch_location``. ``name`` must correspond to the name of the top-level testsuite directory. ``fetch_method`` should be one of ``bzr``, ``bzr-export``, or ``git``. ``fetch_location`` should be a valid location for the supplied ``fetch_method``. If ``bzr`` is selected as the fetch method then ``fetch_location`` should point to a repository that is a valid testsuite, i.e. has a tslist.run, ts_control (if needed), and test case directories. While ``bzr`` does a ``bzr branch`` in the implementation, ``bzr-export`` does a ``bzr export`` and accepts a bzr location that can point to a sub-directory within a repository that is a valid testsuite. If ``dev`` is selected as the fetch method then ``fetch_location`` should point to a valid testsuite directory. The utah client will run ``cp -r ``. This method is provided to allow testsuite/testcase authors to run the client on a development tree without needing to push changes each time to a repository. One caveat to note is that a ``fetch_method`` of ``bzr`` will get the revision information from the local copy of the branched repository but ``bzr-export`` will have a small race condition between the ``get`` and ``revision`` calls since the actual bzr repository must be queried for the revision. ``include_tests`` will limit the testcases that are run to only those listed by this option. ``exclude_tests`` will run all the tests in the testsuite's tslist.run file except those listed by this option. Test suites can be divided in categories (for our internal use, or for test cases submitted to our test case base) or they can be in a repository anywhere. **repeating a runlist:** Sometimes a runlist may need to be executed more than once. For example, in the case you want to determine statistical accuracy of results. A runlist supports an optional field:: repeat_count: This value defaults to 0 which means execute the runlist once. If non-zero the runlist will be repeated that many times. eg, a repeat_count of 2 means the runlist will be executed 3 times. Each test suite has a default runlist and a control file that contain as follows: tslist.run ---------- :: - test: t1 # directory overrides: - run_as: nobody # user that runs the test - test: t2 # directory/binary overrides: # array of control properties to override - timeout: 200 # timeout in seconds ``test`` is the only required field in this file and must match a directory name in the testsuite tree. ``run_as`` utilizes sudo to run the testcase as the given user. NOTE: this requires that the user running the utah client either has cached credentials or is allowed to run commands without a password. The ``overrides`` array accepts any option from the tc_control file. Overrides should be used sparingly since it makes more sense to adjust the options in the tc_control file rather than here. .. _tslist-auto: tslist.auto ----------- tslist.auto is an alternative to the static nature of tslist.run. When trying to add an existing test suite to UTAH, this option may work better.:: # the contents of this file help dynamically construct the names of each # existing test case and how to run it. # utah execute's the discovery command, and then pass each line of the output # as a cli parameter(s) to the test_cmd - # this will result in testcases name "1", "2" and "3". The output will be # will just be "1", "2", and "3" discovery_cmd: seq 3 # the '{}' in test-cmd will be replaced with the output of the discovery # command. test_cmd: echo {} - # a more complex example that breaks up unity test for autopilot discovery_cmd: "autopilot list unity | grep unity.tests | sed -e 's/^[[:space:]]*//' | cut -d. -f1,3 | sort | uniq" test_cmd: autopilot run -v {} **NOTE:** The items returned by the discovery_cmd become the name of each testcase. So its wise to choose something sensible/consistent. ts_control ---------- :: build_cmd: make timeout: 300 ts_setup: tsetup/setup.sh # tsetup/setup.sh is the set up for # the whole test suite ts_cleanup: tcleanup/cleanup.sh # clean up test suite There are no required fields in this file. To avoid excessive typing and repetition in runlists and control files the following rules apply: * ``tslist.run``'s **test** is relative to the test suite. So if the suite is called *sample_tests* and the entry in ``tslist.run`` for test is *test_one* then the path to the test folder will be *sample_tests/test_one*. * ``tslist.run``'s **overrides** are the same options available in the ``tc_control`` file and take precedence over those found in ``tc_control``. * ``ts_control`` is optional. * if ``build_cmd``, or ``ts_setup`` fails no tests in the suite will be run. * if ``build_cmd``, or ``tc_setup`` fails the test will not be run. * ``ts_cleanup`` will be run whether or not ``ts_setup``, ``build_cmd``, or any tests fail. * ``tc_cleanup`` will be run whether or not ``tc_setup``, ``build_cmd``, or the test fails. Each test case, with all the code required for it to compile or run, lives in a directory, and contains also a control file that **includes its documentation** amongst other things: tc_control ---------- :: build_cmd: make # or scons or build.sh command: python test1.py a1 a2 a3 # command to run the test from within t1 description: dependencies: coreutils actions: | 1. Action 1 2. Action 2 expected_results: | 1. Expected result 1 2. Expected result 2 type: userland/kernel # currently unimplemented timeout: 100 tc_setup: t1/t1 setup # in case the test case has a setup tc_cleanup: t1/t1 cleanup # in case the test has a cleanup In this file ``command``, ``description``, ``dependencies``, ``actions``, ``expected_results``, and ``timeout`` are required. ``command`` is the command to actually run the test. ``description`` is a textual description of the test. ``dependencies`` is a list of items the test depends on. ``action`` is a list of actions the test executes and ``expected_results`` is a list of what those actions should result in. ``timeout`` is the ammount of time the test should complete within and is used to avoid tests that might loop indefinitely. ``type`` currently defaults to ``userland`` and may in the future have other options as the need arises. ``build_cmd`` can be used to build any binary test cases. ``tc_setup`` can be used to setup any needed data or files for the test and ``tc_cleanup`` can be used to remove the data or files added by ``tc_setup``. ``build_cmd``, ``tc_setup``, and ``tc_cleanup`` are all simple shell commands similar to ``command``. For each test case that is a binary or a script, there will be an option to define a setup and a cleanup function if they wish so, and it is the test code developer’s responsibility to add the right parameters to their code so that the harness can run their setup and clean up functions. **In terms of settings precedence:** The test case control file (``tc_control``) provides the default values for the options available for the testcase. The **overrides** in ``tslist.run`` take precedence over those options in the ``tc_control`` file. Tests That Need Reboots ~~~~~~~~~~~~~~~~~~~~~~~ In a situation where a reboot is required. This is handled in UTAH by marking a testcase (in the ``tc_control`` file) with:: reboot: This will make the **utah client** reboot the system under test at the **end** of the testcase after any ``tc_cleanup`` command has finished. Any thing that needs to be tested after the reboot should be in the testcase immediately following the testcase marked for reboot. The possible choices are ``always``, ``pass``, and ``never``. ``never`` is the default and simply means the system under test will not reboot. ``pass`` means the reboot will only happen if the testcase passes. ``always``, means just that, the reboot will happen even if the test fails. Some things to keep in mind. The **utah-client** resumes by replacing ``/etc/rc.local`` with a file that will resume the current run and will resume with the testcase following the one that triggered the reboot. Also note that in order to have both pre- and post-reboot logging the ``-o`` flag should be used so that post-reboot output will be appended to the pre-reboot output.