Test Definition Format¶

Directory Structure¶

This is what a UTAH test suite looks like:

testsuite1/
  tslist.run (or tslist.auto)
  ts_control
  testcase1/
    tc_control
    Makefile
    test.c
  testcase2/
    tc_control
    test.py

The utah-client package includes a utility called phoenix that helps create a skeleton layout.

master.run¶

A runlist is a collection of test cases that we want to run in one go. The syntax for runlists is:

- testsuites:
  - name: testsuite1
    fetch_method: git
    fetch_location: repo

  - name: testsuite2
    fetch_method: bzr
    fetch_location: lp:utah/dev/
    include_tests:  # optionally include specific tests
      - t1
      - t2
      - t3...
    exclude_tests: # optionally exclude specific tests
      - st4
      - st5

The only required fields are name, fetch_method and fetch_location. name must correspond to the name of the top-level testsuite directory. fetch_method should be one of bzr, bzr-export, or git. fetch_location should be a valid location for the supplied fetch_method. If bzr is selected as the fetch method then fetch_location should point to a repository that is a valid testsuite, i.e. has a tslist.run, ts_control (if needed), and test case directories. While bzr does a bzr branch in the implementation, bzr-export does a bzr export and accepts a bzr location that can point to a sub-directory within a repository that is a valid testsuite. If dev is selected as the fetch method then fetch_location should point to a valid testsuite directory. The utah client will run cp -r <fetch_location> <testsuite_name>. This method is provided to allow testsuite/testcase authors to run the client on a development tree without needing to push changes each time to a repository.

One caveat to note is that a fetch_method of bzr will get the revision information from the local copy of the branched repository but bzr-export will have a small race condition between the get and revision calls since the actual bzr repository must be queried for the revision.

include_tests will limit the testcases that are run to only those listed by this option. exclude_tests will run all the tests in the testsuite’s tslist.run file except those listed by this option.

Test suites can be divided in categories (for our internal use, or for test cases submitted to our test case base) or they can be in a repository anywhere.

repeating a runlist:

Sometimes a runlist may need to be executed more than once. For example, in the case you want to determine statistical accuracy of results. A runlist supports an optional field:

repeat_count: <integer>

This value defaults to 0 which means execute the runlist once. If non-zero the runlist will be repeated that many times. eg, a repeat_count of 2 means the runlist will be executed 3 times.

Each test suite has a default runlist and a control file that contain as follows:

tslist.run¶

- test: t1                             # directory
  overrides:
  - run_as: nobody                     # user that runs the test

- test: t2                             # directory/binary
  overrides:                           # array of control properties to override
  - timeout: 200                       # timeout in seconds

test is the only required field in this file and must match a directory name in the testsuite tree. run_as utilizes sudo to run the testcase as the given user. NOTE: this requires that the user running the utah client either has cached credentials or is allowed to run commands without a password. The overrides array accepts any option from the tc_control file. Overrides should be used sparingly since it makes more sense to adjust the options in the tc_control file rather than here.

tslist.auto¶

tslist.auto is an alternative to the static nature of tslist.run. When trying to add an existing test suite to UTAH, this option may work better.:

# the contents of this file help dynamically construct the names of each
# existing test case and how to run it.
# utah execute's the discovery command, and then pass each line of the output
# as a cli parameter(s) to the test_cmd
-
  # this will result in testcases name "1", "2" and "3". The output will be
  # will just be "1", "2", and "3"
  discovery_cmd: seq 3
  # the '{}' in test-cmd will be replaced with the output of the discovery
  # command.
  test_cmd: echo {}

-
  # a more complex example that breaks up unity test for autopilot
  discovery_cmd: "autopilot list unity | grep unity.tests | sed -e 's/^[[:space:]]*//' | cut -d. -f1,3 | sort | uniq"
  test_cmd: autopilot run -v {}

NOTE: The items returned by the discovery_cmd become the name of each testcase. So its wise to choose something sensible/consistent.

ts_control¶

build_cmd:      make
timeout:        300
ts_setup:       tsetup/setup.sh      # tsetup/setup.sh is the set up for
                                     # the whole test suite
ts_cleanup:     tcleanup/cleanup.sh  # clean up test suite

There are no required fields in this file.

To avoid excessive typing and repetition in runlists and control files the following rules apply:

• tslist.run‘s test is relative to the test suite. So if the suite is called sample_tests and the entry in tslist.run for test is test_one then the path to the test folder will be sample_tests/test_one.
• tslist.run‘s overrides are the same options available in the tc_control file and take precedence over those found in tc_control.
• ts_control is optional.
• if build_cmd, or ts_setup fails no tests in the suite will be run.
• if build_cmd, or tc_setup fails the test will not be run.
• ts_cleanup will be run whether or not ts_setup, build_cmd, or any tests fail.
• tc_cleanup will be run whether or not tc_setup, build_cmd, or the test fails.

Each test case, with all the code required for it to compile or run, lives in a directory, and contains also a control file that includes its documentation amongst other things:

tc_control¶

build_cmd:      make    # or scons or build.sh
command: python test1.py a1 a2 a3  # command to run the test from within t1
description:
 dependencies: coreutils
 actions: |
   1. Action 1
   2. Action 2
 expected_results: |
   1. Expected result 1
   2. Expected result 2
type:           userland/kernel    # currently unimplemented
timeout:        100
tc_setup:       t1/t1 setup    # in case the test case has a setup
tc_cleanup:     t1/t1 cleanup  # in case the test has a cleanup

In this file command, description, dependencies, actions, expected_results, and timeout are required. command is the command to actually run the test. description is a textual description of the test. dependencies is a list of items the test depends on. action is a list of actions the test executes and expected_results is a list of what those actions should result in. timeout is the ammount of time the test should complete within and is used to avoid tests that might loop indefinitely.

type currently defaults to userland and may in the future have other options as the need arises.

build_cmd can be used to build any binary test cases. tc_setup can be used to setup any needed data or files for the test and tc_cleanup can be used to remove the data or files added by tc_setup. build_cmd, tc_setup, and tc_cleanup are all simple shell commands similar to command.

For each test case that is a binary or a script, there will be an option to define a setup and a cleanup function if they wish so, and it is the test code developer’s responsibility to add the right parameters to their code so that the harness can run their setup and clean up functions.

In terms of settings precedence: The test case control file (tc_control) provides the default values for the options available for the testcase. The overrides in tslist.run take precedence over those options in the tc_control file.

reboot: <always, pass, never>