UTAH and autopilot
==================

How do I run autopilot test cases through UTAH in a touch image?
----------------------------------------------------------------

To run ``autopilot`` test cases in a touch image through UTAH the following
steps would be needed:

* Write UTAH a test suite that includes the autopilot test cases

  The easiest way to write the test suite, is to use a dynamic runlist as was
  described in the :ref:`tslist.auto <tslist-auto>` section. The reason for
  this is that it isn't needed to update the test suite if some test case is
  added/removed/renamed in the autopilot code. Thanks to the discovery command,
  there's no need to explicitly set the list of test cases.

  Note that a setup script should be added to install additional packages that
  might be required like, for example, ``autopilot-touch``.

* Write UTAH a runlist that includes the test suite

  When writing a runlist, note that it's possible to let the touch device get
  the test suite using the ``bzr-export`` fetch method or do that outside of
  the device, push it to a ``/tmp`` location and let the device use the ``dev``
  fetch method. Currently, the recommended option is the second one because
  it's more reliable.

Once the runlist is ready, the device has to be provisioned. This includes
flashing the image, configuring the network and installing the UTAH client
packages::

 run_utah_phablet.py --skip-install --skip-network \
     -s <serial> --ppa ppa:utah/stable

After the device is provisioned, any files that are expected to be in the
device like the ones for the test suite have to be pushed and the test case can
be executed::

 /usr/bin/run_utah_phablet.py --skip-install --skip-network --skip-utah \
     -s <serial> --results-dir <results_dir> --runlist <remote_runlist>

How does run_utah_phablet.py work?
----------------------------------

Preparation
~~~~~~~~~~~

Before really running the ``autopilot`` test cases, ``run_utah_phablet.py``
performs the following actions:

* Write ``/usr/local/bin/utah-autorun.sh`` into the device. This script:

  * Looks into the working directory (``/etc/utah/autorun``) for executable
    files
  * Moves them one by one to ``/var/cache/utah/autorun/inprogress``
  * Runs them
  * Moves them to ``/var/cache/utah/autorun/complete``
  * Renames them to a suffix with their completion timestamp

* Write ``/etc/init/run_utah.conf``. This is an upstart job description file
  that launches ``/usr/local/bin/utah-autorun.sh`` when the ``run_utah`` event
  is emitted and writes its output to ``/tmp/utah.log``.

* Write ``/usr/local/bin/utah-autopilot`` into the device. This is a script
  that makes sure that permissions to ``/dev/uinput`` are correctly set and
  wraps ``autopilot`` to make sure it's called using the ``phablet`` user. To
  use this script, it must be explicitly called as a comment in the test suite
  as shown in the following ``tslist.auto`` file from the
  `web browser application smoke tests <http://bazaar.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/smoke-touch-apps/view/head:/webbrowser-app-autopilot/tslist.auto>`_::

   -
     discovery_cmd: "autopilot list webbrowser_app | grep webbrowser_app | sed -e 's/^.*webbrowser_app.tests.//'"
     test_cmd: utah-autopilot run -v webbrowser_app.tests.{}

* Write a script to ``/etc/utah/autorun/01_run-utah`` that calls utah with the
  required runlist and enables/disables the shell if needed.

* Launch a process to capture ``/var/log/syslog`` and write it to stdout.
* Emit the ``run_utah`` event.

Run
~~~

Once the ``run_utah`` event is emitted, is when all the preparation work comes
into play. Specifically:

* The upstart job defined in ``/etc/init/run_utah.conf`` is triggered.
* ``/usr/local/bin/utah-autorun.sh`` is executed and the output written to
  ``/tmp/utah.log``.
* ``/etc/utah/autorun/01_run-utah`` is found in the working directory.
* ``/etc/utah/autorun/01_run-utah`` is moved to
  ``/var/cache/utah/autorun/inprogress``.
* ``/etc/utah/autorun/01_run-utah`` is executed.
* ``/usr/bin/utah`` is launched with the runlist passed to
  ``run_utah_phablet.py``.
* ``/usr/local/bin/utah-autopilot`` is called for each autopilot test case
  found by the discovery command.
* ``/etc/utah/autorun/01_run-utah`` is moved to
  ``/var/cache/utah/autorun/complete``.

Results
~~~~~~~

Once the test cases have been executed, ``run_utah_phablet.py`` takes care of
the following actions:

* Pull ``/var/lib/utah/utah.yaml`` from the device to the results directory.
* Pull ``/tmp/utah.log`` from the device to the results directory.
* Pull any other result files from the device that might have been requested.