# Comparative Testing Scripts for Fuchsia
The collection of python scripts inside of `//tools/fuchsia/comparative_tester`
exist to facilitate the automated building, deployment, and execution of tests
on Linux and Fuchsia devices on the same LAN as the hosting PC which will run
the scripts, as well as generating comparisons and valuable statistical data and
displaying it in an easily viewed form.
## Test Support
The automated test building and launching should currently work for any test
target within the base `chromium/src` directory. Work is also underway to
support executing Catapult tests automatically, with the same stats collection
capabilities.
## Usage
The general usage pattern for these scripts is as follows:
1. Check `target_spec.py`. Make sure that the output directories are where you
want them, and that you are running the chromium test targets that you're
interested in collecting results from. Also check to make sure that the
specifications for the Linux and Fuchsia devices are appropriate for your
specific network and OS configuration.
2. Execute `comparative_tester.py`, with any flags necessary to collect the data
of interest.
3. Run `generate_perf_report.py`. This should require no extra configuration on
your part.
4. View your results by loading `display_perf_results.html` in any web browser
and giving it the result JSONs in the output directory you specified.
### test_results.py
(_Non-Invokable_)
This file is used internally to parse test output and return objects that can be
manipulated easily by the rest of the code.
### target_spec.py
(_Non-Invokable_)
This file contains constant definitions used by other files in this folder to
represent what tests to run, where the output will live, where the test binaries
live, and so on. To add more tests for automatic building and deploying, they
should be added here.
### comparative_tester.py
_Invocation_: `comparative_tester.py --do-config --do-build --is-debug --num-repetitions=1`
This is where tests are actually executed. It has four flags of note:
* `--do-config`: makes the test script to generate an args.gn file for
the output directory, and over-writes any existing `args.gn` file. This
option is off by default, and no files will be generated or changed.
* `--is-debug`: requires the do_config flag to be set as well. Makes
the args.gn files that the script will generate have the
`is_debug = true` line. This option is off by default, and will
cause the line `is_debug = false` to appear in the `args.gn` file
* `--do-build`: makes the test script build the targets specified inside of
`target_spec.py`. By default, this is off, and the targets will not be
rebuilt for the test cases.
* `--num-repetitions`: tells the script how many times to run each test in the
battery of tests specified in `target_spec.py`. By default, this is one,
so tests will only be executed one time each.
* `--gtest_filter`: Works just like it does on the tests. Positive filters go
before the '-' and negative ones after. All individual filters are
separated from each other via a ':'. Support globbing.
* Ex: `--gtest_filter=TestFoo.Case1:TestBar.*-TestBaz.*:TestQuux.Case2`
filters out all cases in TestBaz, Case2 of TestQuux, and only runs
TestFoo.Case1, and all cases in TestBar.
More complex configuration options are present in `target_spec.py.`
### generate_perf_report.py
_Invocation_: `generate_perf_report.py`
This script takes no command line flags, but works off of many of the same
fields inside of `target_spec.py` that `comparative_tester.py` does, namely
fields instructing it where the raw data lives, and where to place the generated
statistics when it's finished generating them. It generates the mean, standard
deviation, and coefficient of variance for each target, test, and individual
line scraped from the test output, and writes them to appropriately named JSONs.
### display_perf_results.html and .js
The HTML file is just a thin shim around `display_perf_results.js` that can be
opened in any web browser to view the data. Due to the isolation between
javascript interpreters and the host's filesystem, the web page must be manually
given the final JSON files that you want to display, at which point it will draw
a table full of the data, and in a more human readable format. It accepts
multiple files at a time.
Codebase Browser
chromium