<!-- Copyright 2020 The Chromium Authors
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file.
-->
# Chrome Benchmarking System
# Overview
This directory contains benchmarks and infrastructure to test Chrome and
Chromium and output performance measurements. These benchmarks are continuously
run on the [perf waterfall](https://ci.chromium.org/p/chrome/g/chrome.perf/console).
Also, see more information on [how Chrome measures performance](/docs/speed/how_does_chrome_measure_performance.md).
# Using The Chrome Benchmarking System
## Analyzing Results From The Perf Waterfall
The [ChromePerf Dashboard](https://chromeperf.appspot.com/) is the destination
for all metrics generated by the perf waterfall. It provides tools to set up a
dashboard for performance of a set of tests + metrics over time. In addition, it
provides the ability to launch a bisection by selecting a point on the
dashboard.
## Running A Single Test
The Chrome Benchmarking System has two methods for manually running performance tests:
run_benchmark and Pinpoint.
run_benchmark is useful for creating and debugging benchmarks using local
devices. Run from the command line, it has a number of flags useful for
determining the internal state of the benchmark. For more information, see
[Run Benchmarks Locally](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry/docs/run_benchmarks_locally.md).
[Pinpoint](https://pinpoint-dot-chromeperf.appspot.com/) wraps run_benchmark and
provides the ability to remotely run A/B benchmarks using any platform available
in our lab. It will run a benchmark for as many iterations as needed to get a
statistically significant result, then visualize it.
If your're trying to debug a test or figure out how the infrastructure works,
the easiest way is to set up the debugger in VSCode (guide
[here](../../docs/vscode_python.md))] and set a breakpoint in
/tools/perf/core/benchmark_runner.py.
## Creating New Tests (stories)
[The telemetry readme](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry)
provides an overview of how tests are structured and some of the underlying
technologies. After reading that doc, figure out if your story fits into an
existing benchmark by checking
[here](https://goto.google.com/chrome-benchmarking-sheet) (or
[here](https://bit.ly/chrome-benchmarks) for non-Googlers).
* If it does, follow the instructions next to it. If there are no instructions,
find the test type in src/tools/perf/page_sets.
* Otherwise, read the [Guideline for adding new benchmark](https://docs.google.com/document/d/1ni2MIeVnlH4bTj4yvEDMVNxgL73PqK_O9_NUm3NW3BA/edit).
After figuring out where your story fits, create a new one. There is a
considerable amount of variation between different benchmarks, so use a nearby
story as a model. You may also need to introduce custom JavaScript to drive
interactions on the page or to deal with nondeterminsim. For an example, search
[this file](https://source.chromium.org/chromium/chromium/src/+/main:tools/perf/page_sets/system_health/browsing_stories.py?q=browsing_stories.py&ss=chromium)
for browse:tools:sheets:2019.
Next, we need to create a recording of all the content requested by the test. We then use this by serving it from a web server running on either the test device or a host device (for Android tests), removing one more source of nondeterminism. To do so, follow [these instructions](https://source.chromium.org/chromium/chromium/src/+/main:tools/perf/recording_benchmarks.md).
# Tools In This Directory
This directory contains a variety of tools that can be used to run benchmarks,
interact with speed services, and manage performance waterfall configurations.
It also has commands for running functional unittests.
## `run_tests`
This command allows you to run functional tests against the python code in this
directory. For example, try:
```
./run_tests results_dashboard_unittest
```
Note that the positional argument can be any substring within the test name.
This may require you to set up your `gsutil config` first.
## `run_benchmark`
This command allows running benchmarks defined in the chromium repository,
specifically in [tools/perf/benchmarks][benchmarks_dir]. If you need it,
documentation is available on how to [run benchmarks locally][run_locally] and
how to properly [set up your device][device_setup].
[benchmarks_dir]: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/
[run_locally]: https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry/docs/run_benchmarks_locally.md
[device_setup]: https://source.chromium.org/chromium/chromium/src/+/main:docs/speed/benchmark/telemetry_device_setup.md
## `update_wpr`
A helper script to automate various tasks related to the update of
[Web Page Recordings][wpr] for our benchmarks. In can help creating new
recordings from live websites, replay those to make sure they work, upload them
to cloud storage, and finally send a CL to review with the new recordings.
[wpr]: https://github.com/catapult-project/catapult/tree/master/web_page_replay_go
Codebase Browser
chromium