perf_waterfall.md | Explore in Territory

# chromium.perf Waterfall

## Overview

The [chrome.perf waterfall](https://ci.chromium.org/p/chrome/g/chrome.perf/console)
continuously builds and runs our performance tests on real Android, Windows,
Mac, and Linux hardware; see [list of platforms](perf_lab_platforms.md).
Results are reported to the
[Performance Dashboard](https://chromeperf.appspot.com/) for analysis. The
[Perfbot Health Sheriffing Rotation](http://goto.google.com/perf-bot-health-sheriffs) ensures that the benchmarks stay green. The [Perf Sheriff Rotation](perf_regression_sheriffing.md) ensures that any regressions detected by those benchmarks are addressed quickly. Together, these rotations maintain
[Chrome's Core Principles](https://www.chromium.org/developers/core-principles)
of speed:

> "If you make a change that regresses measured performance, you will be
> required to fix it or revert".

## How It Works

The perf waterfall is split into three stages - builders, testers, and processors.

### Builders

For each commit to the Chromium repo, a Builder is invoked. The builder builds
Google Chrome, acquires test assets, and bundles everything up to be passed
along to Testers. We execute a builder for each platform we support. To ensure
we can keep up with the rapid flow of commits, we have a set of builders per
platform, each building a different commit.

### Testers

For each platform, we have a single tester job running continuously. Each run of
a tester covers the set of commits from the end commit of the previous
invocation to the latest _built_ commit. Each tester has one or more subdevices
(shards). The tester delegates a subset of tests to each shard and aggregates
high-level results (pass/fail, runtimes).

### Processors

Processors analyze the raw data generated by each Tester and convert it into a form that can be utilized
by the [Chrome Performance Dashboard](https://chromeperf.appspot.com/report).
For some jobs, this work is executed by the tester instead.

## Contact

  * You can reach the Chromium performance sheriffs at [email protected].
  * Bugs on waterfall issues should have Component:
    [Speed>Benchmarks>Waterfall](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3ASpeed%3EBenchmarks%3EWaterfall+&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+Modified&x=m&y=releaseblock&cells=ids).
  * For domain knowledge for a specific benchmark, consider reaching out to
    benchmark owners listed in
    [benchmark.csv](https://docs.google.com/spreadsheets/d/1xaAo0_SU3iDfGdqDJZX_jRV0QtkufwHUKH3kQKF3YQs/edit#gid=0)

## Links

  * [Perf Sheriff Rotation](perf_regression_sheriffing.md)
  * [Perfbot Health Sheriffing Rotation](http://goto.google.com/perf-bot-health-sheriffs)
  * [How to SSH to Bots in Lab](https://chrome-internal.googlesource.com/infra/infra_internal/+/main/doc/ssh.md)
    (googlers only!)
  * TODO: Page on how to repro failures locally
  * TODO: Page on how to hack on buildbot/recipe code
  * TODO: Page on bringing up new hardware
chromium/docs/speed/perf_waterfall.md
