# Targets definitions
This directory contains target definitions used for configuring the targets and
tests that will be executed by builders in the chromium project that use the
chromium_tests recipe module for executing tests. This includes most builders
that build and run tests. Builders running PRESUBMIT.py tests and builders
running fuzz tests are not included.
## Files
* [basic_suites.star](./basic_suites.star) - This contains definitions of basic
suites, which specify tests to include as well as optionally specify various
details for those tests such as which test target to actually build and run,
additional args that should be used or default swarming details. These suites
can be referenced by compound suites in compound_suites.star, matrix compound
suites in matrix_compound_suites.star or set as a test suite for a builder in
[waterfalls.pyl][waterfalls.pyl].
* [binaries.star](./binaries.star) - This contains declarations with binary
mappings used when declaring tests. The declarations map the GN target name to
the ninja label that needs to be built to produce the actual executable for
the test. Additional details can be specified as part of the declaration that
will be used as part of the process of generating target spec files. Tests
defined using targets.tests.gtest_test and target.tests.isolated_script_test
all require a binary to be defined, which is set via the binary field. If the
binary field is not set, then there must be a binary declared that has the
same name as the test. Tests defined using targets.tests.junit_test implicitly
define a binary that results in an entry in the generated gn_isolate_map.pyl
file.
Available binary types are:
* console_test_launcher
* generated_script
* script
* windowed_test_launcher
* [bundles.star](./bundles.star) - This contains declarations of bundles, which
are groupings of tests and compile targets that can be referenced by builders
that set their tests in starlark. These bundles cannot be referenced by any
legacy suites or by builders setting their tests in
[waterfalls.pyl][waterfalls.pyl].
* [compile_targets.star](./compile_targets.star) - This contains definition of
compile-only targets. These targets can only be built by a builder and have no
provision for execution. These targets can be included in the
additional_compile_targets field for a builder in
[waterfalls.pyl][waterfalls.pyl].
* [compound_suites.star](./compound_suites.star) - This contains definitions of
compound suites, which specify basic suites to collect into a larger suite.
These suites can set as a test suite for a builder in
[waterfalls.pyl][waterfalls.pyl].
* [matrix_compound_suites.star](./matrix_compound_suites.star) - This contains
definitions of matrix compound suites, which specify basic suites to collect
into a larger suite, optionally applying mixins to all the tests of a basic
suite and/or expanding tests in a basic suite with one or more variants. For
each suite where variants are specified, each test in the suite will generate
a test spec for each variant with the variant applying modifications to the
generated test spec. These suites can be set as a test suite for a builder in
[waterfalls.pyl][waterfalls.pyl].
* [mixins.star](./mixins.star) - This contains definitions of mixins that can be
used to modify the generated test specs. These mixins can be specified when
defining the details of a test in a basic suite to modify that test, when
configuring a basic suite in a matrix compound suite to modify all tests in
that expansion of the basic suite or on a builder in
[waterfalls.pyl][waterfalls.pyl] to modify all tests for the builder.
* [tests.star](./tests.star) - This contains definitions of test targets. These
targets can be referenced in basic suites. Additionally, they can contain
details such as args or mixins. These details will be transferred to the
generated test_suites.pyl since //testing/buildbot doesn't use separate target
declarations for tests.
Available test types are:
* gtest_test
* gpu_telemetry_test
* isolated_script_test
* junit_test
* script_test
* [variants.star](./variants.star) - This contains definitions of variants,
which allow for expanding a single test definition into multiple generated
test specs and apply additional modifications to the generated test specs.
They can be specified when configuring a basic suite in a matrix compound
suite to expand all of the tests in the basic suite with each of the specified
variants.
* [cros-skylab-variants.json](./cros-skylab-variants.json),
[lacros-version-skew-variants.json](./lacros-version-skew-variants.json) -
These files are consumed by variants.star to define certain subsets of
variants. Json files are used because they are auto-generated by builders
running periodically and json is easily manipulated programatticaly.
## Tests in starlark
Currently, a migration is in process to enable tests for a builder to be
configured as part of the builder definition itself. In order to avoid having to
manually sync definitions in two locations/systems, some of the .pyl that used
to be hand-written files in //testing/buildbot are now generated from the
starlark definitions. Due to intentional design decisions in lucicfg, the files
generated from the //infra/config starlark can't be located outside of
//infra/config/generated. //testing/buildbot/generate_buildbot_json.py has been
updated to read gn_isolate_map.pyl, mixins.pyl, test_suites.pyl and variants.pyl
from //infra/config/generated/testing.
Updating these .pyl files requires the following process:
1. Modify starlark files
| .pyl file | starlark file to edit |
|-----------|-----------------------|
|gn_isolate_map.pyl|binaries.star (or tests.star for test defined using targets.test.junit_test)|
|mixins.pyl|mixin.star|
|variants.pyl|variants.star|
|test_suites.pyl (basic_suites entries)|basic_suites.star (or tests.star/binaries.star or for values that apply to all instances of a test/binary)|
|test_suites.pyl (compound_suites entries)|compound_suites.star|
|test_suites.pyl (matrix_compound_suites entries)|matrix_compound_suites.star|
1. Regenerate the config by running [main.star](/infra/config/main.star)
`lucicfg infra/config/main.star`
On mac or linux, you can just do: `infra/config/main.star`
Then you can make any edits you wish to the hand-written .pyl files in
//testing/buildbot ([waterfalls.pyl][waterfalls.pyl] and
[test_suite_exceptions.pyl](/testing/buildbot/test_suite_exceptions.pyl)) and
run [generate_buildbot_json.py](/testing/buildbot/generate_buildbot_json.py) to
generate the targets spec files.
[waterfalls.pyl]: /testing/buildbot/waterfalls.pyl
Due to angle using mixins.pyl via a subtree repo that exports
//testing/buildbot, if mixins.pyl was modified by the above steps, it's
necessary to sync that file to //testing/buildbot.
1. Copy mixins.pyl from //infra/config/generated/testing to
//testing/buildbot by running
[sync-pyl-files.py](/infra/config/scripts/sync-pyl-files.py)
`infra/config/scripts/sync-pyl-files.py`
### Setting tests for a builder in starlark
It is now possible to specify test in starlark for builders in limited
conditions:
* No legacy matrix compound suites can be referenced
* Cannot rely on any fields being set on the builder's waterfall in
waterfalls.pyl except mixins
* Cannot rely on any fields besides additional_compile_targets, test_suites,
mixins, os_type and use_swarming being set for the builder in waterfalls.pyl
(fields that are present in targets.mixin can be specified by specifying an
in-place mixin in the mixins field instead)
* Under test_suites, test can only be specified for the scripts,
gtest_tests and isolated_scripts keys
* Only the following fields can be set on gtests & isolated scripts either
directly or via mixins:
* args
* android_args (expanding these isn't supported yet, so this won't take
effect, but won't be rejected)
* android_swarming (expanding these isn't supported yet, so this won't
take effect, but won't be rejected)
* ci_only
* isolate_profile_data
* merge
* precommit_args
* resultdb
* swarming
* test (only allowed in declaration of gtest itself)
* use_isolated_script_api
* Cannot use variants
These conditions will be removed as more support is implemented.
To set tests for a builder in starlark, the builder must be setting the
builder_spec argument. If that is the case, simply set the targets field of a
builder to the name of a bundle or a list of bundle names to include. Supported
target types create a bundle with the same name as the target, so single targets
can be referenced wherever a bundle is expected.
```starlark
ci.builder(
name = "foo-builder",
builder_spec = ...,
# Configures the builder to build all
targets = targets.bundle(
additional_compile_targets = "all",
),
)
ci.builder(
name = "bar-builder",
builder_spec = ...,
# Configures the builder to build all and run any tests and build any compile
# targets specified by the public_build_scripts bundle
targets = targets.bundle(
additional_compile_targets = "all",
targets = "public_build_scripts",
),
)
```
Fields that would be specified on the waterfall can instead be set on
targets.builder_defaults, which will apply to all builders defined in the file.
```starlark
targets.builder_defaults.set(
mixins = [
"chromium-tester-service-account",
"linux-jammy",
],
)
```
This will cause json files containing the targets specs to be written out in the
per-builder generated directory. Try builders that mirror other builders do not
need to set the targets field (and it is an error to do so). The mirroring
relationship indicates that it should use the targets configured for the
builders that it mirrors, so mirroring try builders will have their own json
files generated containing the logical union of the contents of the mirrored
builders. This makes it more clear what builders are affected by modifications
to a CI builder's targets.
Codebase Browser
chromium