# Declarative Net Request
This doc gives a brief overview of the implementation of the
`declarativeNetRequest` API which allows extensions to specify declarative rules
to block, redirect, upgrade or modify headers on a network request.
## Rule Indexing
[Declarative Net
Request](https://developer.chrome.com/docs/extensions/reference/declarativeNetRequest/)
consumes JSON rules from extensions and evaluates them in the browser on the
extension's behalf. Before these rules can be evaluated, they are indexed into
the [flatbuffer](https://google.github.io/flatbuffers/) format. See
[extension\_ruleset.fbs](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/flat/extension_ruleset.fbs?q=extension_ruleset.fbs)
for the flatbuffer schema we use.
First a JSON
[Rule](https://developer.chrome.com/docs/extensions/reference/declarativeNetRequest/#type-Rule)
is parsed into a `base::Value`. Since reading untrustworthy JSON using C++ in a
privileged process like the browser is dangerous as per the [Rule of
2](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/security/rule-of-2.md),
we do this either in the renderer (for dynamic and session-scoped rule APIs
where extensions specify rules as part of API calls), or in a sandboxed process
for static rules. (For unpacked extensions, this is still done in the browser
process, see
[crbug.com/761107](https://bugs.chromium.org/p/chromium/issues/detail?id=761107)).
The `base::Value` is then parsed into the
`extensions::api::declarative_net_request::Rule` struct. For static rules, this
happens in the browser and any rules which fail to be parsed correctly are
ignored to maintain backwards (consider an extension written for a newer browser
version working on an older browser) and forwards (consider an extension written
for an old browser version working on a newer version) compatibility. For
dynamic and session-scoped rules, the renderers ensure that only correctly
specified JSON rules are passed to the browser, else an API error is raised.
The parsed rule is then converted to an intermediate
<code>[IndexedRule](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/indexed_rule.h)</code>
struct. Any failures here correspond to incorrectly specified rules. These
failures result in an error for dynamic and session rules APIs. However, such
failures are ignored for packed extensions where rulesets are indexed lazily.
Finally the `IndexedRule` is indexed as a flatbuffer
<code>[UrlRule](https://source.chromium.org/chromium/chromium/src/+/main:components/url_pattern_index/flat/url_pattern_index.fbs;l=82;bpv=1;bpt=0;drc=58b5cd9b129b1b8465c040e52ad4cbfef9f8265b)</code>.
Indexing rulesets in this way provides for more efficient matching since we are
able to utilize the
<code>[url_pattern_index](https://source.chromium.org/chromium/chromium/src/+/main:components/url_pattern_index/)</code>
component which does pre-computation to allow for fast matching of filter-list
style rules. Another benefit to using the flatbuffer format is that we can load
these rules from disk with minimal deserialization (excess work is only needed
for regex-style rules which are not handled by the
<code>url_pattern_index</code> component). This ensures that the browser is not
slowed down at startup when extensions utilizing `declarativeNetRequest` are
loaded.
## Rulesets
Each extension can specify static, dynamic, and session-scoped rules.
* Static rulesets are specified as JSON files and are part of the extension
package. These can be specified via the manifest to be enabled or disabled
by default. These can then be selectively enabled or disabled at runtime
using the
[updateEnabledRulesets](http://localhost:8080/docs/extensions/reference/declarativeNetRequest/#method-updateEnabledRulesets)
API.
* Dynamic rules are persisted across different sessions and can be updated
using the
[updateDynamicRules](http://localhost:8080/docs/extensions/reference/declarativeNetRequest/#method-updateDynamicRules)
API.
* Session-scoped rules are scoped to a single session and can be updated using
the
[updateSessionRules](http://localhost:8080/docs/extensions/reference/declarativeNetRequest/#method-updateSessionRules)
API.
Hence, an extension can have multiple rulesets (N different static rulesets and
1 each dynamic and session-scoped ruleset). A single Ruleset source is
represented in code as a
<code>[RulesetSource](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/ruleset_source.h)</code>
which provides utilities to index the ruleset and to load it for further
matching. A <code>RulesetSource</code> can be file-backed (represented as
<code>[FileBackedRulesetSource](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/file_backed_ruleset_source.h)</code>
in code) or not.
* Session-scoped rulesets are never persisted to disk and are entirely backed
in memory.
* Static rulesets are file backed. Static JSON rulesets are part of the
original extension package and their path is specified via the manifest.
Static indexed rulesets are also persisted within the extension directory by
Chrome under the Chrome-reserved `_metadata` folder.
* The dynamic ruleset is also file backed. The corresponding JSON and indexed
rulesets are stored under the `$profile_path/DNR Extension
Rules/$extension_id/` directory path wth `rules.json` and `rules.fbs` file
names respectively.
A static ruleset is immutable (an extension can’t change the constituent rules).
Static rulesets are indexed in a lazy manner as needed (for packed extensions)
while dynamic and session-scoped rulesets are reindexed each time they are
modified.
## Loading indexed rulesets for matching
When an extension is enabled, all its rulesets must be loaded (from disk for
static/dynamic rulesets and from memory for the session-scoped ruleset) for the
extension to work. Similarly, when an extension is disabled, all its rulesets
must be disabled as well. Additionally, when an extension is uninstalled, all
its rulesets must be deleted (from disk/memory).
<code>[RulesMonitorService](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/rules_monitor_service.h)</code>
is the profile-bound class that observes extension
loading/unloading/uninstallation and des all this work. It uses the
<code>[FileSequenceHelper](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/file_sequence_helper.h)</code>
class to aid in file-bound operations like loading rulesets from disk.
Declarative Net Request extension function implementations further forward the
implementation to the <code>RulesMonitorService</code> in most cases as well.
## Ruleset Integrity
For dynamic and static rules, the indexed and JSON rulesets are loaded from
disk. On-disk modification of resources is outside Chrome’s security model,
however we do try to ensure integrity on a best effort basis.
To ensure integrity of the indexed rulesets, Chrome maintains expected checksums
for the indexed rulesets in extension prefs which are verified prior to loading
the indexed ruleset. Additionally, flatbuffer provides utilities to ensure that
the serialized data does indeed correspond to a flatbuffer indexed blob of the
correct schema.
If loading the indexed ruleset fails (due to failed integrity checks or some
other reason), we try to reindex the JSON ruleset on disk. If even the
reindexing fails, we don't load that ruleset and a user-visible warning is
raised using
<code>[WarningService](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/warning_service.h)</code>.
We don’t have any explicit integrity checks for the JSON ruleset on disk. (There
is a [bug](https://bugs.chromium.org/p/chromium/issues/detail?id=1063206) to
ensure static JON rulesets undergo content verification since they are part of
the extension package). However, we do ensure that a reindexed ruleset still
corresponds to the last recorded checksum, thereby ensuring integrity by proxy.
(For example, if we were to change both the JSON and indexed dynamic ruleset on
disk, the dynamic ruleset will fail to load, leading to a user visible warning).
## Indexed Ruleset Schema Versioning
As mentioned previously, a file backed indexed ruleset can fail to load due to
multiple reasons (failed integrity checks, file deleted from disk, etc.). When
this happens, Chrome reindexes the JSON ruleset on disk.
Additionally, as the API evolves, the indexed flatbuffer schema format also
changes, sometimes in a backwards incompatible manner. Chrome cannot handle
indexed rulesets with an incompatible schema. To handle this, we maintain the
indexed ruleset format version in Chrome and also persist it to disk as part of
the indexed ruleset. See calls to
<code>[GetIndexedRulesetFormatVersion()](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/utils.cc;l=70;drc=58b5cd9b129b1b8465c040e52ad4cbfef9f8265b;bpv=1;bpt=1)</code>.
This indexed ruleset format version is incremented each time the flatbuffer
schema changes.
Upon reading an indexed ruleset, Chrome parses the version header from the file
and compares it to the version it understands. In case of a mismatch, the
indexed ruleset is reindexed. Note that Chrome ignores checksum mismatch in this
case, since on a schema update it’s expected for the indexed ruleset checksum to
change. It also updates the checksum stored in extension prefs.
## Rule Matching
There are several classes which aid in rule matching. At the top, we have the
`RulesetManager` class which is owned by `RulesMonitorService`.
The
<code>[RulesetManager](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/ruleset_manager.h)</code>
class manages the set of active rulesets across all enabled extensions. This
class is the entry-point for all our rule matching logic.
The active rulesets for a single extension are represented by the
<code>[CompositeMatcher](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/composite_matcher.h)</code>
class. These are owned by <code>RulesetManager</code>.
A single `RulesetSource` when loaded (and ready for matching) corresponds to a
<code>[RulesetMatcher](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/ruleset_matcher.h)</code>.
These are owned by the corresponding <code>CompositeMatcher</code> for the
extension. A <code>RulesetMatcher</code> further owns one instance each of
<code>RegexRulesMatcher</code> and <code>ExtensionUrlPatternIndexMatcher</code>.
Hence if an extension has 5 static rulesets and also uses session-scoped rules,
it will have a single `CompositeMatcher` with 6 `RulesetMatchers` (one for the
session-scoped ruleset and 5 for static rulesets).
<code>[RegexRulesMatcher](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/regex_rules_matcher.h)</code>
deals with the matching of regular expression rules within a Ruleset. This uses
the
<code>[FilteredRE2](https://source.chromium.org/chromium/chromium/src/+/main:third_party/re2/src/re2/filtered_re2.h)</code>
class from the <code>[re2](https://github.com/google/re2)</code> library for
efficient matching.
<code>[ExtensionUrlPatternIndexMatcher](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/extension_url_pattern_index_matcher.h)</code>
deals with the matching of filter-list style rules. This further uses the
<code>url_pattern_index</code> component for efficient matching (which we share
with the
<code>[subresource_filter](https://source.chromium.org/chromium/chromium/src/+/main:components/subresource_filter)</code>
component). Both `RegexRulesMatcher` and `ExtensionUrlPatternIndexMatcher` share
a common base class called
<code>[RulesetMatcherBase](https://source.chromium.org/chromium/chromium/src/+/main:extensions/browser/api/declarative_net_request/ruleset_matcher_base.h)</code>
to share logic.
## Threading
The `declarativeNetRequest` API operates mostly on the UI thread in the browser
(as does most browser communication with the network service), with the
exception of some ruleset indexing and loading operations which happen on the
file sequence. Additionally, the static JSON rulesets for packed extensions are
read in a sandboxed process, as discussed previously.
## Rule Matching algorithm
Before a network request is sent to the server, each extension is queried for an
action to take. The following actions are considered at this stage:
* Actions which block requests of type `block`
* Actions which redirect requests of type `redirect` or `upgradeScheme`
* Actions which allow requests of type `allow` or `allowAllRequests`
If more than one extension returns an action, the extension whose action type
comes first in the list above gets priority. If more than one extension returns
an action with the same priority (position in the list), the most recently
installed extension gets priority.
When an extension is queried for how to handle a request, the highest priority
matching rule is returned. If more than one matching rule has the highest
priority, the tie is broken based on the action type, in the following order of
decreasing precedence:
* `allow`
* `allowAllRequests`
* `block`
* `upgradeScheme`
* `redirect`
If the request was not blocked or redirected, the matching `modifyHeaders` rules
are evaluated with the most recently installed extensions getting priority.
Within each extension, all `modifyHeaders` rules with a priority lower than
matching `allow` or `allowAllRequests` rules are ignored.
## Interaction with webRequest API
Declarative Net Request actions are calculated during the
<code>[onBeforeRequest](https://developer.chrome.com/docs/extensions/reference/webRequest/#event-onBeforeRequest)</code>
stage of the `webRequest` API i.e. before any TCP connection is made with the
server. Actions to block, collapse, upgrade or redirect the request are applied
at this stage while header modification actions are calculated but not applied
yet (there are no headers to apply the actions to at this stage). This happens
before webRequest extensions get a chance to intercept the request, thereby
giving extensions using `declarativeNetRequest` more priority over those using
`webRequest`.
If the request was not blocked or redirected, the request proceeds. During the
<code>[onBeforeSendHeaders](https://developer.chrome.com/docs/extensions/reference/webRequest/#event-onBeforeSendHeaders)</code>
stage, after the initial set of request headers to be sent to the server have
been prepared, webRequest extensions are given a chance to propose request
header modifications. After the browser receives the response from all
`onBeforeSendHeaders` listeners, the request header modifications are applied.
Modifications proposed by `declarativeNetRequest` extensions are given priority
over those requested by `webRequest` extensions.
During the
<code>[onHeadersReceived](https://developer.chrome.com/docs/extensions/reference/webRequest/#event-onHeadersReceived)</code>
stage of the request, corresponding webRequest listeners are fired which can
propose response header modifications. After the browser receives the response
from all `onHeadersReceived` listeners, the response header modifications are
applied. Modifications proposed by `declarativeNetRequest` extensions are again
given priority over those requested by `webRequest` extensions.
Codebase Browser
chromium