Skip to content

Commit 3f583b5

Browse files
committed
test_runner: extend tag filter with boolean expression DSL
Upgrades the experimental tag filter introduced by stage 1 to accept a boolean expression instead of a literal tag name. Grammar: `and`/`&&`, `or`/`||`, `not`/`!`, parentheses for grouping, and `*` wildcards inside identifiers. Standard precedence (`not > and > or`); binary operators are left-associative. Word forms require whitespace separation; punctuation forms do not. Untagged tests evaluate `false` for any include expression and `true` for `not X`, so excluding tags does not accidentally remove untagged tests. The flag and `testTagFilters` option are still repeatable; multiple expressions still AND together. Malformed expressions fail fast at the parent process at startup. Tag value validation tightens to reject whitespace, operator characters (`& | ! ( ) *`), and the reserved words `and`/`or`/`not` in any casing - a breaking change relative to the stage 1 ship, acceptable at Stability 1.0 (Early development). Signed-off-by: atlowChemi <chemi@atlow.co.il>
1 parent b06aee0 commit 3f583b5

11 files changed

Lines changed: 905 additions & 94 deletions

doc/api/cli.md

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1413,22 +1413,28 @@ Enable module mocking in the test runner.
14131413

14141414
This feature requires `--allow-worker` if used with the [Permission Model][].
14151415

1416-
### `--experimental-test-tag-filter=<tag>`
1416+
### `--experimental-test-tag-filter='<expr>'`
14171417

14181418
<!-- YAML
14191419
added: REPLACEME
14201420
-->
14211421

14221422
> Stability: 1.0 - Early development
14231423
1424-
Run only tests whose tag set contains `<tag>`. Tests declare tags via the
1425-
`tags` option on `test()`, `it()`, `suite()`, or `describe()`; tags
1426-
inherit from suites to nested tests by union. Filtering is
1427-
case-insensitive.
1424+
Run only tests that match the provided boolean tag-filter expression. Tests
1425+
declare tags via the `tags` option on `test()`, `it()`, `suite()`, or
1426+
`describe()`. Tags inherit from suites to nested tests by union.
14281427

1429-
The flag may be specified more than once; tests must contain **every**
1430-
filter value to run. See [Test tags][] for details on declaring and
1431-
inheriting tags.
1428+
The expression supports boolean operators (`and`/`&&`, `or`/`||`,
1429+
`not`/`!`), parentheses for grouping, and `*` wildcards inside identifiers.
1430+
Standard precedence applies: `not` binds tighter than `and`, which binds
1431+
tighter than `or`. See [Test tags][] for the full grammar and behavior.
1432+
1433+
The flag may be specified more than once; multiple expressions are combined
1434+
with AND, so a test must satisfy every expression to run.
1435+
1436+
A malformed expression causes the test runner to exit with a non-zero status
1437+
before running any tests.
14321438

14331439
### `--experimental-vm-modules`
14341440

doc/api/test.md

Lines changed: 76 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -489,8 +489,8 @@ added: REPLACEME
489489
490490
Tags annotate tests and suites with arbitrary string labels. The
491491
[`--experimental-test-tag-filter`][] CLI flag (or the `testTagFilters`
492-
option on [`run()`][]) selects tests whose tag set contains every
493-
provided filter value.
492+
option on [`run()`][]) selects tests by a boolean expression over those
493+
labels.
494494

495495
Tags are an alternative to encoding metadata into test names. They are
496496
useful for cross-cutting axes such as subsystem, speed bucket, flakiness,
@@ -523,37 +523,89 @@ describe('database', { tags: ['db'] }, () => {
523523
});
524524
```
525525

526-
Tag values must be non-empty strings. Tags are matched case-insensitively;
527-
the canonical form is lowercase. Duplicates within a single `tags` array
528-
are collapsed on the lowercased form, preserving the first-seen
529-
declaration order.
526+
Tag values must be non-empty strings that contain no whitespace, no
527+
operator characters (`& | ! ( ) *`), and are not the reserved words
528+
`'and'`, `'or'`, or `'not'` in any casing. Tags are matched
529+
case-insensitively; the canonical form is lowercase. Duplicates within a
530+
single `tags` array are collapsed on the lowercased form, preserving the
531+
first-seen declaration order.
530532

531533
Hooks (`before`, `after`, `beforeEach`, `afterEach`) do not declare their
532534
own tags. They run as part of their owning suite, which carries the
533535
suite's tags.
534536

535-
### Filtering by tag
537+
### Filtering syntax
536538

537-
Each [`--experimental-test-tag-filter`][] value is a literal tag name. A
538-
test runs only when its tag set contains that name. The flag may be
539-
specified more than once; tests must match **every** filter to run. The
540-
same applies to the `testTagFilters` array on [`run()`][]. Filters are
541-
case-insensitive and AND'd with [`--test-name-pattern`][],
542-
[`--test-skip-pattern`][], and `.only` filtering.
539+
The filter expression supports:
543540

544-
Untagged tests are excluded under any non-empty filter, since the filter
545-
requires the tag to be present.
541+
* Identifiers—any non-whitespace, non-operator characters. A literal
542+
identifier matches a tag of the same value (case-insensitive).
543+
* `*` wildcards inside an identifier match any sequence of characters.
544+
A bare `*` matches any tagged test.
545+
* Boolean operators with two equivalent forms:
546+
* `and` / `&&`
547+
* `or` / `||`
548+
* `not` / `!`
549+
* Parentheses for grouping.
546550

547-
### Reading tags from inside a test
551+
The word forms (`and`, `or`, `not`) require whitespace separation; the
552+
punctuation forms do not.
553+
554+
#### Operator precedence
555+
556+
The expression is evaluated with the standard precedence
557+
`not > and > or`. Binary operators are left-associative.
558+
559+
| Expression | Equivalent grouping |
560+
| -------------- | ------------------- |
561+
| `a or b and c` | `a or (b and c)` |
562+
| `not a and b` | `(not a) and b` |
563+
564+
Use parentheses to override:
565+
566+
| Expression | Selects |
567+
| ------------------------------ | ------------------------------------------ |
568+
| `(unit or smoke) and not slow` | unit-or-smoke tests that are not also slow |
569+
| `db && !flaky` | db tests that are not flaky |
570+
| `*` | every tagged test |
571+
572+
#### Untagged tests
573+
574+
Untagged tests behave as if they have an empty tag set. As a result:
575+
576+
| Filter expression | Untagged test | Why |
577+
| ------------------------ | ------------- | ------------------------------------------------ |
578+
| `db` | excluded | Positive match against an empty tag set is false |
579+
| `*` | excluded | The bare wildcard requires at least one tag |
580+
| `db or unit` | excluded | Both branches are false against an empty tag set |
581+
| `not flaky` | included | Negation against an empty tag set is true |
582+
| `not flaky and not slow` | included | Both negations are true against an empty tag set |
583+
| `db or not flaky` | included | The negated branch is true |
584+
585+
For example, `--experimental-test-tag-filter='not flaky'` runs every test
586+
that is not tagged `flaky`, including all untagged tests.
587+
588+
#### Composing multiple filters
589+
590+
[`--experimental-test-tag-filter`][] may be specified more than once on the
591+
command line. Multiple expressions compose by AND—a test must satisfy
592+
every expression to run. The same applies to passing an array to
593+
`testTagFilters` on [`run()`][]. The tag filter is also AND'd with
594+
[`--test-name-pattern`][], [`--test-skip-pattern`][], and `.only`
595+
filtering.
596+
597+
#### Reading tags from inside a test
548598

549599
The [`TestContext`][] object exposes the test's tags as a frozen array
550600
through [`context.tags`][], so tests can branch on their own metadata.
551601

552-
### Errors
602+
#### Errors
553603

554604
A tag value that violates the validation rules above throws
555605
`ERR_INVALID_ARG_VALUE` at the registration site, before any test runs.
556-
A non-array `tags` value throws `ERR_INVALID_ARG_TYPE`.
606+
A non-array `tags` value throws `ERR_INVALID_ARG_TYPE`. A malformed
607+
filter expression on the CLI causes the test runner to exit with a
608+
non-zero status before running any test files.
557609

558610
## Extraneous asynchronous activity
559611

@@ -826,7 +878,7 @@ test runner functionality:
826878

827879
* `--test` - Prevented to avoid recursive test execution
828880
* `--experimental-test-coverage` - Managed by the test runner
829-
* `--experimental-test-tag-filter` - Filter values are validated by the parent
881+
* `--experimental-test-tag-filter` - Filter expressions are validated by the parent
830882
process and re-emitted to child processes
831883
* `--watch` - Watch mode is handled at the parent level
832884
* `--experimental-default-config-file` - Config file loading is handled by the parent
@@ -1737,10 +1789,11 @@ changes:
17371789
For each test that is executed, any corresponding test hooks, such as
17381790
`beforeEach()`, are also run.
17391791
**Default:** `undefined`.
1740-
* `testTagFilters` {string|string\[]} A tag name, or an array of tag names,
1741-
used to filter tests by their declared tags. Tests must contain every
1742-
listed tag to run. Equivalent to passing [`--experimental-test-tag-filter`][]
1743-
on the command line. See [Test tags][]. **Default:** `undefined`.
1792+
* `testTagFilters` {string|string\[]} A boolean expression, or an array of
1793+
boolean expressions, used to filter tests by their declared tags.
1794+
Multiple expressions compose by AND. Equivalent to passing
1795+
[`--experimental-test-tag-filter`][] on the command line. See
1796+
[Test tags][]. **Default:** `undefined`.
17441797
* `timeout` {number} A number of milliseconds the test execution will
17451798
fail after.
17461799
If unspecified, subtests inherit this value from their parent.

doc/node.1

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -776,9 +776,11 @@ collecting code coverage from tests for more details.
776776
Enable module mocking in the test runner.
777777
This feature requires \fB--allow-worker\fR if used with the Permission Model.
778778
.
779-
.It Fl -experimental-test-tag-filter Ar tag
780-
Run only tests whose tag set contains \fItag\fR. May be specified multiple
781-
times; tests must contain every filter to run.
779+
.It Fl -experimental-test-tag-filter Ar expr
780+
Run only tests that match the boolean tag-filter expression.
781+
The expression supports \fBand\fR/\fB&&\fR, \fBor\fR/\fB||\fR, \fBnot\fR/\fB!\fR,
782+
parentheses, and \fB*\fR wildcards. May be specified multiple times; multiple
783+
expressions are AND'd together.
782784
.
783785
.It Fl -experimental-vm-modules
784786
Enable experimental ES Module support in the \fBnode:vm\fR module.

lib/internal/test_runner/runner.js

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ const {
9393
parseCommandLine,
9494
} = require('internal/test_runner/utils');
9595
const {
96-
validateAndCanonicalizeTagFilter,
96+
parseTagFilterExpression,
9797
} = require('internal/test_runner/tag_filter');
9898
const { Glob } = require('internal/fs/glob');
9999
const { once } = require('events');
@@ -217,7 +217,7 @@ function getRunArgs(path, { forceExit,
217217
ArrayPrototypeForEach(testSkipPatterns, (pattern) => ArrayPrototypePush(runArgs, `--test-skip-pattern=${pattern}`));
218218
}
219219
if (testTagFilterExpressions != null) {
220-
ArrayPrototypeForEach(testTagFilterExpressions, (value) => ArrayPrototypePush(runArgs, `--experimental-test-tag-filter=${value}`));
220+
ArrayPrototypeForEach(testTagFilterExpressions, (expr) => ArrayPrototypePush(runArgs, `--experimental-test-tag-filter=${expr}`));
221221
}
222222
if (only === true) {
223223
ArrayPrototypePush(runArgs, '--test-only');
@@ -809,19 +809,34 @@ function run(options = kEmptyObject) {
809809
});
810810
}
811811

812+
// The public contract of testTagFilters is `string | string[]`. The
813+
// parseCommandLine bootstrap path piggybacks the already-parsed AST array
814+
// on the same field, identifiable by the sibling testTagFilterExpressions
815+
// field which only that path sets. When that marker is present and the
816+
// first element isn't a string, treat the array as ASTs and skip the
817+
// public validation loop. Otherwise validate every element as a string,
818+
// so any non-string input throws ERR_INVALID_ARG_TYPE with the offending
819+
// index regardless of position.
812820
let testTagFilterExpressions = null;
813821
if (testTagFilters != null) {
814822
if (!ArrayIsArray(testTagFilters)) {
815823
testTagFilters = [testTagFilters];
816824
}
817825
if (testTagFilters.length === 0) {
818826
testTagFilters = null;
827+
} else if (options.testTagFilterExpressions != null &&
828+
typeof testTagFilters[0] !== 'string') {
829+
// Internal bootstrap: trust the AST array as already-parsed.
819830
} else {
820831
emitExperimentalWarning('Test tags');
821-
testTagFilters = ArrayPrototypeMap(testTagFilters, (value, i) => (
822-
validateAndCanonicalizeTagFilter(value, `options.testTagFilters[${i}]`)
823-
));
824-
testTagFilterExpressions = testTagFilters;
832+
testTagFilterExpressions = ArrayPrototypeSlice(testTagFilters);
833+
testTagFilters = ArrayPrototypeMap(testTagFilters, (value, i) => {
834+
const name = `options.testTagFilters[${i}]`;
835+
if (typeof value !== 'string') {
836+
throw new ERR_INVALID_ARG_TYPE(name, 'string', value);
837+
}
838+
return parseTagFilterExpression(value, name);
839+
});
825840
}
826841
}
827842
testTagFilterExpressions ??= options.testTagFilterExpressions;
@@ -922,7 +937,6 @@ function run(options = kEmptyObject) {
922937
inspectPort,
923938
testNamePatterns,
924939
testSkipPatterns,
925-
testTagFilters,
926940
testTagFilterExpressions,
927941
hasFiles: files != null,
928942
globPatterns,

0 commit comments

Comments
 (0)