test_runner: extend tag filter with boolean expression DSL by atlowChemi · Pull Request #63054 · nodejs/node

atlowChemi · 2026-04-30T16:29:26Z

Summary

Builds on #63221, which landed the tags option, inheritance, reporter
event payloads, and a literal-tag-name filter. This PR upgrades the
filter to a boolean expression DSL:

Grammar: and/&&, or/||, not/!, parentheses for grouping,
and * wildcards inside identifiers (bare * matches any tagged
test). Standard precedence (not > and > or); binary operators are
left-associative. Word forms (and/or/not) require whitespace
separation; punctuation forms do not.
--experimental-test-tag-filter and the testTagFilters run()
option now accept an expression rather than a literal tag name. The
flag is still repeatable; multiple expressions still AND together.
Untagged tests: any include expression evaluates false against an
empty tag set; not X evaluates true. So not flaky keeps every
untagged test, while db excludes them.
Tag value validation tightens: tag names may no longer contain
whitespace, operator characters (& | ! ( ) *), or the reserved
words and/or/not in any casing. This is a breaking change
relative to test_runner: add tags option and tag-name filter #63221 - acceptable since the feature is at Stability 1.0
(Early development).
Malformed expressions fail fast at parent-process startup before any
test file is spawned.

Prior art

Tag filtering with boolean composition is well-trodden in the JS
testing ecosystem. The expression syntax here is closest to Vitest's:

Vitest test tags - https://vitest.dev/guide/test-tags
mocha-tags - https://www.npmjs.com/package/mocha-tags
jest-runner-groups - https://www.npmjs.com/package/jest-runner-groups
jest-tags - https://github.com/gitbrik/jest-tags

Vitest additionally lets a tag carry per-test config overrides such as
timeout or retry (with priority resolution between overlapping
tags). That's intentionally out of scope for this PR - landing it
would commit to a separate tags: [{ name, timeout, retry, priority }]
config surface that's orthogonal to filtering. It can be layered on
later without breaking the union-inheritance semantics established in
#63221.

nodejs-github-bot · 2026-04-30T16:29:32Z

Review requested:

@nodejs/config
@nodejs/test_runner

MoLow · 2026-04-30T17:18:39Z

+declare tags via the `tags` option on `test()`, `it()`, `suite()`, or
+`describe()`. Tags inherit from suites to nested tests by union.
+
+The expression supports boolean operators (`and`/`&&`, `or`/`||`,


isnt there a better way then this tag filtering syntax?.
i.e accept in the run method a string[] or function(string) => boolean and in case you want some complex logic - use the run api without the node cli - That makes much more sense to me

that is more consistent to our name filter flag

Without the boolean syntax this feature mostly collapses into name-pattern...

The goal (I had in mind) of this feature is to introduce a easy (and common, see prior art) way to filter without the need for a programmatic API. Sure, you could already achieve complex filtering via run() API, but the goal here is to add a built-in logic for this.

Vitest, mocha-tags, and jest-runner-groups all expose such a boolean composition syntax.

Before I even saw Moshe's comment, I was thinking the same thing. But after a minute, I can see the use of it.

I know && and || align with javascript, but it seems a little long, and the simpler & and | look pretty straightforward to me (and aligns with others, like query params and old skool forum query syntax). What happens when you combine them though?

--experimental-test-tag-filter=foo&bar&qux|zed

Is that "foo and bar and (qux or zed)" or "(foo and bar and qux) or zed"? For some reason, my eyes see the former, but syntactically, it's usually the latter. IMO the original spec for syntax made a huge mistake not requiring parentheses when combining operators. If we support combining operators, I think parentheses should be required.

--experimental-test-tag-filter=(foo&bar&qux)|zed

--experimental-test-tag-filter=foo&bar&(qux|zed)

@JakobJingleheimer I feel that & vs && is not really "a little long", and I feel using & and | could be confusing with bitwise operators, while removing just a single character...

I considered that before posting: I think nobody would be confused here because bitwise is not appropriate.

&& vs & is a thought musing.

My bigger concern would be parens for combined operators.

thats why I think we should release a initial version that does not include this new langauge

@MoLow @JakobJingleheimer I opened #63221 to separate these concerns.

codecov · 2026-04-30T18:06:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.07%. Comparing base (b06aee0) to head (3f583b5).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #63054      +/-   ##
==========================================
+ Coverage   90.04%   90.07%   +0.02%     
==========================================
  Files         714      714              
  Lines      225245   225641     +396     
  Branches    42571    42668      +97     
==========================================
+ Hits       202821   203242     +421     
- Misses      14186    14192       +6     
+ Partials     8238     8207      -31

Files with missing lines	Coverage Δ
lib/internal/test_runner/runner.js	`93.93% <100.00%> (+0.35%)`	⬆️
lib/internal/test_runner/tag_filter.js	`100.00% <100.00%> (ø)`
lib/internal/test_runner/utils.js	`65.19% <100.00%> (-0.05%)`	⬇️

... and 41 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JakobJingleheimer · 2026-04-30T20:18:32Z

Sweet! I'll try to carve out time on Saturday to reciew

JakobJingleheimer

Sorry for the delay. GitHub is having a bit of a stroke. Every time I try to add a comment to the review "Something went wrong" and I have to copy it, reload, paste, and try a couple more times.

I'll try to finish tomorrow (I got through 10 of 18 files, which doesn't include tag_filter.js).

JakobJingleheimer · 2026-05-06T18:08:28Z

+declare tags via the `tags` option on `test()`, `it()`, `suite()`, or
+`describe()`. Tags inherit from suites to nested tests by union.
+
+The expression supports boolean operators (`and`/`&&`, `or`/`||`,


Before I even saw Moshe's comment, I was thinking the same thing. But after a minute, I can see the use of it.

I know && and || align with javascript, but it seems a little long, and the simpler & and | look pretty straightforward to me (and aligns with others, like query params and old skool forum query syntax). What happens when you combine them though?

--experimental-test-tag-filter=foo&bar&qux|zed

Is that "foo and bar and (qux or zed)" or "(foo and bar and qux) or zed"? For some reason, my eyes see the former, but syntactically, it's usually the latter. IMO the original spec for syntax made a huge mistake not requiring parentheses when combining operators. If we support combining operators, I think parentheses should be required.

--experimental-test-tag-filter=(foo&bar&qux)|zed

--experimental-test-tag-filter=foo&bar&(qux|zed)

JakobJingleheimer · 2026-05-06T18:20:57Z

+* Any include expression (a tag, wildcard, `and`, or `or`) is **false**
+  for an untagged test, so untagged tests are excluded under any positive
+  filter.
+* `not X` is **true** for an untagged test, so excluding tags does not
+  accidentally remove untagged tests.


This is generally difficult to follow. I think a table of examples would be more straightforward.

JakobJingleheimer · 2026-05-06T18:35:02Z

-  fail(nesting, loc, testNumber, name, details, directive, testId) {
+  fail(nesting, loc, testNumber, name, details, directive, testId, tags) {


General comment: this is getting to be a bit of a junk-drawer of arguments 😬

I agree, perhaps we could open a good first issue Issues that are suitable for first-time contributors. issue to handle this

atlowChemi · 2026-05-06T19:11:09Z

GitHub is having a bit of a stroke.

@JakobJingleheimer github having issues? no way 😂

Upgrades the experimental tag filter introduced by stage 1 to accept a boolean expression instead of a literal tag name. Grammar: `and`/`&&`, `or`/`||`, `not`/`!`, parentheses for grouping, and `*` wildcards inside identifiers. Standard precedence (`not > and > or`); binary operators are left-associative. Word forms require whitespace separation; punctuation forms do not. Untagged tests evaluate `false` for any include expression and `true` for `not X`, so excluding tags does not accidentally remove untagged tests. The flag and `testTagFilters` option are still repeatable; multiple expressions still AND together. Malformed expressions fail fast at the parent process at startup. Tag value validation tightens to reject whitespace, operator characters (`& | ! ( ) *`), and the reserved words `and`/`or`/`not` in any casing - a breaking change relative to the stage 1 ship, acceptable at Stability 1.0 (Early development). Signed-off-by: atlowChemi <chemi@atlow.co.il>

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Apr 30, 2026

atlowChemi added the test_runner Issues and PRs related to the test runner subsystem. label Apr 30, 2026

MoLow reviewed Apr 30, 2026

View reviewed changes

JakobJingleheimer added this to Test Runner Board Apr 30, 2026

TheOneTheOnlyJJ reviewed May 1, 2026

View reviewed changes

Comment thread test/parallel/test-runner-tag-filter-cli.mjs Outdated

atlowChemi force-pushed the test_runner-tag-filter branch 3 times, most recently from 3591324 to 31a7737 Compare May 5, 2026 16:47

JakobJingleheimer reviewed May 6, 2026

View reviewed changes

atlowChemi force-pushed the test_runner-tag-filter branch from 31a7737 to 853f715 Compare May 7, 2026 19:13

atlowChemi mentioned this pull request May 10, 2026

test_runner: add tags option and tag-name filter #63221

Merged

atlowChemi force-pushed the test_runner-tag-filter branch from 853f715 to 131d524 Compare May 12, 2026 17:30

atlowChemi force-pushed the test_runner-tag-filter branch from 131d524 to 3f583b5 Compare May 12, 2026 17:51

atlowChemi changed the title ~~test_runner: add experimental tag-based test filtering~~ test_runner: extend tag filter with boolean expression DSL May 12, 2026

		fail(nesting, loc, testNumber, name, details, directive, testId) {
		fail(nesting, loc, testNumber, name, details, directive, testId, tags) {

Uh oh!

Conversation

atlowChemi commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Prior art

Uh oh!

nodejs-github-bot commented Apr 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JakobJingleheimer commented Apr 30, 2026

Uh oh!

Uh oh!

JakobJingleheimer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

atlowChemi commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

atlowChemi commented Apr 30, 2026 •

edited

Loading

codecov Bot commented Apr 30, 2026 •

edited

Loading