-
Notifications
You must be signed in to change notification settings - Fork 5.4k
Move libraries tests to xunit3 #125019
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move libraries tests to xunit3 #125019
Changes from 43 commits
3e5900d
ad4e927
f14edd1
104ba64
27e260a
d86dfb5
fb4696a
ae4b874
e9d947f
c09fc04
2506b40
2449ec2
a7f4dc6
fd9dc89
2db602c
d7163d7
c8fb448
578e86d
d047202
6947ca8
c27315b
ef1f6bb
e5a5759
cef6950
46430e5
c838430
c58a133
6203711
4fd1f09
14daea8
82dc328
f43ecdc
f658a30
772732a
b85c0d6
6f3b46e
d17b5a5
a6651df
b90e02a
cb57605
fdb536b
edae283
8d1b4ee
cafda2d
d6ee800
73b6324
82d3253
398ebc0
748749c
e484bf5
c462353
8eab22a
94499f8
3c7732f
f59c85d
790abcb
6f0b123
543b9eb
6c1aae0
74ba9e0
4f00ef5
b81c8ab
b9a346d
5065c8a
805fc33
a0da926
6a6a818
ab5dd0b
3956ada
42b5f88
4d766ac
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -338,6 +338,45 @@ A common usage in the libraries tests is the following: | |
|
|
||
| This is put on test classes to indicate that none of the tests in that class (which as usual run serially with respect to each other) may run concurrently with tests in another class. This is used for tests that use a lot of disk space or memory, or dominate all the cores, such that they are likely to disrupt any tests that run concurrently. | ||
|
|
||
| ## Empty `[MemberData]` in xunit v3 | ||
|
|
||
| In xunit v3, a `[Theory]` whose `[MemberData]` source returns **zero rows** is a hard failure ("No data found"), not a silent no-op. When running through the test harness this surfaces as: | ||
|
|
||
| ``` | ||
| [FATAL ERROR] System.InvalidOperationException | ||
| Cannot find test case metadata for ID <sha256-hash> | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Mismatched test case IDs might imply trying to pre-enumerate unstable data, which is a problem for both VSTest and MTP: https://xunit.net/docs/theory-data-stability-in-vs
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This also seems to be associated with theories returning no data. |
||
| ``` | ||
|
|
||
| This commonly happens when a `[MemberData]` source filters its output based on platform support (e.g., `Where(x => SomeAlgorithm.IsSupported)`) and all items are filtered out on the current platform. | ||
|
|
||
| **To diagnose**, run the test assembly directly with `-list full` to map the failing IDs to test method names, then inspect the `[MemberData]` source for conditional logic that can produce an empty enumerable. | ||
|
|
||
| **To fix**, switch to an unconditional data source and move the platform check into the test body: | ||
|
|
||
| ```cs | ||
| // BROKEN: MemberData can return zero rows | ||
| public static IEnumerable<object[]> SupportedAlgorithmsTestData => | ||
| AllAlgorithms.Where(a => MyAlgorithm.IsSupported(a)).Select(a => new object[] { a }); | ||
|
|
||
| [Theory] | ||
| [MemberData(nameof(SupportedAlgorithmsTestData))] | ||
| public void MyTest(MyAlgorithm algorithm) { /* ... */ } | ||
| ``` | ||
|
|
||
| ```cs | ||
| // FIXED: MemberData always returns rows; skip at runtime | ||
| public static IEnumerable<object[]> AllAlgorithmsTestData => | ||
| AllAlgorithms.Select(a => new object[] { a }); | ||
|
|
||
| [Theory] | ||
| [MemberData(nameof(AllAlgorithmsTestData))] | ||
| public void MyTest(MyAlgorithm algorithm) | ||
| { | ||
| Assert.SkipUnless(MyAlgorithm.IsSupported(algorithm), "Not supported on this platform."); | ||
| /* ... */ | ||
| } | ||
| ``` | ||
|
|
||
| ## FactAttribute and `Skip` | ||
|
|
||
| Another way to disable the test entirely is to use the `Skip` named argument that is used on the `FactAttribute`. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should've been true in v2 as well, and I'm surprised it became a no-op. We even added a feature to say "if this theory has no data, skip it rather than fail it": https://github.com/xunit/xunit/blob/63aad206c62c2db373a9420486aa8ebc1a3daad9/src/xunit.v3.core/TheoryAttribute.cs#L31
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I don't know why this happened to work, but it seemed to pre-v3.