test_runner: support test order randomization #61747
Conversation
|
Review requested:
|
nodejs-github-bot
added
c++
pmarchini
force-pushed
the
test_runner/deterministic-test-randomizer
branch
2 times, most recently
from
5556788 to
e6f9a59
Compare
pmarchini
added
test_runner
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #61747 +/- ##
==========================================
- Coverage 89.75% 89.72% -0.03%
==========================================
Files 674 675 +1
Lines 204416 204651 +235
Branches 39285 39331 +46
==========================================
+ Hits 183472 183631 +159
- Misses 13227 13291 +64
- Partials 7717 7729 +12
🚀 New features to boost your workflow:
|
pmarchini
force-pushed
the
test_runner/deterministic-test-randomizer
branch
from
e6f9a59 to
b3b2388
Compare
pmarchini
force-pushed
the
test_runner/deterministic-test-randomizer
branch
from
b3b2388 to
08db017
Compare
|
|
||
| The value must be an integer between `0` and `4294967295`. | ||
|
|
||
| This flag cannot be used with `--watch`. |
There was a problem hiding this comment.
IMHO, there’s little value in randomizing a run while in watch mode.
If you think it could be useful to users, I can follow up by supporting it!
There was a problem hiding this comment.
I dont think a lot of value but why is it hard to support?
| node --test "**/*.test.js" "**/*.spec.js" | ||
| ``` | ||
|
|
||
| ### Randomizing test file execution order |
There was a problem hiding this comment.
Does this have an actual value without randomising the order of the tests themself?
There was a problem hiding this comment.
We can decide to randomize at the test level!
Do you see this as a blocker?
P.S.: Thinking about it, I agree that randomizing at the test level is also needed, even as part of the initial feature. At this point, it also adds value to allow it during watch mode.
There was a problem hiding this comment.
Does this have an actual value without randomising the order of the tests themself?
I agree it would be highly valuable.
There was a problem hiding this comment.
I'm gonna push a commit ASAP that introduces it!
| const { isTestRunner } = globalOptions; | ||
|
|
||
| if (randomize) { | ||
| testFiles = shuffleArrayWithSeed(testFiles, randomSeed); |
There was a problem hiding this comment.
Does this randomize just the execution order or also the output order? I think we want the output order to be consistent?
There was a problem hiding this comment.
The output order remains consistent with the execution order, so in case of randomization, it reflects the randomized execution sequence.
IMHO, the test stream should always align with the real underlying execution!
Furthermore, in order to keep the original output, the run necessarily has to be completed!
There was a problem hiding this comment.
IMHO, the test stream should always align with the real underlying execution!
I agree, thats why I asked
|
|
||
| This flag cannot be used with `--watch`. | ||
|
|
||
| ### `--test-randomize` |
There was a problem hiding this comment.
this feature should either be dissallowed when using --test-rerun-failures, or we should somehow save the seed in the rerun state so a rerun will be deterministic.
4 participants
This PR adds test-file randomization to the test runner.
--test-randomizeexecutes test files in randomized order, and--test-random-seed=<seed>replays the same order deterministically. When no seed is provided, one is generated and printed to make reproduction straightforward.