test_runner: support custom message for expectFailure

[Han5991]

Summary

Update expectFailure option to accept a string message and display it in the TAP reporter output.

Example output: # EXPECTED FAILURE <reason>

This aligns expectFailure behavior with skip and todo which already support custom messages.

Test Plan

Added a new test file test/parallel/test-runner-xfail-message.js that verifies:

1. expectFailure accepts a string.
2. The string is correctly output in the TAP reporter.

Documentation

Updated doc/api/test.md to include:

• expectFailure in the options list for test().
• An example of using expectFailure with a message.

[nodejs-github-bot]

Review requested:

• @nodejs/test_runner

[aduh95]

You wouldn't want expectFailure for a flaky test

Suggested change

	it('should do the thing', { expectFailure: 'flaky test' }, () => {
	it('should do the thing', { expectFailure: 'doTheThing is not doing the thing because ...' }, () => {

[JakobJingleheimer]

We already have a plan for the value of expectFail: it will soon accept a regular expression to match against.

[Han5991]

We already gave a plan for the value of expectFail: it will soon accept a regular expression to match against.

@JakobJingleheimer
Ah, thanks for the context! I missed that there was an existing plan for error matching.

In that case, I'd be happy to pivot this PR to implement the expectFailure validation logic (accepting a string/regex to match the error) instead of just a message. Does that sound good, or is there someone else already working on it?"

[JakobJingleheimer]

@vassudanagunta you were part of the original discussion; did you happen to start an implementation?

To my knowledge though, no-one has started.

I had planned to pick it up next week, but if you would like to do, go ahead.

If you do, I think it would probably be better to start a new PR than to pivot this one. So open a draft and I'll add it to the test-runner team's kanban board so it gets proper visibility.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.77%. Comparing base (8c380de) to head (5a15080).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #61563   +/-   ##
=======================================
  Coverage   89.76%   89.77%           
=======================================
  Files         673      673           
  Lines      203788   203790    +2     
  Branches    39168    39166    -2     
=======================================
+ Hits       182926   182945   +19     
+ Misses      13189    13164   -25     
- Partials     7673     7681    +8     

Files with missing lines	Coverage Δ
lib/internal/test_runner/reporter/tap.js	98.23% <100.00%> (ø)
lib/internal/test_runner/test.js	97.31% <100.00%> (+<0.01%)	⬆️

... and 41 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[vassudanagunta]

@JakobJingleheimer nope, haven't started this, though I had long ago implemented it in node-test-extra (not yet released).

I think it's important to get the requirements nailed. IMHO, #61570.

[JakobJingleheimer]

As I said, let's put together a proposal in the nodejs/test_runner repo 🙂

[vassudanagunta]

As I said

?

I should start a discussion in that repo?

[ljharb]

reviewed before reading the discussion; imo a string should work as in this PR whether or not it also supports accepting a regex.

[JakobJingleheimer]

It could do. My concern is supporting this without considering the intended regex feature accidentally precluding that intended feature, or inadvertently creating a breaking change, or creating heavily conflicting PRs (very frustrating for the implementators).

I think we can likely get both; we can easily avoid those problems with a quick proposal so everyone is on the same page 🙂

---

I should start a discussion in that repo?

Please start a proposal like the ones already in that repo 🙂 https://github.com/nodejs/test-runner/tree/main/proposals we can discuss it in that PR

[ljharb]

conflicts fair; as long as the "should expect failure" uses truthiness (does an empty string count as true or false, though?), i can't foresee any semantic collision.