Implement early exit for retry mechanisms by daniel-mohedano · Pull Request #10604 · DataDog/dd-trace-java

daniel-mohedano · 2026-02-17T11:59:25Z

What Does This Do

Implements early exit mechanism for the RunNTimes execution policy to avoid unnecessary retries. Currently applies to:
- Early Flake Detection: retries will be stopped early when test shows signs of flakiness (a passing and failing execution).
- Attempt To Fix: retries will be stopped early when test fails.
The early exit policy is evaluated when determining if there are any retries left for the test.

Motivation

With these changes we will avoid waisting execution time and resources in retries that won't add meaningful information for each feature:

Early Flake Detection aims at detecting flaky tests
Attempt to Fix aims at detecting if a flaky test has been fixed by detecting if it passes X times in a row

Additional Notes

test-environment-trigger: skip

Most changes in the PR are deletions of retried test's spans for Attempt to Fix + failed tests scenarios.
Some fixture updates also contain the recently added test.final_status tag, which wasn't correctly updated.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors

Jira ticket: SDTEST-3387

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

pr-commenter · 2026-02-17T12:47:56Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/retry-early-exit
git_commit_date	1770902831	1771411093
git_commit_sha	`2ebc964`	`8350acb`
release_version	1.60.0-SNAPSHOT~2ebc964340	1.60.0-SNAPSHOT~8350acbf3b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1771412846	1771412846
ci_job_id	1434719043	1434719043
ci_pipeline_id	97171138	97171138
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-uojewhk3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-uojewhk3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 metrics, 9 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~8350acbf3b, baseline=1.60.0-SNAPSHOT~2ebc964340

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1065951
Total [baseline] (8.743 s) : 0, 8743019
Agent [candidate] (1.072 s) : 0, 1071959
Total [candidate] (8.746 s) : 0, 8746297
section iast
Agent [baseline] (1.232 s) : 0, 1232192
Total [baseline] (9.448 s) : 0, 9448239
Agent [candidate] (1.231 s) : 0, 1231482
Total [candidate] (9.354 s) : 0, 9354464

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.066 s	-
Agent	iast	1.232 s	166.241 ms (15.6%)
Total	tracing	8.743 s	-
Total	iast	9.448 s	705.219 ms (8.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.072 s	-
Agent	iast	1.231 s	159.523 ms (14.9%)
Total	tracing	8.746 s	-
Total	iast	9.354 s	608.167 ms (7.0%)

gantt
    title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~8350acbf3b, baseline=1.60.0-SNAPSHOT~2ebc964340

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.218 ms) : 0, 1218
BytebuddyAgent [baseline] (628.808 ms) : 0, 628808
BytebuddyAgent [candidate] (633.403 ms) : 0, 633403
AgentMeter [baseline] (28.995 ms) : 0, 28995
AgentMeter [candidate] (29.164 ms) : 0, 29164
GlobalTracer [baseline] (257.741 ms) : 0, 257741
GlobalTracer [candidate] (259.03 ms) : 0, 259030
AppSec [baseline] (32.752 ms) : 0, 32752
AppSec [candidate] (33.12 ms) : 0, 33120
Debugger [baseline] (60.6 ms) : 0, 60600
Debugger [candidate] (62.152 ms) : 0, 62152
Remote Config [baseline] (625.064 µs) : 0, 625
Remote Config [candidate] (618.462 µs) : 0, 618
Telemetry [baseline] (13.769 ms) : 0, 13769
Telemetry [candidate] (11.619 ms) : 0, 11619
Flare Poller [baseline] (6.1 ms) : 0, 6100
Flare Poller [candidate] (6.192 ms) : 0, 6192
section iast
crashtracking [baseline] (1.177 ms) : 0, 1177
crashtracking [candidate] (1.173 ms) : 0, 1173
BytebuddyAgent [baseline] (796.174 ms) : 0, 796174
BytebuddyAgent [candidate] (795.528 ms) : 0, 795528
AgentMeter [baseline] (11.282 ms) : 0, 11282
AgentMeter [candidate] (11.248 ms) : 0, 11248
GlobalTracer [baseline] (247.51 ms) : 0, 247510
GlobalTracer [candidate] (248.429 ms) : 0, 248429
AppSec [baseline] (34.775 ms) : 0, 34775
AppSec [candidate] (33.125 ms) : 0, 33125
Debugger [baseline] (66.134 ms) : 0, 66134
Debugger [candidate] (67.054 ms) : 0, 67054
Remote Config [baseline] (565.686 µs) : 0, 566
Remote Config [candidate] (554.095 µs) : 0, 554
Telemetry [baseline] (8.777 ms) : 0, 8777
Telemetry [candidate] (8.714 ms) : 0, 8714
Flare Poller [baseline] (3.513 ms) : 0, 3513
Flare Poller [candidate] (3.496 ms) : 0, 3496
IAST [baseline] (26.943 ms) : 0, 26943
IAST [candidate] (26.888 ms) : 0, 26888

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~8350acbf3b, baseline=1.60.0-SNAPSHOT~2ebc964340

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1065667
Total [baseline] (10.777 s) : 0, 10776678
Agent [candidate] (1.063 s) : 0, 1062841
Total [candidate] (10.849 s) : 0, 10848968
section appsec
Agent [baseline] (1.239 s) : 0, 1239376
Total [baseline] (11.087 s) : 0, 11087383
Agent [candidate] (1.244 s) : 0, 1244051
Total [candidate] (11.098 s) : 0, 11098443
section iast
Agent [baseline] (1.236 s) : 0, 1235644
Total [baseline] (11.164 s) : 0, 11163732
Agent [candidate] (1.241 s) : 0, 1241026
Total [candidate] (11.227 s) : 0, 11226717
section profiling
Agent [baseline] (1.189 s) : 0, 1189338
Total [baseline] (10.989 s) : 0, 10989268
Agent [candidate] (1.208 s) : 0, 1208246
Total [candidate] (11.085 s) : 0, 11084840

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.066 s	-
Agent	appsec	1.239 s	173.71 ms (16.3%)
Agent	iast	1.236 s	169.977 ms (16.0%)
Agent	profiling	1.189 s	123.671 ms (11.6%)
Total	tracing	10.777 s	-
Total	appsec	11.087 s	310.705 ms (2.9%)
Total	iast	11.164 s	387.054 ms (3.6%)
Total	profiling	10.989 s	212.59 ms (2.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.063 s	-
Agent	appsec	1.244 s	181.21 ms (17.0%)
Agent	iast	1.241 s	178.185 ms (16.8%)
Agent	profiling	1.208 s	145.404 ms (13.7%)
Total	tracing	10.849 s	-
Total	appsec	11.098 s	249.476 ms (2.3%)
Total	iast	11.227 s	377.749 ms (3.5%)
Total	profiling	11.085 s	235.872 ms (2.2%)

gantt
    title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~8350acbf3b, baseline=1.60.0-SNAPSHOT~2ebc964340

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.178 ms) : 0, 1178
crashtracking [candidate] (1.176 ms) : 0, 1176
BytebuddyAgent [baseline] (629.681 ms) : 0, 629681
BytebuddyAgent [candidate] (628.92 ms) : 0, 628920
AgentMeter [baseline] (29.034 ms) : 0, 29034
AgentMeter [candidate] (28.903 ms) : 0, 28903
GlobalTracer [baseline] (257.908 ms) : 0, 257908
GlobalTracer [candidate] (257.936 ms) : 0, 257936
AppSec [baseline] (32.719 ms) : 0, 32719
AppSec [candidate] (32.694 ms) : 0, 32694
Debugger [baseline] (63.675 ms) : 0, 63675
Debugger [candidate] (62.063 ms) : 0, 62063
Remote Config [baseline] (615.257 µs) : 0, 615
Remote Config [candidate] (638.063 µs) : 0, 638
Telemetry [baseline] (10.91 ms) : 0, 10910
Telemetry [candidate] (10.706 ms) : 0, 10706
Flare Poller [baseline] (4.66 ms) : 0, 4660
Flare Poller [candidate] (4.586 ms) : 0, 4586
section appsec
crashtracking [baseline] (1.172 ms) : 0, 1172
crashtracking [candidate] (1.178 ms) : 0, 1178
BytebuddyAgent [baseline] (658.818 ms) : 0, 658818
BytebuddyAgent [candidate] (660.507 ms) : 0, 660507
AgentMeter [baseline] (12.01 ms) : 0, 12010
AgentMeter [candidate] (12.031 ms) : 0, 12031
GlobalTracer [baseline] (258.015 ms) : 0, 258015
GlobalTracer [candidate] (259.039 ms) : 0, 259039
AppSec [baseline] (167.515 ms) : 0, 167515
AppSec [candidate] (168.883 ms) : 0, 168883
Debugger [baseline] (67.408 ms) : 0, 67408
Debugger [candidate] (67.383 ms) : 0, 67383
Remote Config [baseline] (660.64 µs) : 0, 661
Remote Config [candidate] (658.062 µs) : 0, 658
Telemetry [baseline] (9.657 ms) : 0, 9657
Telemetry [candidate] (9.718 ms) : 0, 9718
Flare Poller [baseline] (3.718 ms) : 0, 3718
Flare Poller [candidate] (3.762 ms) : 0, 3762
IAST [baseline] (25.125 ms) : 0, 25125
IAST [candidate] (25.52 ms) : 0, 25520
section iast
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.198 ms) : 0, 1198
BytebuddyAgent [baseline] (799.385 ms) : 0, 799385
BytebuddyAgent [candidate] (801.251 ms) : 0, 801251
AgentMeter [baseline] (11.297 ms) : 0, 11297
AgentMeter [candidate] (11.372 ms) : 0, 11372
GlobalTracer [baseline] (248.48 ms) : 0, 248480
GlobalTracer [candidate] (250.664 ms) : 0, 250664
AppSec [baseline] (32.402 ms) : 0, 32402
AppSec [candidate] (34.338 ms) : 0, 34338
Debugger [baseline] (68.106 ms) : 0, 68106
Debugger [candidate] (67.204 ms) : 0, 67204
Remote Config [baseline] (535.412 µs) : 0, 535
Remote Config [candidate] (535.467 µs) : 0, 535
Telemetry [baseline] (8.55 ms) : 0, 8550
Telemetry [candidate] (8.539 ms) : 0, 8539
Flare Poller [baseline] (3.443 ms) : 0, 3443
Flare Poller [candidate] (3.436 ms) : 0, 3436
IAST [baseline] (26.901 ms) : 0, 26901
IAST [candidate] (27.228 ms) : 0, 27228
section profiling
crashtracking [baseline] (1.216 ms) : 0, 1216
crashtracking [candidate] (1.23 ms) : 0, 1230
BytebuddyAgent [baseline] (680.83 ms) : 0, 680830
BytebuddyAgent [candidate] (693.52 ms) : 0, 693520
AgentMeter [baseline] (8.581 ms) : 0, 8581
AgentMeter [candidate] (8.74 ms) : 0, 8740
GlobalTracer [baseline] (215.797 ms) : 0, 215797
GlobalTracer [candidate] (218.936 ms) : 0, 218936
AppSec [baseline] (32.638 ms) : 0, 32638
AppSec [candidate] (33.147 ms) : 0, 33147
Debugger [baseline] (67.073 ms) : 0, 67073
Debugger [candidate] (68.572 ms) : 0, 68572
Remote Config [baseline] (619.95 µs) : 0, 620
Remote Config [candidate] (629.234 µs) : 0, 629
Telemetry [baseline] (8.893 ms) : 0, 8893
Telemetry [candidate] (8.989 ms) : 0, 8989
Flare Poller [baseline] (3.769 ms) : 0, 3769
Flare Poller [candidate] (3.748 ms) : 0, 3748
ProfilingAgent [baseline] (100.103 ms) : 0, 100103
ProfilingAgent [candidate] (99.921 ms) : 0, 99921
Profiling [baseline] (100.674 ms) : 0, 100674
Profiling [candidate] (100.51 ms) : 0, 100510

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/retry-early-exit
git_commit_date	1770902831	1771411093
git_commit_sha	`2ebc964`	`8350acb`
release_version	1.60.0-SNAPSHOT~2ebc964340	1.60.0-SNAPSHOT~8350acbf3b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1771413438	1771413438
ci_job_id	1434719045	1434719045
ci_pipeline_id	97171138	97171138
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-3-bk0qebsq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-3-bk0qebsq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 5 performance improvements and 1 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-231.684µs; -104.195µs] or [-13.220%; -5.946%]	unstable [-1311.882µs; -354.123µs] or [-24.580%; -6.635%]	unstable [+50.694op/s; +553.618op/s] or [+2.512%; +27.436%]	1.585ms	4.504ms	2320.031op/s	1.752ms	5.337ms	2017.875op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load	better [-250.266µs; -168.727µs] or [-8.795%; -5.930%]	better [-628.343µs; -234.031µs] or [-7.856%; -2.926%]	unstable [-51.483op/s; +227.171op/s] or [-4.076%; +17.985%]	2.636ms	7.568ms	1350.938op/s	2.845ms	7.999ms	1263.094op/s
scenario:load:petclinic:code_origins:high_load	worse [+0.739ms; +1.589ms] or [+4.291%; +9.225%]	unsure [+0.528ms; +1.781ms] or [+1.856%; +6.269%]	unstable [-41.665op/s; +10.602op/s] or [-15.667%; +3.987%]	18.394ms	29.572ms	250.406op/s	17.230ms	28.418ms	265.938op/s
scenario:load:petclinic:appsec:high_load	better [-985.374µs; -521.219µs] or [-5.182%; -2.741%]	better [-2.177ms; -0.849ms] or [-6.980%; -2.721%]	unstable [-14.745op/s; +33.870op/s] or [-6.120%; +14.058%]	18.262ms	29.677ms	250.500op/s	19.016ms	31.190ms	240.938op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~8350acbf3b, baseline=1.60.0-SNAPSHOT~2ebc964340
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.192 ms) : 1180, 1204
.   : milestone, 1192,
iast (3.203 ms) : 3161, 3245
.   : milestone, 3203,
iast_FULL (5.93 ms) : 5870, 5990
.   : milestone, 5930,
iast_GLOBAL (3.632 ms) : 3569, 3696
.   : milestone, 3632,
profiling (2.246 ms) : 2225, 2266
.   : milestone, 2246,
tracing (1.738 ms) : 1724, 1753
.   : milestone, 1738,
section candidate
no_agent (1.194 ms) : 1182, 1207
.   : milestone, 1194,
iast (3.23 ms) : 3186, 3275
.   : milestone, 3230,
iast_FULL (5.95 ms) : 5890, 6010
.   : milestone, 5950,
iast_GLOBAL (3.39 ms) : 3335, 3445
.   : milestone, 3390,
profiling (1.943 ms) : 1926, 1960
.   : milestone, 1943,
tracing (1.803 ms) : 1788, 1818
.   : milestone, 1803,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.192 ms [1.18 ms, 1.204 ms]	-
iast	3.203 ms [3.161 ms, 3.245 ms]	2.011 ms (168.7%)
iast_FULL	5.93 ms [5.87 ms, 5.99 ms]	4.738 ms (397.5%)
iast_GLOBAL	3.632 ms [3.569 ms, 3.696 ms]	2.44 ms (204.7%)
profiling	2.246 ms [2.225 ms, 2.266 ms]	1.054 ms (88.4%)
tracing	1.738 ms [1.724 ms, 1.753 ms]	546.192 µs (45.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.194 ms [1.182 ms, 1.207 ms]	-
iast	3.23 ms [3.186 ms, 3.275 ms]	2.036 ms (170.5%)
iast_FULL	5.95 ms [5.89 ms, 6.01 ms]	4.755 ms (398.2%)
iast_GLOBAL	3.39 ms [3.335 ms, 3.445 ms]	2.196 ms (183.8%)
profiling	1.943 ms [1.926 ms, 1.96 ms]	748.819 µs (62.7%)
tracing	1.803 ms [1.788 ms, 1.818 ms]	608.83 µs (51.0%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~8350acbf3b, baseline=1.60.0-SNAPSHOT~2ebc964340
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.149 ms) : 18954, 19345
.   : milestone, 19149,
appsec (19.377 ms) : 19180, 19574
.   : milestone, 19377,
code_origins (17.548 ms) : 17374, 17722
.   : milestone, 17548,
iast (17.457 ms) : 17282, 17632
.   : milestone, 17457,
profiling (18.732 ms) : 18545, 18919
.   : milestone, 18732,
tracing (17.597 ms) : 17425, 17769
.   : milestone, 17597,
section candidate
no_agent (18.458 ms) : 18270, 18647
.   : milestone, 18458,
appsec (18.635 ms) : 18447, 18822
.   : milestone, 18635,
code_origins (18.638 ms) : 18453, 18824
.   : milestone, 18638,
iast (17.403 ms) : 17233, 17574
.   : milestone, 17403,
profiling (18.737 ms) : 18553, 18921
.   : milestone, 18737,
tracing (17.711 ms) : 17533, 17889
.   : milestone, 17711,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.149 ms [18.954 ms, 19.345 ms]	-
appsec	19.377 ms [19.18 ms, 19.574 ms]	228.029 µs (1.2%)
code_origins	17.548 ms [17.374 ms, 17.722 ms]	-1.601 ms (-8.4%)
iast	17.457 ms [17.282 ms, 17.632 ms]	-1.692 ms (-8.8%)
profiling	18.732 ms [18.545 ms, 18.919 ms]	-417.261 µs (-2.2%)
tracing	17.597 ms [17.425 ms, 17.769 ms]	-1.553 ms (-8.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.458 ms [18.27 ms, 18.647 ms]	-
appsec	18.635 ms [18.447 ms, 18.822 ms]	176.377 µs (1.0%)
code_origins	18.638 ms [18.453 ms, 18.824 ms]	179.94 µs (1.0%)
iast	17.403 ms [17.233 ms, 17.574 ms]	-1.055 ms (-5.7%)
profiling	18.737 ms [18.553 ms, 18.921 ms]	278.923 µs (1.5%)
tracing	17.711 ms [17.533 ms, 17.889 ms]	-747.302 µs (-4.0%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/retry-early-exit
git_commit_date	1770902831	1771411093
git_commit_sha	`2ebc964`	`8350acb`
release_version	1.60.0-SNAPSHOT~2ebc964340	1.60.0-SNAPSHOT~8350acbf3b

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1771413111	1771413111
ci_job_id	1434719048	1434719048
ci_pipeline_id	97171138	97171138
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-200adn9e 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-200adn9e 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~8350acbf3b, baseline=1.60.0-SNAPSHOT~2ebc964340
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.47 ms) : 1458, 1482
.   : milestone, 1470,
appsec (3.758 ms) : 3538, 3978
.   : milestone, 3758,
iast (2.238 ms) : 2170, 2307
.   : milestone, 2238,
iast_GLOBAL (2.294 ms) : 2224, 2364
.   : milestone, 2294,
profiling (2.056 ms) : 2002, 2110
.   : milestone, 2056,
tracing (2.046 ms) : 1993, 2099
.   : milestone, 2046,
section candidate
no_agent (1.467 ms) : 1456, 1479
.   : milestone, 1467,
appsec (3.703 ms) : 3489, 3918
.   : milestone, 3703,
iast (2.238 ms) : 2169, 2306
.   : milestone, 2238,
iast_GLOBAL (2.28 ms) : 2211, 2349
.   : milestone, 2280,
profiling (2.057 ms) : 2003, 2111
.   : milestone, 2057,
tracing (2.046 ms) : 1993, 2099
.   : milestone, 2046,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.458 ms, 1.482 ms]	-
appsec	3.758 ms [3.538 ms, 3.978 ms]	2.288 ms (155.6%)
iast	2.238 ms [2.17 ms, 2.307 ms]	768.43 µs (52.3%)
iast_GLOBAL	2.294 ms [2.224 ms, 2.364 ms]	824.152 µs (56.1%)
profiling	2.056 ms [2.002 ms, 2.11 ms]	586.134 µs (39.9%)
tracing	2.046 ms [1.993 ms, 2.099 ms]	576.321 µs (39.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.467 ms [1.456 ms, 1.479 ms]	-
appsec	3.703 ms [3.489 ms, 3.918 ms]	2.236 ms (152.4%)
iast	2.238 ms [2.169 ms, 2.306 ms]	770.522 µs (52.5%)
iast_GLOBAL	2.28 ms [2.211 ms, 2.349 ms]	812.534 µs (55.4%)
profiling	2.057 ms [2.003 ms, 2.111 ms]	589.821 µs (40.2%)
tracing	2.046 ms [1.993 ms, 2.099 ms]	578.65 µs (39.4%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~8350acbf3b, baseline=1.60.0-SNAPSHOT~2ebc964340
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.958 s) : 14958000, 14958000
.   : milestone, 14958000,
appsec (15.045 s) : 15045000, 15045000
.   : milestone, 15045000,
iast (18.147 s) : 18147000, 18147000
.   : milestone, 18147000,
iast_GLOBAL (17.663 s) : 17663000, 17663000
.   : milestone, 17663000,
profiling (15.129 s) : 15129000, 15129000
.   : milestone, 15129000,
tracing (14.657 s) : 14657000, 14657000
.   : milestone, 14657000,
section candidate
no_agent (15.243 s) : 15243000, 15243000
.   : milestone, 15243000,
appsec (14.836 s) : 14836000, 14836000
.   : milestone, 14836000,
iast (18.083 s) : 18083000, 18083000
.   : milestone, 18083000,
iast_GLOBAL (17.96 s) : 17960000, 17960000
.   : milestone, 17960000,
profiling (15.065 s) : 15065000, 15065000
.   : milestone, 15065000,
tracing (14.787 s) : 14787000, 14787000
.   : milestone, 14787000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.958 s [14.958 s, 14.958 s]	-
appsec	15.045 s [15.045 s, 15.045 s]	87.0 ms (0.6%)
iast	18.147 s [18.147 s, 18.147 s]	3.189 s (21.3%)
iast_GLOBAL	17.663 s [17.663 s, 17.663 s]	2.705 s (18.1%)
profiling	15.129 s [15.129 s, 15.129 s]	171.0 ms (1.1%)
tracing	14.657 s [14.657 s, 14.657 s]	-301.0 ms (-2.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.243 s [15.243 s, 15.243 s]	-
appsec	14.836 s [14.836 s, 14.836 s]	-407.0 ms (-2.7%)
iast	18.083 s [18.083 s, 18.083 s]	2.84 s (18.6%)
iast_GLOBAL	17.96 s [17.96 s, 17.96 s]	2.717 s (17.8%)
profiling	15.065 s [15.065 s, 15.065 s]	-178.0 ms (-1.2%)
tracing	14.787 s [14.787 s, 14.787 s]	-456.0 ms (-3.0%)

nikita-tkachenko-datadog · 2026-02-18T15:51:53Z

...gent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/exit/NoExit.java

+package datadog.trace.civisibility.execution.exit;
+
+/** Policy that ignores early exit on retries. Currently only used in testing. */
+public class NoExit implements EarlyExitPolicy {


Maybe we could rename it to NoEarlyExit so it sounds less dramatic :)

nikita-tkachenko-datadog · 2026-02-18T15:52:19Z

...gent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/exit/NoExit.java

This class doesn't seem to be used anywhere, do we need it?

Created it for testing purposes to maintain the old test cases, but in hindsight I can totally remove it and align the test cases with the only two possible policies currently (ExitOnFlake and ExitOnFailure)

nikita-tkachenko-datadog · 2026-02-18T15:53:06Z

...ent-ci-visibility/src/main/java/datadog/trace/civisibility/events/TestEventsHandlerImpl.java

        test.setTag(Tags.TEST_FINAL_STATUS, outcome.finalStatus());

-        if (outcome.failedAllRetries()) {
+        if (retryReason != null && outcome.failedAllRetries()) {


Out of curiosity, what's the reason for this change?

We want to avoid tagging tests with @test.has_failed_all_retries when they haven't actually been retried. This specific path could be triggered for an Attempt to Fix test that fails on the first run and exits early out of being retried, causing its only execution (a failure) to be the final one

Looking at it now, it might be worth considering renaming the Outcome attributes to succeededAllExecutions and failedAllExecutions given that it is more closely aligned with what the variables actually hold (as for event tag, there's a similar case to be made 😢 )

nikita-tkachenko-datadog · 2026-02-18T15:54:58Z

...-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RunNTimes.java

        lastExecution,
-        lastExecution && successfulExecutionsSeen == 0,
-        succeededAllRetries,
+        lastExecution && failedExecutionsSeen == executions,


Looks like a skipped test will be considered as "failed all retries" with this update, is this intended?

If a test is skipped, lastExecution == true but failedExecutionsSeen should stay at 0 compared to the executions being incremented to 1, so the check for failedAllRetries should be false. Am I missing anything?

daniel-mohedano added 3 commits February 17, 2026 12:31

feat: implement early exit policy

0043a80

chore: update test fixtures removing additional retries

26dafe3

chore: missed final status updates

7cec3c3

daniel-mohedano added type: enhancement Enhancements and improvements comp: ci visibility Continuous Integration Visibility labels Feb 17, 2026

daniel-mohedano added 2 commits February 18, 2026 11:02

fix: only tag "test.has_failed_all_retries" if test retried > 0

c4bbae6

chore: lint

8350acb

daniel-mohedano marked this pull request as ready for review February 18, 2026 14:12

daniel-mohedano requested review from a team as code owners February 18, 2026 14:12

anmarchenko approved these changes Feb 18, 2026

View reviewed changes

nikita-tkachenko-datadog approved these changes Feb 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement early exit for retry mechanisms#10604

Implement early exit for retry mechanisms#10604
daniel-mohedano wants to merge 5 commits intomasterfrom
daniel.mohedano/retry-early-exit

daniel-mohedano commented Feb 17, 2026 •

edited by atlassian bot

Loading

Uh oh!

pr-commenter bot commented Feb 17, 2026 •

edited

Loading

Uh oh!

nikita-tkachenko-datadog Feb 18, 2026

Uh oh!

nikita-tkachenko-datadog Feb 18, 2026

Uh oh!

daniel-mohedano Feb 18, 2026

Uh oh!

nikita-tkachenko-datadog Feb 18, 2026

Uh oh!

daniel-mohedano Feb 18, 2026

Uh oh!

daniel-mohedano Feb 18, 2026

Uh oh!

nikita-tkachenko-datadog Feb 18, 2026

Uh oh!

daniel-mohedano Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

daniel-mohedano commented Feb 17, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

daniel-mohedano commented Feb 17, 2026 •

edited by atlassian bot

Loading

pr-commenter bot commented Feb 17, 2026 •

edited

Loading