Add telemetry for LLMObs manual api span finish calls#10616
Add telemetry for LLMObs manual api span finish calls#10616
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~e566c2e7fd, baseline=1.60.0-SNAPSHOT~16f36474a2
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.067 s) : 0, 1067496
Total [baseline] (10.902 s) : 0, 10901735
Agent [candidate] (1.067 s) : 0, 1066581
Total [candidate] (10.822 s) : 0, 10822075
section appsec
Agent [baseline] (1.244 s) : 0, 1244263
Total [baseline] (11.075 s) : 0, 11074764
Agent [candidate] (1.24 s) : 0, 1239629
Total [candidate] (11.132 s) : 0, 11132302
section iast
Agent [baseline] (1.246 s) : 0, 1245831
Total [baseline] (11.231 s) : 0, 11230848
Agent [candidate] (1.233 s) : 0, 1232763
Total [candidate] (11.183 s) : 0, 11183101
section profiling
Agent [baseline] (1.196 s) : 0, 1196079
Total [baseline] (10.966 s) : 0, 10966375
Agent [candidate] (1.206 s) : 0, 1205834
Total [candidate] (10.988 s) : 0, 10987611
gantt
title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~e566c2e7fd, baseline=1.60.0-SNAPSHOT~16f36474a2
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.191 ms) : 0, 1191
crashtracking [candidate] (1.18 ms) : 0, 1180
BytebuddyAgent [baseline] (629.797 ms) : 0, 629797
BytebuddyAgent [candidate] (629.186 ms) : 0, 629186
AgentMeter [baseline] (28.923 ms) : 0, 28923
AgentMeter [candidate] (28.938 ms) : 0, 28938
GlobalTracer [baseline] (257.646 ms) : 0, 257646
GlobalTracer [candidate] (257.714 ms) : 0, 257714
AppSec [baseline] (33.084 ms) : 0, 33084
AppSec [candidate] (32.87 ms) : 0, 32870
Debugger [baseline] (63.271 ms) : 0, 63271
Debugger [candidate] (61.673 ms) : 0, 61673
Remote Config [baseline] (632.339 µs) : 0, 632
Remote Config [candidate] (623.501 µs) : 0, 624
Telemetry [baseline] (13.057 ms) : 0, 13057
Telemetry [candidate] (13.738 ms) : 0, 13738
Flare Poller [baseline] (4.564 ms) : 0, 4564
Flare Poller [candidate] (5.381 ms) : 0, 5381
section appsec
crashtracking [baseline] (1.202 ms) : 0, 1202
crashtracking [candidate] (1.187 ms) : 0, 1187
BytebuddyAgent [baseline] (659.541 ms) : 0, 659541
BytebuddyAgent [candidate] (658.22 ms) : 0, 658220
AgentMeter [baseline] (12.112 ms) : 0, 12112
AgentMeter [candidate] (11.972 ms) : 0, 11972
GlobalTracer [baseline] (259.86 ms) : 0, 259860
GlobalTracer [candidate] (258.336 ms) : 0, 258336
IAST [baseline] (25.511 ms) : 0, 25511
IAST [candidate] (25.164 ms) : 0, 25164
AppSec [baseline] (168.78 ms) : 0, 168780
AppSec [candidate] (167.949 ms) : 0, 167949
Debugger [baseline] (67.78 ms) : 0, 67780
Debugger [candidate] (67.241 ms) : 0, 67241
Remote Config [baseline] (658.141 µs) : 0, 658
Remote Config [candidate] (648.106 µs) : 0, 648
Telemetry [baseline] (9.72 ms) : 0, 9720
Telemetry [candidate] (9.803 ms) : 0, 9803
Flare Poller [baseline] (3.742 ms) : 0, 3742
Flare Poller [candidate] (3.881 ms) : 0, 3881
section iast
crashtracking [baseline] (1.186 ms) : 0, 1186
crashtracking [candidate] (1.171 ms) : 0, 1171
BytebuddyAgent [baseline] (804.789 ms) : 0, 804789
BytebuddyAgent [candidate] (795.743 ms) : 0, 795743
AgentMeter [baseline] (11.571 ms) : 0, 11571
AgentMeter [candidate] (11.291 ms) : 0, 11291
GlobalTracer [baseline] (250.357 ms) : 0, 250357
GlobalTracer [candidate] (248.307 ms) : 0, 248307
IAST [baseline] (27.509 ms) : 0, 27509
IAST [candidate] (27.299 ms) : 0, 27299
AppSec [baseline] (31.702 ms) : 0, 31702
AppSec [candidate] (34.732 ms) : 0, 34732
Debugger [baseline] (70.387 ms) : 0, 70387
Debugger [candidate] (66.244 ms) : 0, 66244
Remote Config [baseline] (558.05 µs) : 0, 558
Remote Config [candidate] (537.854 µs) : 0, 538
Telemetry [baseline] (8.75 ms) : 0, 8750
Telemetry [candidate] (8.683 ms) : 0, 8683
Flare Poller [baseline] (3.581 ms) : 0, 3581
Flare Poller [candidate] (3.499 ms) : 0, 3499
section profiling
crashtracking [baseline] (1.225 ms) : 0, 1225
crashtracking [candidate] (1.232 ms) : 0, 1232
BytebuddyAgent [baseline] (686.374 ms) : 0, 686374
BytebuddyAgent [candidate] (692.477 ms) : 0, 692477
AgentMeter [baseline] (8.653 ms) : 0, 8653
AgentMeter [candidate] (8.734 ms) : 0, 8734
GlobalTracer [baseline] (217.1 ms) : 0, 217100
GlobalTracer [candidate] (218.609 ms) : 0, 218609
AppSec [baseline] (32.804 ms) : 0, 32804
AppSec [candidate] (33.205 ms) : 0, 33205
Debugger [baseline] (67.55 ms) : 0, 67550
Debugger [candidate] (68.134 ms) : 0, 68134
Remote Config [baseline] (620.188 µs) : 0, 620
Remote Config [candidate] (627.339 µs) : 0, 627
Telemetry [baseline] (8.877 ms) : 0, 8877
Telemetry [candidate] (8.944 ms) : 0, 8944
Flare Poller [baseline] (3.781 ms) : 0, 3781
Flare Poller [candidate] (3.772 ms) : 0, 3772
ProfilingAgent [baseline] (98.783 ms) : 0, 98783
ProfilingAgent [candidate] (99.972 ms) : 0, 99972
Profiling [baseline] (99.368 ms) : 0, 99368
Profiling [candidate] (100.548 ms) : 0, 100548
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~e566c2e7fd, baseline=1.60.0-SNAPSHOT~16f36474a2
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1063812
Total [baseline] (8.777 s) : 0, 8777327
Agent [candidate] (1.067 s) : 0, 1066922
Total [candidate] (8.75 s) : 0, 8750115
section iast
Agent [baseline] (1.248 s) : 0, 1247795
Total [baseline] (9.397 s) : 0, 9397380
Agent [candidate] (1.237 s) : 0, 1237242
Total [candidate] (9.368 s) : 0, 9368271
gantt
title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~e566c2e7fd, baseline=1.60.0-SNAPSHOT~16f36474a2
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.186 ms) : 0, 1186
crashtracking [candidate] (1.198 ms) : 0, 1198
BytebuddyAgent [baseline] (628.221 ms) : 0, 628221
BytebuddyAgent [candidate] (630.038 ms) : 0, 630038
AgentMeter [baseline] (28.902 ms) : 0, 28902
AgentMeter [candidate] (29.028 ms) : 0, 29028
GlobalTracer [baseline] (257.273 ms) : 0, 257273
GlobalTracer [candidate] (258.068 ms) : 0, 258068
AppSec [baseline] (32.761 ms) : 0, 32761
AppSec [candidate] (32.559 ms) : 0, 32559
Debugger [baseline] (61.252 ms) : 0, 61252
Debugger [candidate] (62.321 ms) : 0, 62321
Remote Config [baseline] (620.024 µs) : 0, 620
Remote Config [candidate] (614.178 µs) : 0, 614
Telemetry [baseline] (13.698 ms) : 0, 13698
Telemetry [candidate] (12.194 ms) : 0, 12194
Flare Poller [baseline] (4.55 ms) : 0, 4550
Flare Poller [candidate] (5.347 ms) : 0, 5347
section iast
crashtracking [baseline] (1.196 ms) : 0, 1196
crashtracking [candidate] (1.199 ms) : 0, 1199
BytebuddyAgent [baseline] (807.376 ms) : 0, 807376
BytebuddyAgent [candidate] (802.094 ms) : 0, 802094
AgentMeter [baseline] (11.706 ms) : 0, 11706
AgentMeter [candidate] (11.478 ms) : 0, 11478
GlobalTracer [baseline] (250.612 ms) : 0, 250612
GlobalTracer [candidate] (248.474 ms) : 0, 248474
IAST [baseline] (27.457 ms) : 0, 27457
IAST [candidate] (26.955 ms) : 0, 26955
AppSec [baseline] (34.403 ms) : 0, 34403
AppSec [candidate] (33.837 ms) : 0, 33837
Debugger [baseline] (66.699 ms) : 0, 66699
Debugger [candidate] (65.436 ms) : 0, 65436
Remote Config [baseline] (548.661 µs) : 0, 549
Remote Config [candidate] (531.086 µs) : 0, 531
Telemetry [baseline] (8.712 ms) : 0, 8712
Telemetry [candidate] (8.525 ms) : 0, 8525
Flare Poller [baseline] (3.502 ms) : 0, 3502
Flare Poller [candidate] (3.437 ms) : 0, 3437
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 18 metrics, 18 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~e566c2e7fd, baseline=1.60.0-SNAPSHOT~16f36474a2
dateFormat X
axisFormat %s
section baseline
no_agent (1.163 ms) : 1152, 1175
. : milestone, 1163,
iast (3.062 ms) : 3022, 3102
. : milestone, 3062,
iast_FULL (5.734 ms) : 5677, 5790
. : milestone, 5734,
iast_GLOBAL (3.574 ms) : 3522, 3627
. : milestone, 3574,
profiling (2.062 ms) : 2044, 2081
. : milestone, 2062,
tracing (1.77 ms) : 1755, 1784
. : milestone, 1770,
section candidate
no_agent (1.178 ms) : 1166, 1189
. : milestone, 1178,
iast (3.142 ms) : 3100, 3183
. : milestone, 3142,
iast_FULL (5.727 ms) : 5669, 5784
. : milestone, 5727,
iast_GLOBAL (3.623 ms) : 3561, 3685
. : milestone, 3623,
profiling (2.035 ms) : 2017, 2052
. : milestone, 2035,
tracing (1.803 ms) : 1786, 1819
. : milestone, 1803,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~e566c2e7fd, baseline=1.60.0-SNAPSHOT~16f36474a2
dateFormat X
axisFormat %s
section baseline
no_agent (18.343 ms) : 18154, 18532
. : milestone, 18343,
appsec (19.238 ms) : 19043, 19433
. : milestone, 19238,
code_origins (17.721 ms) : 17545, 17896
. : milestone, 17721,
iast (17.688 ms) : 17510, 17865
. : milestone, 17688,
profiling (18.529 ms) : 18346, 18712
. : milestone, 18529,
tracing (17.528 ms) : 17355, 17701
. : milestone, 17528,
section candidate
no_agent (18.253 ms) : 18068, 18437
. : milestone, 18253,
appsec (19.158 ms) : 18965, 19352
. : milestone, 19158,
code_origins (18.042 ms) : 17864, 18219
. : milestone, 18042,
iast (17.606 ms) : 17428, 17784
. : milestone, 17606,
profiling (18.794 ms) : 18602, 18986
. : milestone, 18794,
tracing (17.856 ms) : 17682, 18030
. : milestone, 17856,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~e566c2e7fd, baseline=1.60.0-SNAPSHOT~16f36474a2
dateFormat X
axisFormat %s
section baseline
no_agent (1.469 ms) : 1458, 1481
. : milestone, 1469,
appsec (3.801 ms) : 3578, 4024
. : milestone, 3801,
iast (2.239 ms) : 2170, 2309
. : milestone, 2239,
iast_GLOBAL (2.298 ms) : 2228, 2368
. : milestone, 2298,
profiling (2.068 ms) : 2013, 2123
. : milestone, 2068,
tracing (2.049 ms) : 1996, 2103
. : milestone, 2049,
section candidate
no_agent (1.469 ms) : 1457, 1480
. : milestone, 1469,
appsec (3.776 ms) : 3557, 3996
. : milestone, 3776,
iast (2.24 ms) : 2171, 2309
. : milestone, 2240,
iast_GLOBAL (2.286 ms) : 2216, 2356
. : milestone, 2286,
profiling (2.511 ms) : 2345, 2678
. : milestone, 2511,
tracing (2.058 ms) : 2004, 2111
. : milestone, 2058,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~e566c2e7fd, baseline=1.60.0-SNAPSHOT~16f36474a2
dateFormat X
axisFormat %s
section baseline
no_agent (15.134 s) : 15134000, 15134000
. : milestone, 15134000,
appsec (15.057 s) : 15057000, 15057000
. : milestone, 15057000,
iast (18.155 s) : 18155000, 18155000
. : milestone, 18155000,
iast_GLOBAL (17.925 s) : 17925000, 17925000
. : milestone, 17925000,
profiling (14.961 s) : 14961000, 14961000
. : milestone, 14961000,
tracing (14.759 s) : 14759000, 14759000
. : milestone, 14759000,
section candidate
no_agent (15.401 s) : 15401000, 15401000
. : milestone, 15401000,
appsec (14.505 s) : 14505000, 14505000
. : milestone, 14505000,
iast (18.301 s) : 18301000, 18301000
. : milestone, 18301000,
iast_GLOBAL (17.941 s) : 17941000, 17941000
. : milestone, 17941000,
profiling (14.898 s) : 14898000, 14898000
. : milestone, 14898000,
tracing (14.825 s) : 14825000, 14825000
. : milestone, 14825000,
|
8c9f300 to
6fb2af0
Compare
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
6fb2af0 to
8772fec
Compare
PerfectSlayer
left a comment
There was a problem hiding this comment.
Looking good
❔ question: Any rationale about removing the this. qualifier? I usually found it useful during review to distinguish between vars and local fields.
🎯 suggestion: What about testing for non root span finish too?
Thanks for the quick review! The main rationale is to make things less verbose. Most of the time the span field is accessed only to call a method resulting in two "." selectors being used. Thanks! I added the non-root span test case. |
| m.metricName == 'span.finished' | ||
| m.type == 'count' | ||
| m.value == 1 | ||
| m.tags.contains('integration:llmobs') |
There was a problem hiding this comment.
this tag should actually be tagged with the upstream instrumentation library the span comes from (like llm spans from openai would be tagged with integration:openai)
| m.tags.contains('integration:llmobs') | ||
| m.tags.contains('span_kind:workflow') | ||
| m.tags.contains('autoinstrumented:0') | ||
| m.tags.contains('is_root_span:1') |
There was a problem hiding this comment.
we should also try to have ml_app on the metrics
What Does This Do
Add telemetry for manual api span finish calls
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.