chore(ci) : add final_status property on junit XML [APMSP-2610]#10741
Draft
cbeauchesne wants to merge 3 commits intomasterfrom
Draft
chore(ci) : add final_status property on junit XML [APMSP-2610]#10741cbeauchesne wants to merge 3 commits intomasterfrom
cbeauchesne wants to merge 3 commits intomasterfrom
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 66 metrics, 5 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~7a90a564cf, baseline=1.61.0-SNAPSHOT~70410da0e2
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.071 s) : 0, 1071253
Total [baseline] (8.854 s) : 0, 8853601
Agent [candidate] (1.058 s) : 0, 1057560
Total [candidate] (8.822 s) : 0, 8821742
section iast
Agent [baseline] (1.224 s) : 0, 1224235
Total [baseline] (9.526 s) : 0, 9525850
Agent [candidate] (1.228 s) : 0, 1227869
Total [candidate] (9.574 s) : 0, 9573712
gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~7a90a564cf, baseline=1.61.0-SNAPSHOT~70410da0e2
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.209 ms) : 0, 1209
crashtracking [candidate] (1.213 ms) : 0, 1213
BytebuddyAgent [baseline] (636.691 ms) : 0, 636691
BytebuddyAgent [candidate] (629.255 ms) : 0, 629255
AgentMeter [baseline] (29.631 ms) : 0, 29631
AgentMeter [candidate] (29.038 ms) : 0, 29038
GlobalTracer [baseline] (259.315 ms) : 0, 259315
GlobalTracer [candidate] (257.15 ms) : 0, 257150
AppSec [baseline] (31.947 ms) : 0, 31947
AppSec [candidate] (31.487 ms) : 0, 31487
Debugger [baseline] (59.327 ms) : 0, 59327
Debugger [candidate] (58.53 ms) : 0, 58530
Remote Config [baseline] (601.883 µs) : 0, 602
Remote Config [candidate] (581.836 µs) : 0, 582
Telemetry [baseline] (8.778 ms) : 0, 8778
Telemetry [candidate] (8.626 ms) : 0, 8626
Flare Poller [baseline] (7.447 ms) : 0, 7447
Flare Poller [candidate] (5.7 ms) : 0, 5700
section iast
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (795.203 ms) : 0, 795203
BytebuddyAgent [candidate] (797.569 ms) : 0, 797569
AgentMeter [baseline] (11.286 ms) : 0, 11286
AgentMeter [candidate] (11.294 ms) : 0, 11294
GlobalTracer [baseline] (246.47 ms) : 0, 246470
GlobalTracer [candidate] (247.997 ms) : 0, 247997
IAST [baseline] (25.102 ms) : 0, 25102
IAST [candidate] (25.169 ms) : 0, 25169
AppSec [baseline] (26.296 ms) : 0, 26296
AppSec [candidate] (26.35 ms) : 0, 26350
Debugger [baseline] (62.44 ms) : 0, 62440
Debugger [candidate] (62.104 ms) : 0, 62104
Remote Config [baseline] (530.887 µs) : 0, 531
Remote Config [candidate] (523.692 µs) : 0, 524
Telemetry [baseline] (14.795 ms) : 0, 14795
Telemetry [candidate] (14.76 ms) : 0, 14760
Flare Poller [baseline] (4.894 ms) : 0, 4894
Flare Poller [candidate] (4.869 ms) : 0, 4869
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~7a90a564cf, baseline=1.61.0-SNAPSHOT~70410da0e2
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1056114
Total [baseline] (11.088 s) : 0, 11088310
Agent [candidate] (1.056 s) : 0, 1055867
Total [candidate] (10.985 s) : 0, 10984613
section appsec
Agent [baseline] (1.248 s) : 0, 1248198
Total [baseline] (11.227 s) : 0, 11227306
Agent [candidate] (1.251 s) : 0, 1250506
Total [candidate] (11.101 s) : 0, 11101413
section iast
Agent [baseline] (1.227 s) : 0, 1227250
Total [baseline] (11.276 s) : 0, 11276225
Agent [candidate] (1.228 s) : 0, 1227852
Total [candidate] (11.357 s) : 0, 11357070
section profiling
Agent [baseline] (1.183 s) : 0, 1183289
Total [baseline] (11.084 s) : 0, 11083591
Agent [candidate] (1.18 s) : 0, 1179782
Total [candidate] (10.968 s) : 0, 10967684
gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~7a90a564cf, baseline=1.61.0-SNAPSHOT~70410da0e2
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.183 ms) : 0, 1183
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (626.586 ms) : 0, 626586
BytebuddyAgent [candidate] (627.171 ms) : 0, 627171
AgentMeter [baseline] (29.068 ms) : 0, 29068
AgentMeter [candidate] (29.013 ms) : 0, 29013
GlobalTracer [baseline] (256.401 ms) : 0, 256401
GlobalTracer [candidate] (256.632 ms) : 0, 256632
AppSec [baseline] (31.402 ms) : 0, 31402
AppSec [candidate] (31.259 ms) : 0, 31259
Debugger [baseline] (59.19 ms) : 0, 59190
Debugger [candidate] (59.092 ms) : 0, 59092
Remote Config [baseline] (592.26 µs) : 0, 592
Remote Config [candidate] (586.947 µs) : 0, 587
Telemetry [baseline] (8.633 ms) : 0, 8633
Telemetry [candidate] (8.594 ms) : 0, 8594
Flare Poller [baseline] (7.14 ms) : 0, 7140
Flare Poller [candidate] (6.377 ms) : 0, 6377
section appsec
crashtracking [baseline] (1.198 ms) : 0, 1198
crashtracking [candidate] (1.202 ms) : 0, 1202
BytebuddyAgent [baseline] (658.353 ms) : 0, 658353
BytebuddyAgent [candidate] (661.253 ms) : 0, 661253
AgentMeter [baseline] (12.042 ms) : 0, 12042
AgentMeter [candidate] (12.063 ms) : 0, 12063
GlobalTracer [baseline] (259.335 ms) : 0, 259335
GlobalTracer [candidate] (259.009 ms) : 0, 259009
AppSec [baseline] (177.754 ms) : 0, 177754
AppSec [candidate] (177.801 ms) : 0, 177801
Debugger [baseline] (65.108 ms) : 0, 65108
Debugger [candidate] (65.64 ms) : 0, 65640
Remote Config [baseline] (575.178 µs) : 0, 575
Remote Config [candidate] (579.859 µs) : 0, 580
Telemetry [baseline] (9.856 ms) : 0, 9856
Telemetry [candidate] (9.014 ms) : 0, 9014
Flare Poller [baseline] (3.635 ms) : 0, 3635
Flare Poller [candidate] (3.577 ms) : 0, 3577
IAST [baseline] (24.11 ms) : 0, 24110
IAST [candidate] (24.084 ms) : 0, 24084
section iast
crashtracking [baseline] (1.186 ms) : 0, 1186
crashtracking [candidate] (1.193 ms) : 0, 1193
BytebuddyAgent [baseline] (796.097 ms) : 0, 796097
BytebuddyAgent [candidate] (796.404 ms) : 0, 796404
AgentMeter [baseline] (11.332 ms) : 0, 11332
AgentMeter [candidate] (11.325 ms) : 0, 11325
GlobalTracer [baseline] (247.313 ms) : 0, 247313
GlobalTracer [candidate] (247.662 ms) : 0, 247662
AppSec [baseline] (26.32 ms) : 0, 26320
AppSec [candidate] (26.347 ms) : 0, 26347
Debugger [baseline] (63.562 ms) : 0, 63562
Debugger [candidate] (63.41 ms) : 0, 63410
Remote Config [baseline] (525.169 µs) : 0, 525
Remote Config [candidate] (528.593 µs) : 0, 529
Telemetry [baseline] (14.905 ms) : 0, 14905
Telemetry [candidate] (14.878 ms) : 0, 14878
Flare Poller [baseline] (4.9 ms) : 0, 4900
Flare Poller [candidate] (4.907 ms) : 0, 4907
IAST [baseline] (25.11 ms) : 0, 25110
IAST [candidate] (25.173 ms) : 0, 25173
section profiling
ProfilingAgent [baseline] (94.12 ms) : 0, 94120
ProfilingAgent [candidate] (93.515 ms) : 0, 93515
crashtracking [baseline] (1.174 ms) : 0, 1174
crashtracking [candidate] (1.17 ms) : 0, 1170
BytebuddyAgent [baseline] (683.231 ms) : 0, 683231
BytebuddyAgent [candidate] (681.833 ms) : 0, 681833
AgentMeter [baseline] (8.604 ms) : 0, 8604
AgentMeter [candidate] (8.598 ms) : 0, 8598
GlobalTracer [baseline] (215.706 ms) : 0, 215706
GlobalTracer [candidate] (215.268 ms) : 0, 215268
AppSec [baseline] (31.763 ms) : 0, 31763
AppSec [candidate] (31.784 ms) : 0, 31784
Debugger [baseline] (62.912 ms) : 0, 62912
Debugger [candidate] (63.157 ms) : 0, 63157
Remote Config [baseline] (570.712 µs) : 0, 571
Remote Config [candidate] (572.145 µs) : 0, 572
Telemetry [baseline] (8.918 ms) : 0, 8918
Telemetry [candidate] (9.761 ms) : 0, 9761
Flare Poller [baseline] (5.057 ms) : 0, 5057
Flare Poller [candidate] (3.488 ms) : 0, 3488
Profiling [baseline] (94.689 ms) : 0, 94689
Profiling [candidate] (94.07 ms) : 0, 94070
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 2 performance regressions! Performance is the same for 16 metrics, 18 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~7a90a564cf, baseline=1.61.0-SNAPSHOT~70410da0e2
dateFormat X
axisFormat %s
section baseline
no_agent (1.166 ms) : 1154, 1177
. : milestone, 1166,
iast (3.259 ms) : 3217, 3302
. : milestone, 3259,
iast_FULL (5.843 ms) : 5785, 5902
. : milestone, 5843,
iast_GLOBAL (3.486 ms) : 3432, 3540
. : milestone, 3486,
profiling (2.073 ms) : 2054, 2091
. : milestone, 2073,
tracing (1.785 ms) : 1769, 1800
. : milestone, 1785,
section candidate
no_agent (1.183 ms) : 1171, 1194
. : milestone, 1183,
iast (3.189 ms) : 3146, 3231
. : milestone, 3189,
iast_FULL (5.769 ms) : 5712, 5827
. : milestone, 5769,
iast_GLOBAL (3.618 ms) : 3556, 3681
. : milestone, 3618,
profiling (2.28 ms) : 2259, 2300
. : milestone, 2280,
tracing (1.724 ms) : 1710, 1737
. : milestone, 1724,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~7a90a564cf, baseline=1.61.0-SNAPSHOT~70410da0e2
dateFormat X
axisFormat %s
section baseline
no_agent (18.214 ms) : 18028, 18401
. : milestone, 18214,
appsec (18.657 ms) : 18465, 18850
. : milestone, 18657,
code_origins (17.781 ms) : 17603, 17959
. : milestone, 17781,
iast (17.452 ms) : 17280, 17624
. : milestone, 17452,
profiling (19.611 ms) : 19410, 19812
. : milestone, 19611,
tracing (17.629 ms) : 17451, 17807
. : milestone, 17629,
section candidate
no_agent (17.226 ms) : 17053, 17399
. : milestone, 17226,
appsec (18.576 ms) : 18386, 18765
. : milestone, 18576,
code_origins (17.962 ms) : 17781, 18142
. : milestone, 17962,
iast (17.661 ms) : 17487, 17834
. : milestone, 17661,
profiling (18.947 ms) : 18759, 19136
. : milestone, 18947,
tracing (17.709 ms) : 17533, 17884
. : milestone, 17709,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~7a90a564cf, baseline=1.61.0-SNAPSHOT~70410da0e2
dateFormat X
axisFormat %s
section baseline
no_agent (1.475 ms) : 1463, 1486
. : milestone, 1475,
appsec (3.832 ms) : 3611, 4053
. : milestone, 3832,
iast (2.261 ms) : 2191, 2330
. : milestone, 2261,
iast_GLOBAL (2.309 ms) : 2239, 2379
. : milestone, 2309,
profiling (2.103 ms) : 2047, 2160
. : milestone, 2103,
tracing (2.062 ms) : 2008, 2115
. : milestone, 2062,
section candidate
no_agent (1.473 ms) : 1462, 1485
. : milestone, 1473,
appsec (2.525 ms) : 2470, 2580
. : milestone, 2525,
iast (2.259 ms) : 2190, 2328
. : milestone, 2259,
iast_GLOBAL (2.302 ms) : 2232, 2371
. : milestone, 2302,
profiling (2.126 ms) : 2069, 2183
. : milestone, 2126,
tracing (2.064 ms) : 2010, 2117
. : milestone, 2064,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~7a90a564cf, baseline=1.61.0-SNAPSHOT~70410da0e2
dateFormat X
axisFormat %s
section baseline
no_agent (15.03 s) : 15030000, 15030000
. : milestone, 15030000,
appsec (15.044 s) : 15044000, 15044000
. : milestone, 15044000,
iast (18.141 s) : 18141000, 18141000
. : milestone, 18141000,
iast_GLOBAL (17.682 s) : 17682000, 17682000
. : milestone, 17682000,
profiling (15.061 s) : 15061000, 15061000
. : milestone, 15061000,
tracing (15.095 s) : 15095000, 15095000
. : milestone, 15095000,
section candidate
no_agent (14.914 s) : 14914000, 14914000
. : milestone, 14914000,
appsec (15.08 s) : 15080000, 15080000
. : milestone, 15080000,
iast (18.003 s) : 18003000, 18003000
. : milestone, 18003000,
iast_GLOBAL (17.896 s) : 17896000, 17896000
. : milestone, 17896000,
profiling (14.777 s) : 14777000, 14777000
. : milestone, 14777000,
tracing (15.253 s) : 15253000, 15253000
. : milestone, 15253000,
|
25a04fa to
cf93792
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Add a
<property name="dd_tags[test.final_status]" value="pass" />on every test case reported through junitI'm pushing in favor of having this logic built-in inside Test Optim. Waiting for this, I went for a solution that does not interact at all with the underlying Test Framework, using an XSLT transformation executed before pushing the report.
Adding custom final_status will be done on a later PR, depending on real needs.
Motivation
Some repositories intentionally use mechanisms that ignore test failures (there are valid reasons to still execute those tests behind the scenes).
In these cases, the reported test status cannot be relied upon by Test Optimization to monitor repository health or trigger team notifications.
To ensure a consistent and unified approach, we will instead use final_status — the property specifically designed and used by the Test Optimization integration for this purpose.
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.