[Perf] Windows/x64: 20 Regressions on 7/12/2023 7:15:00 AM #89103
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | 83bf4b66ed3de5905b2e410a194a0f5f3d924c3d |
| Compare | 82bf906968d9bc1ca5b3131c7a7af8f30d5ac2f5 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Tests.Perf_Int32
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'Details
Payloads
System.Tests.Perf_Int32.TryParseSpan(value: "12345")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.190120288945042 > 11.534074752675743.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -24.697939964773763 (T) = (0 -14.407504759071363) / Math.Sqrt((0.000210871079473183 / (19)) + (0.1920708851182739 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.31172436769823975 = (10.98363735085067 - 14.407504759071363) / 10.98363735085067 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.ToString(value: 2147483647)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.993700539555588 > 12.368893105520494.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM, 7/12/2023 9:48:15 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -27.260590010041422 (T) = (0 -15.496795506087762) / Math.Sqrt((0.18115007417294351 / (22)) + (0.08147438563616338 / (7))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (7) - 2, .025) and -0.3297635832937344 = (11.653797487597945 - 15.496795506087762) / 11.653797487597945 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.TryParse(value: "-2147483648")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.396802256987975 > 16.3567919600678.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -26.031347472653543 (T) = (0 -19.70150746050102) / Math.Sqrt((0.011863208373597493 / (19)) + (0.23923420674679818 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.2610591640305737 = (15.622984252008788 - 19.70150746050102) / 15.622984252008788 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.TryParseSpan(value: "-2147483648")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.16131198673049 > 17.506325757575464.
IsChangePoint: Marked as a change because one of 4/26/2023 6:27:29 AM, 4/26/2023 4:28:25 PM, 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -23.009844760978734 (T) = (0 -19.84466188881448) / Math.Sqrt((0.14091681560057817 / (19)) + (0.14438831076899844 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.20687769829617622 = (16.44297671324146 - 19.84466188881448) / 16.44297671324146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.Parse(value: "2147483647")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.329416854116946 > 15.091691023660475.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -16.312934516703194 (T) = (0 -18.19734938679178) / Math.Sqrt((0.0015169596589323792 / (19)) + (0.548154216803538 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.2658786494521896 = (14.375271590737947 - 18.19734938679178) / 14.375271590737947 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.ToString(value: 12345)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.90771421298525 > 9.010497568013985.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 6/22/2023 7:55:14 AM, 7/12/2023 9:48:15 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -21.22220608159406 (T) = (0 -12.28222343184322) / Math.Sqrt((0.07844561528440665 / (22)) + (0.1741739647391373 / (7))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (7) - 2, .025) and -0.4112970353405608 = (8.702791208570343 - 12.28222343184322) / 8.702791208570343 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.ParseSpan(value: "12345")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.66542346740148 > 12.211811377879068.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -25.408789976545123 (T) = (0 -15.937928289814147) / Math.Sqrt((1.1162861616165135E-05 / (19)) + (0.28747723161934796 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.3704405032137533 = (11.629784914003116 - 15.937928289814147) / 11.629784914003116 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.Parse(value: "12345")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.826361980205 > 11.649755632515522.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -16.09636804454305 (T) = (0 -15.181679741034051) / Math.Sqrt((0.0041244532777064075 / (19)) + (0.5743099533001782 / (9))) is less than -2.055529438636913 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (9) - 2, .025) and -0.36665350673470265 = (11.108653119624378 - 15.181679741034051) / 11.108653119624378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.TryParse(value: "2147483647")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.77535313009241 > 15.093973076160415.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -50.573425619394655 (T) = (0 -19.0281170471599) / Math.Sqrt((0.0034886385822024257 / (19)) + (0.08235326096061854 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.3225208125873535 = (14.387763781148871 - 19.0281170471599) / 14.387763781148871 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int32.TryParseSpan(value: "2147483647")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.680236497246323 > 14.97101717214047.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -71.27114399575764 (T) = (0 -19.312547634887263) / Math.Sqrt((0.0017930930556328644 / (19)) + (0.049113813075204606 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.3533653853715328 = (14.270017427396729 - 19.312547634887263) / 14.270017427396729 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | 83bf4b66ed3de5905b2e410a194a0f5f3d924c3d |
| Compare | 82bf906968d9bc1ca5b3131c7a7af8f30d5ac2f5 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Memory.Span<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| EndsWith - Duration of single invocation 📝 - Benchmark Source |
19.98 ns | 23.93 ns | 1.20 | 0.11 | False | |||
| IndexOfAnyFiveValues - Duration of single invocation 📝 - Benchmark Source |
32.27 ns | 43.41 ns | 1.35 | 0.18 | False | |||
| StartsWith - Duration of single invocation 📝 - Benchmark Source |
17.61 ns | 23.07 ns | 1.31 | 0.17 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Int32>*'Details
Payloads
System.Memory.Span<Int32>.EndsWith(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.928167560437533 > 20.999560282601973.
IsChangePoint: Marked as a change because one of 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/8/2023 8:20:44 PM, 6/22/2023 7:55:14 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -8.54359133045494 (T) = (0 -22.311792180136468) / Math.Sqrt((0.22509452045072115 / (19)) + (0.6775623704422623 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.12112244848097427 = (19.901298212668074 - 22.311792180136468) / 19.901298212668074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Memory.Span<Int32>.IndexOfAnyFiveValues(Size: 33)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 43.41256488683741 > 33.98140889328459.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/13/2023 5:43:42 PM, 6/19/2023 6:37:24 PM, 7/5/2023 7:07:19 PM, 7/12/2023 9:48:15 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -59.5690257957098 (T) = (0 -42.79789455195748) / Math.Sqrt((0.1796580122748577 / (22)) + (0.15216707898017826 / (7))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (7) - 2, .025) and -0.31699264779164527 = (32.49668449077653 - 42.79789455195748) / 32.49668449077653 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Memory.Span<Int32>.StartsWith(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.071253447888303 > 18.3165331355214.
IsChangePoint: Marked as a change because one of 5/16/2023 5:40:18 PM, 6/22/2023 7:55:14 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -7.109646695955824 (T) = (0 -22.868928049019054) / Math.Sqrt((0.1943529829533172 / (19)) + (5.747053214192557 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.3119387828064188 = (17.43139874262978 - 22.868928049019054) / 17.43139874262978 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | 83bf4b66ed3de5905b2e410a194a0f5f3d924c3d |
| Compare | 82bf906968d9bc1ca5b3131c7a7af8f30d5ac2f5 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Text.Json.Tests.Perf_Get
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| GetSingle - Duration of single invocation 📝 - Benchmark Source |
4.53 μs | 4.82 μs | 1.06 | 0.02 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'Details
Payloads
System.Text.Json.Tests.Perf_Get.GetSingle
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.819585599387208 > 4.758078055162659.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -12.816471219622851 (T) = (0 -4857.4633481907085) / Math.Sqrt((1694.7576055906093 / (19)) + (4334.554670054745 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.06419276112964237 = (4564.458174884128 - 4857.4633481907085) / 4564.458174884128 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | 83bf4b66ed3de5905b2e410a194a0f5f3d924c3d |
| Compare | 82bf906968d9bc1ca5b3131c7a7af8f30d5ac2f5 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Collections.ContainsFalse<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| List - Duration of single invocation 📝 - Benchmark Source |
16.44 μs | 20.26 μs | 1.23 | 0.10 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse<Int32>*'Details
Payloads
System.Collections.ContainsFalse<Int32>.List(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.259371761658027 > 17.259348547581492.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/13/2023 11:57:38 AM, 6/19/2023 11:51:49 PM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -202.4740960409093 (T) = (0 -20369.29188027515) / Math.Sqrt((94.10078871264959 / (19)) + (3726.473764407471 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.2393978850339125 = (16434.828658528655 - 20369.29188027515) / 16434.828658528655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | 83bf4b66ed3de5905b2e410a194a0f5f3d924c3d |
| Compare | 82bf906968d9bc1ca5b3131c7a7af8f30d5ac2f5 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Collections.ContainsTrue<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| Queue - Duration of single invocation 📝 - Benchmark Source |
13.20 μs | 16.68 μs | 1.26 | 0.18 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue<Int32>*'Details
Payloads
System.Collections.ContainsTrue<Int32>.Queue(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.682858635716826 > 13.861466273593738.
IsChangePoint: Marked as a change because one of 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/14/2023 4:56:58 PM, 6/27/2023 10:31:09 PM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -2.232653896944129 (T) = (0 -15998.855719533469) / Math.Sqrt((3352948.64773084 / (19)) + (3292236.2439783234 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.110170783211233 = (14411.166247102861 - 15998.855719533469) / 14411.166247102861 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | 83bf4b66ed3de5905b2e410a194a0f5f3d924c3d |
| Compare | 82bf906968d9bc1ca5b3131c7a7af8f30d5ac2f5 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Tests.Perf_Int64
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| ParseSpan - Duration of single invocation 📝 - Benchmark Source |
13.02 ns | 15.32 ns | 1.18 | 0.09 | False | |||
| TryParseSpan - Duration of single invocation 📝 - Benchmark Source |
12.21 ns | 15.12 ns | 1.24 | 0.06 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'Details
Payloads
System.Tests.Perf_Int64.ParseSpan(value: "12345")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.318294192831162 > 13.648609928158619.
IsChangePoint: Marked as a change because one of 4/19/2023 8:38:27 PM, 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -12.295197589970563 (T) = (0 -15.048104182117617) / Math.Sqrt((0.005490628588379624 / (19)) + (0.269115252900472 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.15574090629500364 = (13.020309396470024 - 15.048104182117617) / 13.020309396470024 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_Int64.TryParseSpan(value: "12345")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.117739991647596 > 12.997921566357508.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -30.475514094506334 (T) = (0 -15.025658001356609) / Math.Sqrt((0.0063871462073339935 / (19)) + (0.07486717501870088 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.21860734658284697 = (12.33018826243806 - 15.025658001356609) / 12.33018826243806 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | 83bf4b66ed3de5905b2e410a194a0f5f3d924c3d |
| Compare | 82bf906968d9bc1ca5b3131c7a7af8f30d5ac2f5 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Tests.Perf_SByte
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| TryParse - Duration of single invocation 📝 - Benchmark Source |
11.93 ns | 14.20 ns | 1.19 | 0.09 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'Details
Payloads
System.Tests.Perf_SByte.TryParse(value: "127")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.197530899075202 > 12.56178043363791.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 5/13/2023 1:49:08 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -16.861707861582495 (T) = (0 -14.232159728978075) / Math.Sqrt((0.00859080248235329 / (19)) + (0.16584197045771878 / (10))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (10) - 2, .025) and -0.18292654724110605 = (12.031313154794939 - 14.232159728978075) / 12.031313154794939 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | 83bf4b66ed3de5905b2e410a194a0f5f3d924c3d |
| Compare | 82bf906968d9bc1ca5b3131c7a7af8f30d5ac2f5 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in Benchstone.BenchI.BenchE
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| Test - Duration of single invocation 📝 - Benchmark Source |
245.07 ms | 320.11 ms | 1.31 | 0.05 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.BenchE*'Details
Payloads
Benchstone.BenchI.BenchE.Test
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 320.1089333333333 > 257.3510205.
IsChangePoint: Marked as a change because one of 4/25/2023 3:46:39 AM, 5/2/2023 3:18:41 PM, 5/9/2023 1:22:51 PM, 5/16/2023 12:30:38 AM, 5/26/2023 6:22:05 AM, 6/5/2023 10:43:22 AM, 6/9/2023 2:21:24 AM, 6/13/2023 11:57:38 AM, 6/20/2023 9:17:03 AM, 6/30/2023 8:34:57 AM, 7/12/2023 2:12:47 AM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -150.1092793209235 (T) = (0 -322493599.23280424) / Math.Sqrt((9818554358.852018 / (19)) + (2390667044638.784 / (9))) is less than -2.055529438636913 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (9) - 2, .025) and -0.3160149151726372 = (245053149.10545596 - 322493599.23280424) / 245053149.10545596 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository