BaselineCustomAnalyzer raise warning under specific conditions

Author: filzrevCreated Jan 5, 2026Updated Aug 28, 2026

I’m trying to write benchmarks that measure performance difference between .NET version.

But when using following settings BaselineCustomAnalyzer raise warnings and Ratio/Ratio column show ?.

Set baseline on Job (that running benchmark with .NET 8).
Set baseline on benchmark method (It's used when running benchmark with different config)
Enable grouping with .AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod)

Benchmark results

BaselineCustomAnalyzer Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline or benchmark could not be found, or the baseline value is too close to zero.

markdown

| Method                       | Toolchain | Mean     | Error     | StdDev   | Ratio | RatioSD |
|----------------------------- |---------- |---------:|----------:|---------:|------:|--------:|
| OrderBy_EnumerableDataSource | .NET 8.0  | 153.3 ms |  10.27 ms |  0.56 ms |  1.00 |    0.00 |
| OrderBy_EnumerableDataSource | .NET 9.0  | 149.0 ms | 233.89 ms | 12.82 ms |  0.97 |    0.07 |
| OrderBy_EnumerableDataSource | .NET 10.0 | 140.8 ms | 216.04 ms | 11.84 ms |  0.92 |    0.07 |
|                              |           |          |           |          |       |         |
| OrderBy_ArrayDataSource      | .NET 8.0  | 164.7 ms | 136.52 ms |  7.48 ms |     ? |       ? |
| OrderBy_ArrayDataSource      | .NET 9.0  | 147.1 ms | 223.11 ms | 12.23 ms |     ? |       ? |
| OrderBy_ArrayDataSource      | .NET 10.0 | 148.8 ms |  88.03 ms |  4.83 ms |     ? |       ? |

TestCode

csharp

    internal class Program
    {
        static void Main(string[] args)
        {
            var config = DefaultConfig.Instance
                                      .AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod)
                                      .WithOrderer(new DefaultOrderer(jobOrderPolicy: JobOrderPolicy.Numeric))
                                      .WithOptions(ConfigOptions.DisableOptimizationsValidator);

            config = config.AddJob(
                Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
                Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp90),
                Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp10_0));

            var summaries = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
                                             .Run(args, config)
                                             .ToArray();

            var config2 = summaries[0].BenchmarksCases[0].Config;
        }
    }

    public class Benchmarks
    {
        private readonly Consumer Consumer = new();

        public int N { get; set; } = 1_000_000;

        private readonly IEnumerable<int> EnumerableDataSource;
        private readonly int[] ArrayDataSource;

        public Benchmarks()
        {
            var rand = new Random(0);
            var source = Enumerable.Range(1, N).Select(x => rand.Next());

            EnumerableDataSource = source;
            ArrayDataSource = EnumerableDataSource.ToArray();
        }

        [Benchmark(Baseline = true)]
        public void OrderBy_EnumerableDataSource()
        {
            foreach (var i in EnumerableDataSource.OrderBy(x => x))
            {
                Consumer.Consume(i);
            }
        }

        [Benchmark]
        public void OrderBy_ArrayDataSource()
        {
            foreach (var i in ArrayDataSource.OrderBy(x => x))
            {
                Consumer.Consume(i);
            }
        }
    }

Analysis

It seems be resolved by modifying following logics. https://github.com/dotnet/BenchmarkDotNet/blob/94303f4e456f66aa2d6b6cae4a3c4eb9217ed7e7/src/BenchmarkDotNet/Reports/BaseliningStrategy.cs#L35-L45

By adding following code that enforce to use Job's baseline setting.

csharp

if(useDescriptors && useJobs)
    benchmark.Job.Meta.Baseline

I've not created a PR because I'm not sure the side effects of this changes.

Source: dotnet/BenchmarkDotNet

View original on GitHub View discussion on GitHub