Add perf experiment pipeline and fix v2 connection pool regressions

[mdaigle]

Description

Two related changes: a new perf pipeline for A/B testing AppContext switches, and the first fix for regressions it surfaced in the v2 connection pool.

Perf experiment pipeline

The existing perf pipelines cover two use cases: PR vs main, and main vs a published baseline. In both, any AppContext switches apply to baseline and current alike, so they cannot measure the effect of the switch itself.

This adds a third use case: run the same commit against itself with a perf-sensitive switch on in one variant and off in the other.

• New eng/pipelines/perf/sqlclient-perf-experiment.yml with a switchUnderTest dropdown (default UseConnectionPoolV2).
• run-perf-tests.sh / .ps1 take a general --switch-under-test / -SwitchUnderTest.
• interleave_perf.py applies a per-variant runner-config override.

Two things worth calling out:

• This pipeline never ingests into Kusto. Its two variants are the same commit, so the results are not comparable to the trend data the other pipelines produce and would pollute it.
• Two processes are required. InProcessEmitToolchain in BenchmarkConfig.cs pins benchmarks to the host process, so AppContext switches cannot be varied within a single BenchmarkDotNet run.

The pipeline drops the useManagedSni / useConnectionPoolV2 / useOptimizedAsyncBehaviour parameters the other pipelines expose. ADO boolean parameters always emit a value, so leaving them in would fire the "flag ignored" warning on every run. The runnerconfig.jsonc defaults already match what those parameters defaulted to.

v2 connection pool fast path

Running the above with switchUnderTest: UseConnectionPoolV2 showed large regressions on open/close-heavy benchmarks: OpenAsyncConnection +273%, RapidOpenCloseSingleThreadAsync +98%, RapidFireOpenClose +46-85% with allocation deltas of +110-212%.

Three causes, all on the path taken when the pool already holds a usable connection:

• Every async open dispatched to the thread pool via Task.Run with no inline attempt first. v1 tries a non-blocking acquisition on the caller's thread (allowCreate: false) and only queues a pending request on a miss. The 3.6 us to 13.3 us jump on OpenAsyncConnection is thread pool round-trip latency for trivial work.
• GetInternalConnection creates a timer-backed CancellationTokenSource before it knows whether it will ever wait. That is an allocation plus a TimerQueueTimer registration, which also contends a shared lock at higher parallelism. This is the allocation delta on the uncontended benchmarks above; the pool-stress runners are a separate cause, covered below.
• GetInternalConnection is an async method, so it allocates a Task<DbConnectionInternal> even when it completes synchronously.

TryGetPooledConnectionInline performs the transacted-store and idle-channel lookups that GetInternalConnection begins with and returns null the moment neither can satisfy the request. Both the sync and async entry points try it first, so the common case avoids all three costs at once. It deliberately never calls OpenNewInternalConnection, so no caller thread blocks on network I/O.

Remaining differences after the fix

Latest run: 173 benchmarks, 10% threshold, 3 interleaved confirmation runs. 10 confirmed regressions, 3 unconfirmed, 51 improvements. 38 benchmarks flag on time or allocation. Allocation is reported only and never gates the build (compare_perf.py sets status from meanDeltaPct alone), so of the 31 allocation increases above 10%, 24 are on benchmarks that got faster.

Scoping first. Benchmarks with no pool checkout in the measured body (37) have a max allocation delta of +0.81%, and those amortising one checkout over real work (67) max at +1.41%. Zero above 10% in either group; all 31 are checkout-dominated. ParallelAsyncConnectionRunner parameterises pooling directly: Pooling=False gives +0.01%/+0.04%/+0.11% allocation, Pooling=True on the same concurrencies gives +70.55%/+113.16%/+120.56%.

A. Concurrent pool growth (28 of 38), by design. The flagged ConnectionPoolStressRunner cases plus ParallelAsyncConnectionRunner.OpenConnectionsConcurrently. Both runners ClearAllPools() in [IterationCleanup], so every iteration is a cold burst where connection creation dominates. v1 serialises pool growth behind a Semaphore(1, 1); v2 deliberately does not, so it creates more physical connections, finishes sooner, and allocates more. A physical connection costs the same in both pools (v1 57,275 B, v2 57,701 B), and dividing each benchmark's allocation delta by its Login7Count delta lands on 55.5-59 KB in every shape tested.

ConnectionPoolRampRunner was added to test that claim rather than argue it. It keeps the cold pool but makes every caller hold its connection until all have connected, so both pools must create the same number of connections and the only remaining variable is how fast they get there:

	time Δ	alloc Δ
ColdStartRamp P=10 / 25 / 50	−75.23% / −80.62% / −82.20%	+0.08% / +0.58% / +0.64%
ColdStartRampAsync P=10 / 25 / 50	−73.42% / −80.93% / −82.89%	−2.62% / −0.17% / +0.80%

Holding connection count constant makes the allocation delta vanish and leaves v2 4-6x faster. That is the same behaviour RapidFireOpenClose scores as a 33-75% regression, because it holds connections for zero time and so measures a burst that never needed the extra connections.

B. Threadpool saturation on the sync wait path (2 of 38). SteadyStateOpenQueryClose P=50/Max=10 at +136.27%. A sync waiter blocks in mres.Wait(), and with AllowSynchronousContinuations off the returning thread must queue the wake into a pool whose workers are all blocked, so it waits on thread injection. Two controls isolate that:

variant, P=50 / Max=10	time Δ	alloc Δ
SteadyStateOpenQueryClose (threadpool, sync)	+136.27%	+5.87%
SteadyStateOpenQueryCloseDedicatedThreads (sync)	+5.61%	+8.00%
SteadyStateOpenQueryCloseAsync (threadpool)	−4.20%	+10.27%

Allocation is comparable across all three while time differs by 140 points, so the regression is not the pool's connection handling. ConnectionPoolThreadPoolPressureRunner varies only the thread floor and shows the same thing: MinWorkerThreads=8 gives +57.48% (3/3), MinWorkerThreads=128 gives +11.23% (2/3). The ample-pool variants (Max=50, Max=100) are unchanged or faster, so this is specific to Parallelism ≫ MaxPoolSize.

Accepted as an application-configuration boundary: an application should keep parallelism below the threadpool worker count, and pre-warming the threadpool is not the driver's job. Enabling AllowSynchronousContinuations was measured (−7.8%) and rejected, because OpenNewInternalConnection sits in the same retry loop and an inline resume could run a TCP connect, TLS handshake and login on a caller's Close() thread.

C. Saturated async waiter allocation (1 of 38). SteadyStateOpenQueryCloseAsync P=50/Max=10, +10.27% allocation while 4% faster. The only genuine per-operation allocation regression, and the only Async row with a meaningful delta (Max=50 is +0.19%, Max=100 is +0.77%). A saturation sweep holds flat at +24 B/op down to maxPool=25, then jumps to +1,241 (maxPool=10) and +1,481 (maxPool=5) on the async path only; an isolated microbenchmark predicted +10.05% for this shape against the +10.27% measured. 392 B of it is passing a cancellable token to ReadAsync: 104 B defeats UnboundedChannel's cached reader, 192 B is the CTS, 96 B its timer. Sync escapes because _syncOverAsyncSemaphore caps concurrent readers.

Not fixed here. Deferring CTS creation to the first blocking wait was implemented and measured: it saves 24 B of 1,481, because a saturated caller always blocks and so always needs the token. Removing the 392 B means reading with CancellationToken.None, which needs an explicit waiter queue so a timed-out waiter can deregister without stranding the connection a pending read would swallow. That is the same restructuring B needs, so it is one workstream rather than two.

D. Noise (3 of 38). AsyncLargeDataReadRunner contributes the +68.23% (2/3) and two 1/3 flags. Across all 16 of its rows the deltas scatter from −13.92% to +68.23% with allocation pinned at ≈0.00% throughout, and three of those rows are flagged improvements of comparable size; the same 5 MB payload reads +68.23% with a 1 MB buffer and −13.92% with an 8 KB buffer. It takes one checkout and then reads, so the switch has nothing to act on after connect. The previous run's two 1/3 flags (MarsOverheadRunner, BeginTransactionRunner) did not reproduce.

Two caveats worth stating. SteadyStateOpenQueryClose P=50/Max=10 moved from +66.61% to +136.27% between runs (both 3/3): this is a tail-latency effect, not a shifted median, so the magnitude is not a stable number to quote. And the MinWorkerThreads=128 control lands at +11.23% rather than parity, because its 50 blocked waiters are still threadpool threads competing for scheduling, which the dedicated-thread variant avoids.

Separately, I investigated and ruled out redundant liveness probing as a cause of the original regressions. v2 calls IsLiveConnection up to 3x per cycle vs v1's 1x, but IsConnectionAlive is gated by a 5 ms window and a successful check resets the timer, so the extra calls collapse to a few DateTime.UtcNow reads.

Issues

N/A

Testing

Two unit tests added to ChannelDbConnectionPoolTest:

• GetConnectionAsync_WithIdleConnection_ShouldCompleteInline is the meaningful one. It asserts the TaskCompletionSource is already completed when TryGetConnection returns, which is impossible to observe if the work was dispatched via Task.Run. Verified it fails without the fix.
• GetConnection_WithIdleConnection_ShouldReturnInline covers the sync path.

The full connection pool unit test suite passes (320 tests, previously 318).

Not automated: the perf improvement itself. Plan is to re-run the experiment pipeline on this branch with switchUnderTest: UseConnectionPoolV2 and confirm the async cases move back toward parity.

One gap worth flagging: the full unit test suite hangs in SimulatedServerTests on macOS. I confirmed this is pre-existing by reproducing the identical hang on a clean tree, and those tests do not use the v2 pool.

Guidelines

Please review the contribution guidelines before submitting a pull request:

• Contributing
• Code of Conduct
• Best Practices
• Coding Style
• Review Process

[mdaigle]

https://dev.azure.com/SqlClientDrivers/ADO.Net/_build/results?buildId=167208&view=results

[Copilot]

Pull request overview

Adds a new perf “switch experiment” pipeline to A/B test a single runner-config/AppContext switch on the same commit, and applies an optimization to the v2 channel-based connection pool to remove avoidable allocations and thread-pool dispatch on the idle-connection fast path.

Changes:

• Introduces sqlclient-perf-experiment.yml to run baseline/current as the same source with exactly one switch flipped (no Kusto ingestion by design).
• Updates perf runner scripts (run-perf-tests.sh / .ps1) and the interleaving orchestrator to support per-variant RUNNER_CONFIG overrides for switch A/B.
• Adds TryGetPooledConnectionInline to ChannelDbConnectionPool.TryGetConnection to satisfy idle/transacted requests inline and avoid Task.Run/CTS/Task<T> allocations; adds unit tests covering sync and async inline completion.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolTest.cs	Adds unit tests verifying idle-connection requests complete inline for both sync and async paths.
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ConnectionPool/ChannelDbConnectionPool.cs	Adds a pooled-connection inline fast path (TryGetPooledConnectionInline) to avoid async state machine + CTS + thread-pool dispatch when an idle/transacted connection is immediately available.
eng/pipelines/perf/sqlclient-perf-experiment.yml	New manual perf pipeline that runs the same commit twice with one switch forced off vs on.
eng/pipelines/perf/scripts/run-perf-tests.sh	Adds --switch-under-test mode, generates per-variant runner configs, and wires them into interleaved/sequential runs.
eng/pipelines/perf/scripts/run-perf-tests.ps1	Windows equivalent of switch-under-test A/B mode, including per-variant runner config generation and plumbing.
eng/pipelines/perf/scripts/interleave_perf.py	Adds optional per-variant environment overrides so baseline/current subprocesses can use different RUNNER_CONFIG values.
eng/pipelines/perf/README.md	Documents the experiment pipeline, its constraints, and why it must not be ingested into Kusto.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolTest.cs:1561

• GetConnection_ConcurrentHeldLoad_GrowsPoolToDemand asserts factory.Created == parallelism, but RunWorkersAsync does not synchronize worker start. Because tasks can begin at different times, some workers may start after others have already returned a connection and end up reusing it, causing factory.Created to be < parallelism and making this test timing/scheduling-sensitive (potentially flaky in CI). Consider adding a start barrier (e.g., CountdownEvent + ManualResetEventSlim) so all workers contend concurrently before any can return, or relax the assertion to a range that still detects the “serialized to 1 connection” failure mode without requiring perfect simultaneity.

            // Act: workers hold their connections, so none can be reused and the pool must grow.
            await RunWorkersAsync(pool, parallelism, iterationsPerWorker, holdMilliseconds: 25);

            // Assert: one connection per concurrent caller, and no more.
            Assert.Equal(parallelism, factory.Created);

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

eng/pipelines/perf/scripts/run-perf-tests.ps1:150

• The comment above the baseline-selector validation still says there are "two baseline selectors" and that "the two baseline selectors are mutually exclusive", but this script now supports three selectors (BaselineVersion, BaselineSourceRef, and SwitchUnderTest). This makes the comment misleading for future maintenance and review.

# The two baseline selectors describe different builds of the same "baseline" pass, so requesting
# both is always a mistake; fail fast rather than silently honouring one of them.
if ((-not [string]::IsNullOrEmpty($BaselineVersion)) -and (-not [string]::IsNullOrEmpty($BaselineSourceRef))) {
    throw "-BaselineVersion and -BaselineSourceRef are mutually exclusive."

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (1)

eng/pipelines/perf/README.md:270

• The README states that the switch-experiment baseline requires “no second build at all”, but the scripts only reuse a single build in the interleaved path. In sequential mode, run-perf-tests.sh/.ps1 still run the baseline and current passes separately (and therefore build twice). Clarifying this avoids misleading users who select sequential mode.

- **Baseline (switch experiment)**: no second build at all — `--switch-under-test` measures one
  source tree twice, so the scripts build the `current` variant once and point both passes at it,
  differing only in the runner config each pass is handed. Used by the experiment pipeline.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/IdleConnectionChannelTest.cs:344

• This test mutates process-wide ThreadPool limits (SetMinThreads/SetMaxThreads). The UnitTests configuration enables parallel test collections, so running this concurrently with other tests can cause hangs/timeouts unrelated to the channel behavior being validated. This should be placed in a non-parallel xUnit collection (DisableParallelization=true) or otherwise isolated so no other tests execute while the ThreadPool is restricted.

                Assert.True(ThreadPool.SetMinThreads(1, minIo));
                Assert.True(ThreadPool.SetMaxThreads(1, maxIo));

                // Occupy the single available worker thread for the duration of the wait.
                ThreadPool.QueueUserWorkItem(_ => blocked.Wait());

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolThreadPoolPressureRunner.cs:103

• ThreadPool.SetMinThreads returns a bool indicating whether the requested minimums were applied. Since this benchmark depends on pinning the thread pool floor for reproducibility, it should fail fast if the call fails (and likewise if restoring the original values fails).

            ThreadPool.GetMinThreads(
                out _originalMinWorkerThreads, out _originalMinCompletionPortThreads);
            ThreadPool.SetMinThreads(MinWorkerThreads, _originalMinCompletionPortThreads);

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolRampRunner.cs:148

• ConnectionPoolRampRunner.ColdStartRamp() can deadlock if any worker throws before calling allConnected.Signal() (e.g., Open fails). Other workers then block forever in allConnected.Wait(), stalling the perf run. Consider using a CancellationTokenSource and passing its token to Wait(...), cancelling it on any worker failure so the benchmark fails fast instead of hanging indefinitely.

        public void ColdStartRamp()
        {
            using var allConnected = new CountdownEvent(Parallelism);

            var tasks = new Task[Parallelism];
            for (int i = 0; i < Parallelism; i++)
            {
                tasks[i] = Task.Factory.StartNew(() =>
                {
                    using var conn = new SqlConnection(_connectionString);
                    conn.Open();

                    allConnected.Signal();
                    allConnected.Wait();
                    // Dispose returns the connection to the pool.
                }, TaskCreationOptions.LongRunning);
            }

            Task.WaitAll(tasks);
        }

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolThreadPoolPressureRunner.cs:109

• ThreadPool.SetMinThreads returns a bool indicating whether the new minimums were applied. If this call fails, the benchmark is no longer running under the intended (pinned) thread pool floor, which can make results noisy/misleading. Consider checking the return value in both setup and cleanup and failing the benchmark run if the thread pool settings can't be applied/restored.

            ThreadPool.GetMinThreads(
                out _originalMinWorkerThreads, out _originalMinCompletionPortThreads);
            ThreadPool.SetMinThreads(MinWorkerThreads, _originalMinCompletionPortThreads);

            var builder = new SqlConnectionStringBuilder(s_config.ConnectionString)
            {
                Pooling = true,
                MaxPoolSize = MaxPoolSize,
                MinPoolSize = 0
            };
            _connectionString = builder.ConnectionString;
        }

        [GlobalCleanup]
        public void Cleanup()
        {
            ThreadPool.SetMinThreads(
                _originalMinWorkerThreads, _originalMinCompletionPortThreads);
        }

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolRampRunner.cs:116

• If any worker throws (e.g., OpenAsync fails), other workers can end up awaiting release.Task forever because release is only completed when the CountdownEvent reaches 0. That can hang the perf run instead of failing fast. Consider completing release with the exception so all workers unblock and the benchmark fails deterministically.

                        release.TrySetResult(true);
                    }

                    await release.Task;
                    // Dispose returns the connection to the pool.

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolRampRunner.cs:143

• If conn.Open() throws in any worker, that worker never signals the CountdownEvent and the remaining workers can block forever in allConnected.Wait(), hanging the run. Signaling in a finally avoids deadlocks and still preserves the intended steady-state behavior on success.

                    using var conn = new SqlConnection(_connectionString);
                    conn.Open();

                    allConnected.Signal();
                    allConnected.Wait();
                    // Dispose returns the connection to the pool.

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolThreadPoolPressureRunner.cs:94

• ThreadPool.SetMinThreads returns a bool indicating whether the requested minimum was applied. If it fails (e.g., due to runtime limits), the benchmark no longer measures the intended configuration and results become misleading. Consider checking the return value and failing fast.

            ThreadPool.GetMinThreads(
                out _originalMinWorkerThreads, out _originalMinCompletionPortThreads);
            ThreadPool.SetMinThreads(MinWorkerThreads, _originalMinCompletionPortThreads);

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

eng/pipelines/perf/scripts/run-perf-tests.ps1:356

• The generated connection string is missing the password key/value: it currently inserts a literal "******" segment, which will produce an invalid connection string and break Windows perf runs. This also leaves $escapedPassword unused even though it’s computed just above.

    $cfg.ConnectionString = "Server=tcp:$SqlServer,1433;User ID=sa;Password=$escapedPassword;Initial Catalog=$DbName;TrustServerCertificate=True;Encrypt=False;"

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolRampRunner.cs:107

• If any worker throws before signaling allConnected (e.g., OpenAsync fails), release is never completed and the successfully-opened workers will await release.Task forever, hanging the benchmark run. The runner should fault release (or otherwise unblock peers) on error so failures don’t deadlock the harness.

                tasks[i] = Task.Run(async () =>
                {
                    using var conn = new SqlConnection(_connectionString);
                    await conn.OpenAsync();

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolRampRunner.cs:143

• ColdStartRamp can deadlock if any worker throws before calling allConnected.Signal() (e.g., conn.Open fails). Other workers then block forever in allConnected.Wait(), stalling the run until timeout. Signal the CountdownEvent in a finally block so failures don’t strand other threads.

                tasks[i] = Task.Factory.StartNew(() =>
                {
                    using var conn = new SqlConnection(_connectionString);
                    conn.Open();

                    allConnected.Signal();
                    allConnected.Wait();
                    // Dispose returns the connection to the pool.

src/Microsoft.Data.SqlClient/tests/PerformanceTests/BenchmarkRunners/ConnectionPoolRampRunner.cs:107

• ColdStartRampAsync can hang indefinitely if any worker fails before signaling the CountdownEvent (e.g., OpenAsync throws). In that case release is never set and the other workers await release.Task forever, stalling the perf run until the outer timeout. Ensure the CountdownEvent is always signaled (even on failure) so the run fails fast instead of deadlocking.

                tasks[i] = Task.Run(async () =>
                {
                    using var conn = new SqlConnection(_connectionString);
                    await conn.OpenAsync();