STTP synchronize metadata optimizations

[ritchiecarroll]

Summary

DataSubscriber.SynchronizeMetadata() synchronized received metadata one row at a time, issuing roughly 4D + 3M + 4P statements for D devices, M measurements and P phasors — about 300,000 round trips for a 100k-measurement set. This reworks it into a bulk diff, splits the 810-line method into per-table operations, and adds SQL Server bulk paths.

Measured on identical input (40 devices / 25,252 measurements / 4 phasors), run sequentially without contention:

Backend	Before	After
SQL Server	540.03 s	2.62 s	206×
SQLite	109.96 s	5.66 s	19×

Full scale on SQL Server — 1,038 devices / 228,472 measurements from a production-shaped source: 24.49 s using 512 database statements.

Review guide

Five commits, meant to be read in order. The second is the one that changes behavior and deserves the most attention; the last three are throughput only.

Commit	What
2588fc92	Correctness fixes + per-phase instrumentation
9fae4ed4	Snapshot diff + split into src/lib/sttp.core/Metadata/
df0cf383	Per-database statement batching
0a41902c	SqlBulkCopy for measurement inserts
f0157f97	Set-based measurement updates

What changed, and why it's faster

Per-row probes → bulk snapshots. Device existence, ownership and record-ID resolution went from three statements per device to two chunked queries total; measurement existence from one probe per row to chunked lookups against the clustered key.

GUIDs supplied directly on INSERT. This removes UPDATE Measurement SET SignalID = … WHERE AlternateTag = <temp guid>, which ran once per new measurement. AlternateTag is varchar(max)/TEXT in all five schema variants and cannot be indexed, so that statement was a full table scan against a growing table — the dominant cost, and quadratic. The TODO at the old line 3781 assumed an index was the fix; it isn't. The temporary-alternate-tag mechanism is gone with it.

ActiveMeasurement no longer queried. It's an 11-table join including a cross join, and its result was then filtered with an extra query per candidate row. One restricted query against Measurement replaces both, carrying the view's enabled-state filtering forward so measurements of disabled devices are still left alone.

Batched deletes, which on SQL Server collapse the INSTEAD OF DELETE trigger's seven dependent deletes from 7×N to 7 per batch.

SQL Server bulk paths for measurement inserts and updates, staging through a temp table. Scoped to measurements deliberately: device inserts drive the trigger that maintains Runtime, which every Runtime*/Iaon* view and ActiveMeasurement.DeviceID depend on, and device/phasor volumes don't justify the risk.

Three findings worth your attention

1. Invalid SQL on every backend except SQLite. The mutual-subscription delete filter was built as " AND Internal == 0". == is accepted only by SQLite and is a syntax error on SQL Server, PostgreSQL, MySQL and Oracle. Reachable whenever MutualSubscription is enabled with Internal false. Fixed in 2588fc92.

2. The old code could not sync two new devices against the .NET 9 schema. There, Device.UniqueID has no default, no GUID trigger, and is nullable but UNIQUE. The old insert-then-fix-up pattern inserted NULL, and SQL Server permits only one NULL under a unique constraint — so the second new device would fail. Supplying the GUID on insert fixes this as a side effect.

3. The old code wrote 25,157 garbage change-tracking rows. Old recorded 75,512 distinct TrackedChange entries against new's 50,355. Every one of the 25,157 extra entries references a SignalID that does not exist in Measurement — a consequence of inserting with a database-generated placeholder GUID (firing the tracker) and then overwriting SignalID with the real value (firing it again). Nothing is recorded by the old code and missed by the new one, so the new behavior is a strict improvement, not a loss.

Decisions that need a reviewer's sign-off

Measurement retirement under SyncIndependentDevices is unchanged, and it does nothing. Those devices are not parented to the subscriber, so the old ActiveMeasurement lookup never matched them and retired measurements were silently left in place. A direct query would begin deleting records that have never been deleted before, so the existing behavior is preserved deliberately and marked in the code. Whether independently synchronized devices should participate in retirement is a separate call.

Chunked transactions were dropped from the plan. Their purpose was reducing per-statement commit overhead, but batching already removes ~99.9% of write statements. Implementing them would have cost the all-or-nothing guarantee UseTransactionForMetadata provides today, so existing transaction semantics are unchanged.

Batched statements bypass the framework parameter helpers. This is required, not an optimization: both frameworks re-parse the statement text on every execution to infer parameter names, and the .NET Framework tokenizer recognizes only space, parenthesis, comma and equals as delimiters — a parameter adjacent to a semicolon is silently dropped and the call fails on a parameter-count mismatch. Parameters are built directly against the command; ANSI string typing is applied on the .NET Framework path to match what the helpers would have done.

New connection-string settings

Setting	Default	Meaning
MetadataSyncBatchSize	0	Rows per batched statement; 0 selects the per-database default, 1 disables batching entirely
UseBulkMetadataSync	true	Allows the SQL Server bulk path

UseBulkMetadataSync is declined automatically, with a status message naming the reason, when the optional audit-log schema is installed. Those triggers assign from an arbitrary single row of the inserted pseudo-table and only record correct history for single-row writes, so neither firing them nor skipping them is acceptable.

The completion status message now reports per-phase timings and statement counts:

Meta-data synchronization completed successfully in 24.49 seconds using 512 database statements
[devices: 3.03 seconds, measurements: 21.17 seconds, phasors: 199.87 milliseconds]

Verification

Against SQL Server Express 2022 (openPDC schema) and SQLite (openPDC-InitialDataSet.db), driving the real DataSubscriber through a harness that reads metadata with the same queries DataPublisher issues.

Destination databases compared row for row after each run:

	SQL Server	SQLite
Measurement rows differing	0 of 25,157	0
Device rows differing	1 — the harness's own subscriber device¹	0
Phasor rows differing	0	0

¹ Created independently in each database, so it carries a different generated UniqueID. All synced devices match exactly.

SQLite destinations also match the SQL Server destinations field for field, with GUIDs preserved as lowercase text.

Delete propagation, verified at full scale (1,033 devices / 193,182 measurements after removals): deleted devices, their cascaded measurements, individually deleted measurements and deleted phasors all disappear from the destination, with zero surviving records lost.

Not tested: PostgreSQL, MySQL, Oracle. PostgreSQL and MySQL run the same generic path SQLite exercises, differing only in batch limits (MySQL additionally disables multi-statement batching). Oracle deliberately falls through to the unbatched base dialect and produces the same statement stream as before. Note the .NET 9 schema only emits views and triggers under IfDatabase(SQLite|SqlServer), so these matter only for .NET 4.8/openPDC deployments.

Follow-ups, not in this PR

• IX_Measurement_DeviceID — Measurement.DeviceID is unindexed on both schemas despite the foreign key, so every device-scoped read and the Device delete cascade scan the table. Highest-value index available.
• Phasor(PrimaryVoltageID) / Phasor(DestinationPhasorID) — self-referencing FK is unindexed.
• Rewriting the AuditLog.sql triggers set-based would fix their multi-row correctness bug and let bulk sync coexist with auditing.
• DataPublisher uses DataTable.Compute inside loops over the metadata tables — an O(N²) pattern on the publisher side of the same refresh.

🤖 Generated with Claude Code