Make MixpanelClient durable with a disk-backed event spool

MixpanelClient previously fired each event over HTTP immediately and
silently dropped it when offline. The Segment path already had durability
via Segment.Analytics.CSharp; this closes that gap for Mixpanel with a
DiskQueue-backed spool and a background flush loop, so events survive
offline stretches and process restarts and are delivered at-least-once.

Design decisions
- EventSpool wraps DiskQueue as a bounded, transport-agnostic durable
  queue: events are enqueued as opaque JSON bytes immediately (the
  write-ahead IS the durability guarantee -- "online?" is only ever
  knowable from a send outcome, so send-first-then-spool was rejected).
  Bounded by both item count (5,000) and total bytes (20MB, drop-oldest),
  so an unbounded offline period can't grow the spool or its eventual
  upload liability without limit.
- Delivery batches via Mixpanel's /import endpoint (strict mode) rather
  than one /track POST per event: one request per drain tick, no /track
  5-day age limit on a replayed backlog, and strict mode reports
  per-record rejections (failed_records) so one poison event doesn't
  cost the rest of a good batch. ProcessBatchAsync gathers a batch in a
  single DiskQueue transaction and commits/rolls back on the batch's
  verdict (Delivered/PoisonDrop commit; RetryableFailure rolls back
  in order for a later retry).
- Delivery is async end-to-end (IEventSender.SendBatchAsync,
  EventSpool.ProcessBatchAsync, MixpanelClient.DrainOnceAsync via a
  Polly retry + circuit breaker pipeline), serialized by a SemaphoreSlim
  (not a monitor lock, since the lock must be held across the network
  await). Sync Flush()/ShutDown() block only at the boundary on a
  bounded drain; FlushAsync()/ShutDownAsync() are the non-blocking
  public alternatives (IClient, Analytics.FlushClientAsync /
  analytics.ShutDownAsync).
- A drain tick is bandwidth-bounded for slow/metered connections: a
  soft 256KB-per-tick byte budget (checked after each event, so one
  oversized event still goes rather than starving the queue) on top of
  the existing sequential-sends/one-connection design.
- Consent revocation (AllowTracking = false) purges the spool
  immediately (dispose + delete the spool directory + reopen, not a
  dequeue-and-flush loop, which left purged bytes sitting in DiskQueue's
  data files) and cancels whatever send is currently in flight first, so
  a batch that was mid-POST when consent was revoked rolls back and gets
  purged instead of still reaching Mixpanel afterward.
- All spooled string values (recursively, including nested object/array
  properties) are path-scrubbed before being persisted, normalizing user
  home directories so stack traces and usage events can't leak OS
  account names.
- The spool's byte total is persisted alongside it and restored on open
  (cross-checked against DiskQueue's own item-count estimate, falling
  back to an exact dequeue-and-verify measurement if the persisted value
  is missing or looks stale), so a restart with a large backlog doesn't
  require synchronously re-measuring the whole spool before Initialize
  can return.
- Track() on a never-initialized client throws InvalidOperationException,
  matching the pre-durability contract; an initialized client whose spool
  couldn't be created (e.g. another process holds the cross-process lock)
  degrades to a no-op instead -- analytics must never crash the host.

Hardening against bad network conditions
- Captive portals (hotel/cafe Wi-Fi login pages) can no longer silently
  eat events: HttpClient no longer follows redirects, and any 3xx is
  classified as retryable rather than risk following a portal to a 200
  that was never actually Mixpanel.
- Flush()/ShutDown() are bounded (~5s / 20 attempts) even against a
  "black hole" server that accepts the TCP connection but never
  responds: a single deadline-bound CancellationTokenSource is threaded
  all the way down to HttpClient, and the Polly retry pipeline is
  bypassed for these bounded attempts (retrying during a bounded
  shutdown/flush has little delivery value and would multiply, not
  bound, the wait).
- The circuit breaker's MinimumThroughput is tuned to how many outcomes
  one fully-failing drain tick actually produces (1 attempt + 2
  retries = 3), so a sustained outage trips it after one bad tick
  instead of never being able to open at all.
- A 200 response with body "0" (Mixpanel's rejected-payload signal) is
  classified PoisonDrop instead of Delivered.
- Events over Mixpanel's documented 1MB per-event limit are refused at
  enqueue time (counted Failed immediately) rather than being spooled
  and later timing out as an indistinguishable-from-offline retryable
  failure that would wedge the head of the queue forever.

Test coverage
- 97 NUnit tests across four fidelity layers: pure unit (PathScrubber,
  AnalyticsEvent serialization), a real EventSpool against a real
  DiskQueue on a temp directory, a real EventSpool driving a real
  MixpanelClient against a scripted/fake IEventSender, and
  WireMock.Net-backed HTTP tests of MixpanelEventSender against real
  status codes and bodies.
- Coverage includes: no-loss retry and crash-window dedup via
  $insert_id, spool item/byte bounding (drop-oldest), consent purge
  (including that purged bytes are actually gone from disk, not just
  marked consumed), cross-process exclusive lock contention, corrupt
  spool recovery (garbage in the data file vs. the transaction log),
  bounded-drain-against-a-hanging-sender, real circuit-breaker
  threshold behavior, captive-portal/redirect/cancellation handling,
  recursive PII scrubbing (including nested object/array properties),
  cap-eviction accounted for in Statistics, and the persisted-byte-total
  fast path (plus its fallback when that value is missing or stale).
- Manually verified end-to-end against the live Mixpanel API from both
  SampleApp and cross-process offline scenarios (per-process firewall
  block; a Windows Sandbox guest with no virtual NIC at all).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: johnml1135

src/DesktopAnalytics/Analytics.cs

@@ -11,6 +11,7 @@
 using System.Reflection;
 using System.Runtime.InteropServices;
 using System.Threading;
+using System.Threading.Tasks;
 using System.Xml.Linq;
 using System.Xml.XPath;
 using JetBrains.Annotations;
@@ -738,6 +739,22 @@ public void Dispose()
 			_client?.ShutDown();
 		}
 
+		/// <summary>
+		/// Async alternative to <see cref="Dispose"/> for hosts that shut down asynchronously:
+		/// flushes/shuts down the underlying client without blocking a thread on any in-flight
+		/// delivery. Bounded just like Dispose -- an offline shutdown still returns promptly, with
+		/// undelivered events left on disk (Mixpanel) or in the transport library's own store
+		/// (Segment) for the next launch. Calling <see cref="Dispose"/> afterwards (e.g. from a
+		/// <c>using</c> block wrapping this object) is a harmless no-op.
+		/// </summary>
+		/// <param name="cancellationToken">Optionally ends the final delivery attempt even sooner
+		/// than its own bound; undelivered events stay queued for the next launch. Cancellation
+		/// never faults the task.</param>
+		public Task ShutDownAsync(CancellationToken cancellationToken = default)
+		{
+			return _client?.ShutDownAsync(cancellationToken) ?? Task.CompletedTask;
+		}
+
 		/// <summary>
 		/// Indicates whether we are tracking or not
 		/// </summary>
@@ -765,6 +782,34 @@ public static bool AllowTracking
 						s_singleton.Initialize(initializationParameters);
 						return; // Initialize sets s_allowTracking = true
 					}
+
+					// Already initialized and now re-enabled: re-arm any background flush loop that a
+					// prior PurgeQueuedEvents (consent revocation) paused. See offline-analytics.md.
+					try
+					{
+						s_singleton._client?.ResumeSending();
+					}
+					catch (Exception e)
+					{
+						Debug.WriteLine("Analytics.AllowTracking: ResumeSending failed: " + e);
+					}
+				}
+
+				if (!value)
+				{
+					// Consent revoked: purge any durable client's on-disk spool immediately (see
+					// offline-analytics.md, "Consent & lifecycle"). Guarded against a null singleton
+					// (AllowTracking can technically be set before an Analytics object is
+					// constructed) and a null client (the Segment/Mixpanel client is only assigned
+					// inside the Analytics constructor).
+					try
+					{
+						s_singleton?._client?.PurgeQueuedEvents();
+					}
+					catch (Exception e)
+					{
+						Debug.WriteLine("Analytics.AllowTracking: PurgeQueuedEvents failed: " + e);
+					}
 				}
 
 				s_allowTracking = value;
@@ -1082,7 +1127,19 @@ private static string GetUserNameForEvent()
 
 		public static void FlushClient()
 		{
-			s_singleton._client?.Flush();
+			s_singleton?._client?.Flush();
+		}
+
+		/// <summary>
+		/// Async counterpart of <see cref="FlushClient"/>: attempts delivery of anything pending
+		/// without blocking a thread on the network. Bounded -- returns promptly even while
+		/// offline, leaving undelivered events queued.
+		/// </summary>
+		/// <param name="cancellationToken">Optionally ends the flush even sooner than its own
+		/// bound; undelivered events stay queued. Cancellation never faults the task.</param>
+		public static Task FlushClientAsync(CancellationToken cancellationToken = default)
+		{
+			return s_singleton?._client?.FlushAsync(cancellationToken) ?? Task.CompletedTask;
 		}
 	}
 }

src/DesktopAnalytics/AnalyticsEvent.cs

@@ -0,0 +1,83 @@
+using System;
+using System.Text;
+using Segment.Serialization;
+
+namespace DesktopAnalytics
+{
+	/// <summary>
+	/// A single spoolable analytics event in a neutral, JSON-serializable form. This is the
+	/// unit written to and read from the on-disk event spool (DiskQueue), independent of the
+	/// eventual transport (Mixpanel, etc). Instances are round-tripped as UTF-8 JSON bytes so
+	/// the spool stays portable and inspectable (see <see cref="ToBytes"/>/<see cref="FromBytes"/>).
+	/// </summary>
+	internal class AnalyticsEvent
+	{
+		public string AnalyticsId { get; set; }
+		public string EventName { get; set; }
+
+		/// <summary>Event properties, e.g. Message/Stack Trace for exception reports.</summary>
+		public JsonObject Properties { get; set; }
+
+		/// <summary>Mixpanel's $insert_id, stamped at enqueue time so at-least-once replay can
+		/// be safely deduplicated by the backend.</summary>
+		public string InsertId { get; set; }
+
+		/// <summary>The original event time, stamped at enqueue time so a later replay is
+		/// back-dated correctly rather than reported at delivery time.</summary>
+		public DateTimeOffset Time { get; set; }
+
+		public AnalyticsEvent()
+		{
+			Properties = new JsonObject();
+		}
+
+		/// <summary>
+		/// Creates a new event, stamping <see cref="InsertId"/> and <see cref="Time"/>. Both are
+		/// overridable so callers/tests can be deterministic; production code can omit them to get
+		/// a real GUID and the real wall-clock time.
+		/// </summary>
+		/// <param name="analyticsId">The stable per-user analytics id.</param>
+		/// <param name="eventName">The event name.</param>
+		/// <param name="properties">Event properties. May be null, in which case an empty
+		/// property bag is used.</param>
+		/// <param name="insertId">The $insert_id. Defaults to a fresh GUID. Never call
+		/// Guid.NewGuid() directly elsewhere in the stamping path -- pass it here so tests are
+		/// deterministic.</param>
+		/// <param name="time">The event time. Defaults to <c>DateTimeOffset.UtcNow</c>. Never
+		/// call DateTime.Now/DateTimeOffset.UtcNow directly elsewhere in the stamping path --
+		/// pass it here (e.g. from an injected <see cref="TimeProvider"/>) so tests are
+		/// deterministic.</param>
+		public static AnalyticsEvent Create(
+			string analyticsId,
+			string eventName,
+			JsonObject properties = null,
+			string insertId = null,
+			DateTimeOffset? time = null
+		)
+		{
+			return new AnalyticsEvent
+			{
+				AnalyticsId = analyticsId,
+				EventName = eventName,
+				Properties = properties ?? new JsonObject(),
+				InsertId = insertId ?? Guid.NewGuid().ToString(),
+				Time = time ?? DateTimeOffset.UtcNow
+			};
+		}
+
+		/// <summary>Serializes this event as UTF-8 JSON bytes, suitable for enqueuing into the
+		/// on-disk spool.</summary>
+		public byte[] ToBytes()
+		{
+			var json = JsonUtility.ToJson(this, false);
+			return Encoding.UTF8.GetBytes(json);
+		}
+
+		/// <summary>Deserializes an event previously produced by <see cref="ToBytes"/>.</summary>
+		public static AnalyticsEvent FromBytes(byte[] bytes)
+		{
+			var json = Encoding.UTF8.GetString(bytes);
+			return JsonUtility.FromJson<AnalyticsEvent>(json);
+		}
+	}
+}

src/DesktopAnalytics/DesktopAnalytics.csproj

@@ -17,10 +17,13 @@
     <AssemblyOriginatorKeyFile>..\..\DesktopAnalytics.snk</AssemblyOriginatorKeyFile>
 </PropertyGroup>
 <ItemGroup>
+    <PackageReference Include="DiskQueue" Version="1.7.2" />
     <PackageReference Include="JetBrains.Annotations" Version="2023.2.0">
       <PrivateAssets>All</PrivateAssets>
     </PackageReference>
+    <PackageReference Include="Microsoft.Bcl.TimeProvider" Version="8.0.0" />
     <PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.1.1" PrivateAssets="All" />
+    <PackageReference Include="Polly" Version="8.4.2" />
 </ItemGroup>
 <ItemGroup>
   <PackageReference Include="mixpanel-csharp" Version="6.0.0" />