What happens?
A multi-process worker pool doing concurrent read-only queries against a DuckDB file with the spatial extension crashes repeatedly with STATUS_ACCESS_VIOLATION (0xC0000005) inside the DuckDB extension binary. There is no interrupt, no signal, no cancellation, and no writer: the worker processes simply die mid-query and the supervisor respawns them.
It reproduces on two different machines, two DuckDB versions (1.4.4 and 1.5.5), two CPython patch levels and two unrelated workloads. I have minidumps and cdb stacks for both versions, and I believe the 1.4.4 and 1.5.5 crashes are one defect even though Windows Error Reporting buckets them as two (see "One defect, two WER buckets" below).
I cannot supply a minimal reproducer and I cannot get past this without maintainer-side symbolication, because the published Windows wheel ships no PDBs. Concrete asks are at the bottom.
Frequency
On the 1.5.5 host, a single ~7 minute run of the workload produced 14 Application Error / Event ID 1000 records for python.exe, and 38 in one day. Each crash kills one worker within seconds of that worker starting; the parent survives, so from the outside "nothing is down".
Fault site (1.5.5)
Exception c0000005, INVALID_POINTER_READ, faulting module _duckdb.cp313-win_amd64.pyd.
ExceptionAddress: 00007ffdcbf5c020 (_duckdb_cp313_win_amd64+0x00000000018dc020)
ExceptionCode: c0000005 (Access violation)
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005__duckdb.cp313-win_amd64.pyd!Unknown
00007ffd`cbf5c020 8a01 mov al, byte ptr [rcx] ds:00028910`ebfa612f=??
00007ffd`cbf5c022 3a0411 cmp al, byte ptr [rcx+rdx]
rax=00028910ebfa612f rbx=000000000000007d rcx=00028910ebfa612f
rdx=ffffffffff86dd00 rsi=000000000000007d rdi=0000000000000001
rip=00007ffdcbf5c020 rsp=00000058e537d818 rbp=0000000000000000
r8=0000000000001f00
The mov al,[rcx] / cmp al,[rcx+rdx] pair is the shape of an inlined byte-compare loop. Note rdx holds a sign-extended-negative value where a pointer or offset argument belongs.
Caveat on the reported fault address: cdb prints Attempt to read from address ffffffffffffffff, but that is its rendering of an address the minidump did not record. The actual bad pointer is rcx = 0x00028910ebfa612f.
Fault offsets observed in one day on this host (Event-1000 Properties[7]):
| module offset |
count |
0x00000000018dc020 |
24 |
0x00000000018dc060 |
11 |
0x00000000018dc0a0 |
1 |
0x000000000058b2fa |
1 |
Three fault sites 0x40 apart inside one region is consistent with an unrolled byte-compare loop.
Stack, 1.5.5 (faulting thread, index 27 of 47). Every duckdb:: name below is the nearest preceding export plus a large offset — the wheel ships no PDBs, so the module attribution is definitive but the function names are almost certainly not the functions executing. Treat repeated names as "same region of the binary".
_duckdb!duckdb::FunctionBinder::BindSortedAggregate+0x1d76a0 <-- fault
_duckdb!duckdb::MetadataBlock::operator=+0x54662
_duckdb!duckdb::MetadataBlock::operator=+0x21bec
_duckdb!duckdb::MetadataBlock::operator=+0x642f4
_duckdb!duckdb::VectorOperations::TryCast+0x488d
_duckdb!duckdb::ExpressionExecutor::ExecuteExpression+0x5b1
_duckdb!duckdb::FunctionBinder::BindSortedAggregate+0x40be5
_duckdb!duckdb::InterruptState::Callback+0x16d0
_duckdb!duckdb::InterruptState::Callback+0x2aa8
_duckdb!duckdb::StreamQueryResult::ExecuteTask+0x11a
_duckdb!duckdb::PendingQueryResult::ExecuteTask+0x59
_duckdb!duckdb::FunctionData::SupportStatementCache+0x874
_duckdb!duckdb::FileSystem::IsPipe+0xdb5
_duckdb!duckdb::FunctionData::SupportStatementCache+0x2539
_duckdb!duckdb::ArenaAllocator::GetTail+0x1d8f4
_duckdb!duckdb::ArenaAllocator::GetTail+0x26d6
_duckdb!duckdb::NumericValueUnion::GetReferenceUnsafe<short>+0x8168
python313+0x2eb52
Fault site (1.4.4, a different machine)
ExceptionAddress: 00007ff813271340 (VCRUNTIME140!memcmp+0x0000000000000050)
ExceptionCode: c0000005 (Access violation)
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_VCRUNTIME140.dll!memcmp
00007ff8`13271340 488b01 mov rax, qword ptr [rcx] ds:000001dc`a53008b0=????????????????
rax=000001dca52ffe6c rbx=0000000000000000 rcx=000001dca53008b0
rdx=fffffffffffff5bc rsi=0000000000000000 rdi=0000000000000001
r8=0000000000000024 rbp=0000000000000000
VCRUNTIME140!memcmp+0x50 <-- fault
_duckdb!duckdb::MetadataBlock::operator=+0x572f1
_duckdb!duckdb::MetadataBlock::operator=+0x26451
_duckdb!duckdb::MetadataBlock::operator=+0x69d9d
_duckdb!duckdb::BufferedFileWriter::WriteData+0x2b1e
_duckdb!duckdb::ExpressionExecutor::ExecuteExpression+0x5be
_duckdb!duckdb::FunctionBinder::BindSortedAggregate+0x184d5
_duckdb!duckdb::InterruptState::Callback+0x16b0
_duckdb!duckdb::InterruptState::Callback+0x22cb
_duckdb!duckdb::VectorOperations::TryCast+0x1606
ucrtbase!thread_start<unsigned int (__cdecl*)(void *),1>+0x30
kernel32!BaseThreadInitThunk+0x17
ntdll!RtlUserThreadStart+0x2c
Second-sourced: an independently captured 1.4.4 minidump from 2026-08-31 gives the same VCRUNTIME140!memcmp+0x50 and the same bucket.
One defect, two WER buckets
I would ask you not to dedup these apart on the bucket names. Evidence that they are the same defect:
- Seven middle frames correspond, in the same order, across a minor-version bump:
MetadataBlock::operator= x3 -> a mid-level frame -> ExpressionExecutor::ExecuteExpression -> FunctionBinder::BindSortedAggregate -> InterruptState::Callback x2.
ExpressionExecutor::ExecuteExpression +0x5be (1.4.4) vs +0x5b1 (1.5.5) — 13 bytes apart. Hedged: this only holds if the nearest-preceding export is the same real function in both builds, which is unverifiable without PDBs.- The faulting operation is a byte-wise comparison entered with a garbage pointer in both. 1.4.4 calls out to
VCRUNTIME140!memcmp (mov rax,[rcx], 8-byte); 1.5.5 inlines the compare into the extension (mov al,[rcx] / cmp al,[rcx+rdx], 1-byte), which is exactly why the bucket flips from VCRUNTIME140.dll!memcmp to ..._duckdb...pyd!Unknown. The inlining is a build difference, not a different bug.
- Under the Windows x64 ABI a
memcmp(a, b, n) takes rcx=a, rdx=b, r8=n. r8 is a plausible length in both (0x24 = 36, 0x1f00 = 7936), and rdx is a sign-extended-negative value in both, with rbp=0 and rdi=1 in both.
Thread kinds
In 1.5.5 the fault is on the Python-owned thread, inside StreamQueryResult::ExecuteTask and bottoming out through python313. In the 1.4.4 dump it is on a native thread not owned by CPython (ucrtbase!thread_start -> kernel32!BaseThreadInitThunk -> ntdll!RtlUserThreadStart, all genuinely resolved symbols). So it faults on both thread kinds. Inference, not measurement: that fits shared state being read after free or realloc rather than a bug local to one thread.
Ruled out on my side
Each of these was tested by changing it and re-running the identical workload; none of them changed the crash rate in the direction the hypothesis predicts:
- Not a per-worker memory cap. Workers die well below the Job Object commit cap (n=24 sampling).
- Not
memory_limit. Raised from 1.5 GB to 5 GB with the process cap at 12288 MB: crashes arrived sooner and more often (1.80/min vs 0.59/min). No OOM line, no eviction, no memory-pressure restart anywhere in the pool output.
- Not
threads. Set to 1 machine-wide and verified inside the live processes' environment blocks: all three fresh workers still died within 2 minutes.
- Not a concurrent writer. No
.wal beside any of the attached database files during the window, and no other process had them open.
Possibly related, but I do not think it is the same
duckdb/duckdb#25011 is also 0xC0000005 on Windows with spatial on 1.5.5, but it is interrupt-triggered under Jupyter and the reporter traced it to EDR hooks and the ipykernel Win32 interrupt event. Mine has no interrupt, no notebook, and no EDR involvement, and it also occurs on 1.4.4. Different mechanism, possibly adjacent code.
To Reproduce
I do not have a minimal reproducer, and I want to be straight about that rather than post a script that does not fail. What I have is a reliable statistical reproduction of my own workload, plus dumps.
The workload, described generically:
- A FastAPI/uvicorn worker pool, 3 worker processes, each with its own DuckDB connection.
- Each worker runs concurrent read-only queries against a ~1 GB DuckDB file attached read-only, with the
spatial extension loaded, while a batch job drives them.
- No writes, no
ATTACH in RW mode, no INSTALL during the run, no interrupts or cancellations.
- Workers are cold-started; the first faults arrive within seconds to ~90 s of a worker starting.
- Deaths cluster at two moments: batch start, and the instant clients disconnect.
Instrumentation that makes it visible (a completed batch is not evidence of health — the pool respawns in place and the port stays listening):
Get-WinEvent -FilterHashtable @{LogName='Application';Id=1000;StartTime=(Get-Date).AddHours(-6)} |
Where-Object { $_.Message -match 'python' } |
ForEach-Object { $_.TimeCreated, $_.Properties[8].Value, $_.Properties[3].Value, $_.Properties[7].Value }Capturing dumps, in case it helps anyone reproducing on Windows: set HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\python.exe with DumpFolder, DumpType=1, DumpCount=N. DumpType=2 (full memory) silently writes nothing for job-capped worker processes, because WerFault is spawned as a child of the crashing process and inherits its commit cap. A minidump is sufficient for this defect. Analyse with cdb -z <dump> -y srv*<cache>*https://msdl.microsoft.com/download/symbols -i <dir holding a copy of the pyd> — without -i the module cannot be resolved at all.
What I can supply on request: three 1.5.5 minidumps (14.8 / 15.7 / 16.9 MB), the exact .pyd by hash below, the 1.4.4 minidumps, and full cdb transcripts. I have not attached the dumps here because minidumps carry memory fragments from a business workload.
Binary identity
The Windows wheel's extension binary carries no version resource — FileVersion and ProductVersion are both empty — which is why Windows crash telemetry reports version: 0.0.0.0 for every one of these crashes. It has to be identified by hash:
_duckdb.cp313-win_amd64.pyd 37,374,464 bytes
SHA256 C91833047245D694CD25E146E4F7FB4F412E71B0EAAE257E190A83E8F264A1DF
PE Timestamp 6A5F82FF (Tue Jul 21 10:32:31 2026) ImageSize 023B0000 <- 1.5.5
PE Timestamp 69740402 (Fri Jan 23 18:28:02 2026) ImageSize 021CE000 <- 1.4.4 (different machine)
Runtime identity on the 1.5.5 host:
duckdb.__version__ = 1.5.5
library_version = v1.5.5
extensions: ('spatial', 'eb1e57c', True) ('json', 'v1.5.5', True)
VCRUNTIME140 14.51.36247.0 (the 1.4.4 host had 14.42.34438.0)
What I am asking for
- Symbolicate the module offsets
0x18dc020, 0x18dc060 and 0x18dc0a0 (24 / 11 / 1 occurrences in one day) against the official 1.5.5 Windows CPython 3.13 build artefact matching the SHA256 above, and the caller of VCRUNTIME140!memcmp+0x50 in 1.4.4. Without PDBs every frame name I have is a nearest-export guess, so this is the single step that would turn the report into a diagnosis, and it is the one thing that cannot be done from outside. Publishing PDBs alongside the Windows wheels would let reporters do it themselves.
- Ship a version resource in
_duckdb*.pyd. version: 0.0.0.0 in Windows crash telemetry makes fleet-wide triage impossible without hashing every file by hand.
- Please treat the 1.4.4 and 1.5.5 reports as one defect unless symbolication says otherwise — see the section above; bucket-based dedup will split them.
If the engine frames (ExpressionExecutor, StreamQueryResult::ExecuteTask) make this a core issue rather than a client one, please move or retarget it — I filed here because the crashing artefact is the wheel's binary and asks 1 and 2 are wheel-build concerns.
OS:
Windows 11 Pro build 22631 (10.0.22631), x86_64, 8 cores, 63.6 GB RAM. Also reproduced on a second Windows x86_64 machine.
DuckDB Package Version:
1.5.5 (also reproduced on 1.4.4). Latest release tested: yes, 1.5.5. Nightly: not tested.
Python Version:
3.13.15 (MSC v.1944, 64-bit) on the 1.5.5 host; 3.13.5 (MSC v.1943, 64-bit) on the 1.4.4 host.
Full Name:
As on my GitHub profile (this account is my own, under my real first name).
Affiliation:
Independent — no corporate affiliation.
Did you include all relevant data sets for reproducing the issue?
No - I cannot share the data sets because they are confidential
Did you include all code required to reproduce the issue?
No — I have no minimal reproducer, only a statistical reproduction of a private workload plus minidumps and stacks. Stated plainly rather than filling the checkbox.
Did you include all relevant configuration to reproduce the issue?
Yes, I have
What happens?
A multi-process worker pool doing concurrent read-only queries against a DuckDB file with the
spatialextension crashes repeatedly withSTATUS_ACCESS_VIOLATION(0xC0000005) inside the DuckDB extension binary. There is no interrupt, no signal, no cancellation, and no writer: the worker processes simply die mid-query and the supervisor respawns them.It reproduces on two different machines, two DuckDB versions (1.4.4 and 1.5.5), two CPython patch levels and two unrelated workloads. I have minidumps and
cdbstacks for both versions, and I believe the 1.4.4 and 1.5.5 crashes are one defect even though Windows Error Reporting buckets them as two (see "One defect, two WER buckets" below).I cannot supply a minimal reproducer and I cannot get past this without maintainer-side symbolication, because the published Windows wheel ships no PDBs. Concrete asks are at the bottom.
Frequency
On the 1.5.5 host, a single ~7 minute run of the workload produced 14
Application Error/ Event ID 1000 records forpython.exe, and 38 in one day. Each crash kills one worker within seconds of that worker starting; the parent survives, so from the outside "nothing is down".Fault site (1.5.5)
Exception
c0000005,INVALID_POINTER_READ, faulting module_duckdb.cp313-win_amd64.pyd.The
mov al,[rcx]/cmp al,[rcx+rdx]pair is the shape of an inlined byte-compare loop. Noterdxholds a sign-extended-negative value where a pointer or offset argument belongs.Caveat on the reported fault address:
cdbprintsAttempt to read from address ffffffffffffffff, but that is its rendering of an address the minidump did not record. The actual bad pointer isrcx = 0x00028910ebfa612f.Fault offsets observed in one day on this host (Event-1000
Properties[7]):0x00000000018dc0200x00000000018dc0600x00000000018dc0a00x000000000058b2faThree fault sites
0x40apart inside one region is consistent with an unrolled byte-compare loop.Stack, 1.5.5 (faulting thread, index 27 of 47). Every
duckdb::name below is the nearest preceding export plus a large offset — the wheel ships no PDBs, so the module attribution is definitive but the function names are almost certainly not the functions executing. Treat repeated names as "same region of the binary".Fault site (1.4.4, a different machine)
Second-sourced: an independently captured 1.4.4 minidump from 2026-08-31 gives the same
VCRUNTIME140!memcmp+0x50and the same bucket.One defect, two WER buckets
I would ask you not to dedup these apart on the bucket names. Evidence that they are the same defect:
MetadataBlock::operator=x3 -> a mid-level frame ->ExpressionExecutor::ExecuteExpression->FunctionBinder::BindSortedAggregate->InterruptState::Callbackx2.ExpressionExecutor::ExecuteExpression+0x5be (1.4.4) vs +0x5b1 (1.5.5) — 13 bytes apart. Hedged: this only holds if the nearest-preceding export is the same real function in both builds, which is unverifiable without PDBs.VCRUNTIME140!memcmp(mov rax,[rcx], 8-byte); 1.5.5 inlines the compare into the extension (mov al,[rcx]/cmp al,[rcx+rdx], 1-byte), which is exactly why the bucket flips fromVCRUNTIME140.dll!memcmpto..._duckdb...pyd!Unknown. The inlining is a build difference, not a different bug.memcmp(a, b, n)takesrcx=a, rdx=b, r8=n.r8is a plausible length in both (0x24= 36,0x1f00= 7936), andrdxis a sign-extended-negative value in both, withrbp=0andrdi=1in both.Thread kinds
In 1.5.5 the fault is on the Python-owned thread, inside
StreamQueryResult::ExecuteTaskand bottoming out throughpython313. In the 1.4.4 dump it is on a native thread not owned by CPython (ucrtbase!thread_start->kernel32!BaseThreadInitThunk->ntdll!RtlUserThreadStart, all genuinely resolved symbols). So it faults on both thread kinds. Inference, not measurement: that fits shared state being read after free or realloc rather than a bug local to one thread.Ruled out on my side
Each of these was tested by changing it and re-running the identical workload; none of them changed the crash rate in the direction the hypothesis predicts:
memory_limit. Raised from 1.5 GB to 5 GB with the process cap at 12288 MB: crashes arrived sooner and more often (1.80/min vs 0.59/min). No OOM line, no eviction, no memory-pressure restart anywhere in the pool output.threads. Set to1machine-wide and verified inside the live processes' environment blocks: all three fresh workers still died within 2 minutes..walbeside any of the attached database files during the window, and no other process had them open.Possibly related, but I do not think it is the same
duckdb/duckdb#25011 is also
0xC0000005on Windows withspatialon 1.5.5, but it is interrupt-triggered under Jupyter and the reporter traced it to EDR hooks and the ipykernel Win32 interrupt event. Mine has no interrupt, no notebook, and no EDR involvement, and it also occurs on 1.4.4. Different mechanism, possibly adjacent code.To Reproduce
I do not have a minimal reproducer, and I want to be straight about that rather than post a script that does not fail. What I have is a reliable statistical reproduction of my own workload, plus dumps.
The workload, described generically:
spatialextension loaded, while a batch job drives them.ATTACHin RW mode, noINSTALLduring the run, no interrupts or cancellations.Instrumentation that makes it visible (a completed batch is not evidence of health — the pool respawns in place and the port stays listening):
Capturing dumps, in case it helps anyone reproducing on Windows: set
HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\python.exewithDumpFolder,DumpType=1,DumpCount=N.DumpType=2(full memory) silently writes nothing for job-capped worker processes, becauseWerFaultis spawned as a child of the crashing process and inherits its commit cap. A minidump is sufficient for this defect. Analyse withcdb -z <dump> -y srv*<cache>*https://msdl.microsoft.com/download/symbols -i <dir holding a copy of the pyd>— without-ithe module cannot be resolved at all.What I can supply on request: three 1.5.5 minidumps (14.8 / 15.7 / 16.9 MB), the exact
.pydby hash below, the 1.4.4 minidumps, and fullcdbtranscripts. I have not attached the dumps here because minidumps carry memory fragments from a business workload.Binary identity
The Windows wheel's extension binary carries no version resource —
FileVersionandProductVersionare both empty — which is why Windows crash telemetry reportsversion: 0.0.0.0for every one of these crashes. It has to be identified by hash:Runtime identity on the 1.5.5 host:
What I am asking for
0x18dc020,0x18dc060and0x18dc0a0(24 / 11 / 1 occurrences in one day) against the official 1.5.5 Windows CPython 3.13 build artefact matching the SHA256 above, and the caller ofVCRUNTIME140!memcmp+0x50in 1.4.4. Without PDBs every frame name I have is a nearest-export guess, so this is the single step that would turn the report into a diagnosis, and it is the one thing that cannot be done from outside. Publishing PDBs alongside the Windows wheels would let reporters do it themselves._duckdb*.pyd.version: 0.0.0.0in Windows crash telemetry makes fleet-wide triage impossible without hashing every file by hand.If the engine frames (
ExpressionExecutor,StreamQueryResult::ExecuteTask) make this a core issue rather than a client one, please move or retarget it — I filed here because the crashing artefact is the wheel's binary and asks 1 and 2 are wheel-build concerns.OS:
Windows 11 Pro build 22631 (10.0.22631), x86_64, 8 cores, 63.6 GB RAM. Also reproduced on a second Windows x86_64 machine.
DuckDB Package Version:
1.5.5 (also reproduced on 1.4.4). Latest release tested: yes, 1.5.5. Nightly: not tested.
Python Version:
3.13.15 (MSC v.1944, 64-bit) on the 1.5.5 host; 3.13.5 (MSC v.1943, 64-bit) on the 1.4.4 host.
Full Name:
As on my GitHub profile (this account is my own, under my real first name).
Affiliation:
Independent — no corporate affiliation.
Did you include all relevant data sets for reproducing the issue?
No - I cannot share the data sets because they are confidential
Did you include all code required to reproduce the issue?
No — I have no minimal reproducer, only a statistical reproduction of a private workload plus minidumps and stacks. Stated plainly rather than filling the checkbox.
Did you include all relevant configuration to reproduce the issue?
Yes, I have