Skip to content

Sync stalls indefinitely mid-file after first TUS chunk succeeds – no error, no retry, no progress #3283

Description

@peter-git-ctrl

I hit this same issue during a bulk migration (~37,500 files, mixed sizes from a few bytes to ~280MB) and did a fairly deep dive to rule out hardware/infra causes. Sharing findings in case it helps triage.

Setup: self-hosted, opencloudeu/opencloud-rolling:7.3.0 (compiled 2026-07-14), docker compose, decomposeds3 driver against a self-hosted MinIO instance on a separate host on the same LAN (sub-0.2ms ping). Desktop client (mirall-based) syncing a personal space.

What I ruled out:

CPU: monitored docker stats continuously during active chunk writes — never sustained above ~20%, mostly under 10%, on a 2-core VM.
Disk: raw dd write test on the same volume the upload staging dir lives on: 2.1 GB/s. Not the bottleneck.
RAM: gave the container's VM generous headroom (2GB+), no swapping, no OOM.
Network: same-LAN, <0.2ms latency between the app server and the MinIO backend.
S3 backend load: initially found MinIO itself CPU-starved (2 vCPUs on its own VM, pegged at 165-180%), fixed by giving it more cores — this genuinely helped one class of slowness, but a large residual slowdown remained afterward.
Two separate problems I found, in case they're actually different bugs conflated under "slow":

Unbounded stale upload-session backlog. opencloud storage-users uploads sessions --processing showed 78,276 sessions stuck in processing: true, of which 76,314 were already past their Expires timestamp — accumulated over a single day of heavy uploads, many for trivially small files (129-byte .d.ts files etc.) that should have finalized in milliseconds. Running the documented opencloud storage-users uploads sessions --expired --clean dropped this from 78,276 → 3,045 and eliminated all nats: timeout / unexpected EOF errors that were previously occurring during concurrent chunk writes (was ~15-30 occurrences per 5-minute window before cleanup, zero after). This also explains "resets after page reload" reports elsewhere — a client restart doesn't touch this server-side backlog, but a --clean does. Might be worth an automatic/scheduled cleanup job for expired sessions rather than requiring a manual CLI run, since in my case this accumulated to 78k+ entries within a single day of continuous uploading.

Residual per-file overhead independent of size, even after the backlog cleanup. After the session cleanup, no more errors, but individual TUS chunk-write requests (10MB fixed client chunk size) still consistently took 3-7 seconds — i.e. roughly 1.5-3 MB/s effective throughput — for chunks written to a local SSD path that benchmarks at 2.1 GB/s raw, with the app server CPU idle throughout. This doesn't scale with chunk size in an obvious way (didn't see faster completion for smaller residual chunks either, though I didn't get a clean controlled comparison). My best guess, not confirmed: the ~34 embedded oCIS services communicate over many small localhost network hops even within the single monolith process/container, and for a single file's upload+postprocessing pipeline (proxy → frontend → storage-users → auth → treesize propagation up the whole ancestor chain → event publish → ...) that adds up to real latency per file even though no individual resource is saturated. Happy to gather docker stats + service-level traces during a live repro if that would help narrow this down further — I didn't have request-level tracing enabled and didn't want to hold up the migration to set it up.

Let me know if a fresh issue with tracing enabled would be more useful than piling onto this one — happy to open one and link back.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status
    Qualification

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions