I hit this same issue during a bulk migration (~37,500 files, mixed sizes from a few bytes to ~280MB) and did a fairly deep dive to rule out hardware/infra causes. Sharing findings in case it helps triage.
Setup: self-hosted, opencloudeu/opencloud-rolling:7.3.0 (compiled 2026-07-14), docker compose, decomposeds3 driver against a self-hosted MinIO instance on a separate host on the same LAN (sub-0.2ms ping). Desktop client (mirall-based) syncing a personal space.
What I ruled out:
CPU: monitored docker stats continuously during active chunk writes — never sustained above ~20%, mostly under 10%, on a 2-core VM.
Disk: raw dd write test on the same volume the upload staging dir lives on: 2.1 GB/s. Not the bottleneck.
RAM: gave the container's VM generous headroom (2GB+), no swapping, no OOM.
Network: same-LAN, <0.2ms latency between the app server and the MinIO backend.
S3 backend load: initially found MinIO itself CPU-starved (2 vCPUs on its own VM, pegged at 165-180%), fixed by giving it more cores — this genuinely helped one class of slowness, but a large residual slowdown remained afterward.
Two separate problems I found, in case they're actually different bugs conflated under "slow":
Unbounded stale upload-session backlog. opencloud storage-users uploads sessions --processing showed 78,276 sessions stuck in processing: true, of which 76,314 were already past their Expires timestamp — accumulated over a single day of heavy uploads, many for trivially small files (129-byte .d.ts files etc.) that should have finalized in milliseconds. Running the documented opencloud storage-users uploads sessions --expired --clean dropped this from 78,276 → 3,045 and eliminated all nats: timeout / unexpected EOF errors that were previously occurring during concurrent chunk writes (was ~15-30 occurrences per 5-minute window before cleanup, zero after). This also explains "resets after page reload" reports elsewhere — a client restart doesn't touch this server-side backlog, but a --clean does. Might be worth an automatic/scheduled cleanup job for expired sessions rather than requiring a manual CLI run, since in my case this accumulated to 78k+ entries within a single day of continuous uploading.
Residual per-file overhead independent of size, even after the backlog cleanup. After the session cleanup, no more errors, but individual TUS chunk-write requests (10MB fixed client chunk size) still consistently took 3-7 seconds — i.e. roughly 1.5-3 MB/s effective throughput — for chunks written to a local SSD path that benchmarks at 2.1 GB/s raw, with the app server CPU idle throughout. This doesn't scale with chunk size in an obvious way (didn't see faster completion for smaller residual chunks either, though I didn't get a clean controlled comparison). My best guess, not confirmed: the ~34 embedded oCIS services communicate over many small localhost network hops even within the single monolith process/container, and for a single file's upload+postprocessing pipeline (proxy → frontend → storage-users → auth → treesize propagation up the whole ancestor chain → event publish → ...) that adds up to real latency per file even though no individual resource is saturated. Happy to gather docker stats + service-level traces during a live repro if that would help narrow this down further — I didn't have request-level tracing enabled and didn't want to hold up the migration to set it up.
Let me know if a fresh issue with tracing enabled would be more useful than piling onto this one — happy to open one and link back.
I hit this same issue during a bulk migration (~37,500 files, mixed sizes from a few bytes to ~280MB) and did a fairly deep dive to rule out hardware/infra causes. Sharing findings in case it helps triage.
Setup: self-hosted, opencloudeu/opencloud-rolling:7.3.0 (compiled 2026-07-14), docker compose, decomposeds3 driver against a self-hosted MinIO instance on a separate host on the same LAN (sub-0.2ms ping). Desktop client (mirall-based) syncing a personal space.
What I ruled out:
CPU: monitored docker stats continuously during active chunk writes — never sustained above ~20%, mostly under 10%, on a 2-core VM.
Disk: raw dd write test on the same volume the upload staging dir lives on: 2.1 GB/s. Not the bottleneck.
RAM: gave the container's VM generous headroom (2GB+), no swapping, no OOM.
Network: same-LAN, <0.2ms latency between the app server and the MinIO backend.
S3 backend load: initially found MinIO itself CPU-starved (2 vCPUs on its own VM, pegged at 165-180%), fixed by giving it more cores — this genuinely helped one class of slowness, but a large residual slowdown remained afterward.
Two separate problems I found, in case they're actually different bugs conflated under "slow":
Unbounded stale upload-session backlog. opencloud storage-users uploads sessions --processing showed 78,276 sessions stuck in processing: true, of which 76,314 were already past their Expires timestamp — accumulated over a single day of heavy uploads, many for trivially small files (129-byte .d.ts files etc.) that should have finalized in milliseconds. Running the documented opencloud storage-users uploads sessions --expired --clean dropped this from 78,276 → 3,045 and eliminated all nats: timeout / unexpected EOF errors that were previously occurring during concurrent chunk writes (was ~15-30 occurrences per 5-minute window before cleanup, zero after). This also explains "resets after page reload" reports elsewhere — a client restart doesn't touch this server-side backlog, but a --clean does. Might be worth an automatic/scheduled cleanup job for expired sessions rather than requiring a manual CLI run, since in my case this accumulated to 78k+ entries within a single day of continuous uploading.
Residual per-file overhead independent of size, even after the backlog cleanup. After the session cleanup, no more errors, but individual TUS chunk-write requests (10MB fixed client chunk size) still consistently took 3-7 seconds — i.e. roughly 1.5-3 MB/s effective throughput — for chunks written to a local SSD path that benchmarks at 2.1 GB/s raw, with the app server CPU idle throughout. This doesn't scale with chunk size in an obvious way (didn't see faster completion for smaller residual chunks either, though I didn't get a clean controlled comparison). My best guess, not confirmed: the ~34 embedded oCIS services communicate over many small localhost network hops even within the single monolith process/container, and for a single file's upload+postprocessing pipeline (proxy → frontend → storage-users → auth → treesize propagation up the whole ancestor chain → event publish → ...) that adds up to real latency per file even though no individual resource is saturated. Happy to gather docker stats + service-level traces during a live repro if that would help narrow this down further — I didn't have request-level tracing enabled and didn't want to hold up the migration to set it up.
Let me know if a fresh issue with tracing enabled would be more useful than piling onto this one — happy to open one and link back.