Skip to content

fix: keep the dead-letter queue on in the shipped loader configs - #199

Merged
catinspace-au merged 1 commit into
mainfrom
fix/loader-dlq-on
Oct 4, 2026
Merged

catinspace-au merged 1 commit into
mainfrom
fix/loader-dlq-on

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

Every loader config here set routing.dlq.enabled: false, so a row ClickHouse refused was dropped and counted lost: No working DLQ for rows only a DLQ can take -- rows lost. On the single profile that was every rejected row.

With the override gone the loader's own default applies. On the Kafka profiles dead letters go to the dfe_loader_dlq topic dfe-engine already bootstraps, with the spool file behind it. On the gRPC profiles, which have no broker, they go to /var/spool/dfe/dlq/loader/dlq.ndjson on the dlq-spool volume dlq-init already makes writable.

Reproduced with the rc.14 loader image (dfe-loader@sha256:26e3e438...), ClickHouse and one Kafka broker, and a row the loader refuses (related_ip as one string into Array(IPv6)):

  • config/loader/kafka.yaml as on main: No working DLQ ... rows lost, dfe_loader_rows_lost_total 1, messages_dlq_total 0.
  • this branch's kafka.yaml: DLQ enabled mode=Cascade kafka_backend=true file_backend=true at startup, dfe_loader_rows_lost_total 0, messages_dlq_total 1, and the dead letter read back off dfe_loader_dlq with its source offset and payload.
  • the file backend alone, as the gRPC profiles get it: the dead letter in /var/spool/dfe/dlq/loader/dlq.ndjson, written by appuser.

docs/troubleshooting.md now says where the loader's dead letters go and what dfe_loader_rows_lost_total means. make check-compose check-hardfail check-docs check-python check-tests passes, 768 tests.

Every loader config here set routing.dlq.enabled: false, so a row ClickHouse refused was dropped and counted lost ("No working DLQ ... rows lost"). With the override gone the loader's default applies: the dfe_loader_dlq topic on the Kafka profiles, with the spool file behind it, and /var/spool/dfe/dlq/loader on the gRPC ones, which dlq-init already makes writable. The troubleshooting doc says where the loader's dead letters go.
@catinspace-au
catinspace-au merged commit cd65212 into main Oct 4, 2026
7 checks passed
@catinspace-au
catinspace-au deleted the fix/loader-dlq-on branch October 4, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant