fix: keep the dead-letter queue on in the shipped loader configs - #199
Merged
Merged
Conversation
Every loader config here set routing.dlq.enabled: false, so a row ClickHouse refused was dropped and counted lost ("No working DLQ ... rows lost"). With the override gone the loader's default applies: the dfe_loader_dlq topic on the Kafka profiles, with the spool file behind it, and /var/spool/dfe/dlq/loader on the gRPC ones, which dlq-init already makes writable. The troubleshooting doc says where the loader's dead letters go.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every loader config here set
routing.dlq.enabled: false, so a row ClickHouse refused was dropped and counted lost:No working DLQ for rows only a DLQ can take -- rows lost. On the single profile that was every rejected row.With the override gone the loader's own default applies. On the Kafka profiles dead letters go to the
dfe_loader_dlqtopic dfe-engine already bootstraps, with the spool file behind it. On the gRPC profiles, which have no broker, they go to/var/spool/dfe/dlq/loader/dlq.ndjsonon thedlq-spoolvolumedlq-initalready makes writable.Reproduced with the rc.14 loader image (
dfe-loader@sha256:26e3e438...), ClickHouse and one Kafka broker, and a row the loader refuses (related_ipas one string intoArray(IPv6)):config/loader/kafka.yamlas on main:No working DLQ ... rows lost,dfe_loader_rows_lost_total 1,messages_dlq_total 0.kafka.yaml:DLQ enabled mode=Cascade kafka_backend=true file_backend=trueat startup,dfe_loader_rows_lost_total 0,messages_dlq_total 1, and the dead letter read back offdfe_loader_dlqwith its source offset and payload./var/spool/dfe/dlq/loader/dlq.ndjson, written by appuser.docs/troubleshooting.mdnow says where the loader's dead letters go and whatdfe_loader_rows_lost_totalmeans.make check-compose check-hardfail check-docs check-python check-testspasses, 768 tests.