Add ingestionCompression to LogCollector and NonClusterHost, and compress fluent-bit log ingestion to Linseed by hjiawei · Pull Request #5224 · tigera/operator

hjiawei · 2026-08-19T23:22:34Z

Description

The fluent-bit outputs that ship logs to Linseed now compress each batch — zstd by default — cutting log ingestion bandwidth by roughly 12x measured on a kubeadm cluster, and both LogCollector and NonClusterHost gain an ingestionCompression field to control it.

ingestionCompression: None | Gzip | Zstd, default Zstd through the CRD (+kubebuilder:default), which the API server fills on every read — so no reconciler write-back is needed, matching the other CRD-defaulted fields in this API.
LogCollector drives the in-cluster outputs: the rendered compress follows the field, and None omits it so out_http posts plain NDJSON. Only the Linseed http outputs carry it — S3, Splunk, Syslog and OpenTelemetry are unaffected, asserted per output in the render test.
NonClusterHost drives non-cluster hosts, which read the resource directly and hot-reload the value. On a cluster whose CRD predates the field, hosts fall back to gzip, which every release's ingestion listener inflates.
The field is named for the ingestion path and "in transit" deliberately: felix already zstd-compresses rotated log files on disk.

Both zstd and gzip are compiled into the fluent-bit image, so nothing about the build changes. zstd is the default because it beat gzip on both axes on real Calico log data (43x vs 28x compression, roughly 6x less CPU); gzip is kept for symmetry with Linseed's accepted encodings and for hosts attached to older clusters.

Requires the Linseed-side change

Depends on tigera/calico-private#13367, which teaches Linseed to inflate zstd and gzip request bodies (and adds the host plugin and Go client sides). Ship together, or Linseed first: fluent-bit's out_http drops a chunk on a 4xx rather than retrying it.

Upgrade order

Clusters are upgraded before non-cluster host packages, and a management plane before its managed clusters — the same rule for MCM and Calico Cloud. A host at version N only meets a cluster at N or newer. ingestionCompression: None is the escape hatch.

CPU cost

Decompression costs Linseed roughly 5–20% more CPU per request (less for bigger batches), measured on a dev laptop and a cluster node (e2-standard-2). It runs at 488 MB/s of logs per node vCPU against about 30 MB/s for the NDJSON parsing that follows, so it is ~15x cheaper per byte than work Linseed already does. Sender-side cost is distributed across nodes: a 10KB batch encodes in 254us.

Release Note

Log ingestion to Linseed is compressed with zstd by default, reducing bandwidth by roughly 12x. The new `ingestionCompression` field on LogCollector and NonClusterHost (None, Gzip, Zstd) controls it; upgrade the cluster before non-cluster host packages.

For PR author

Tests: render test asserts every Linseed output carries the configured value (Gzip / None / default) and no other output does; controller tests cover that an explicit value is left untouched; golden files regenerated with UPDATE_RENDERED_CONFIGS=1.
make gen-files run — CRDs and deepcopy regenerated for the new field.
make gen-versions — n/a, no version change.

Validated end to end on a real cluster: the new operator asserted the CRDs and both resources read Zstd; in-cluster outputs rendered compress: zstd; setting None re-rendered the ConfigMap without it and ingestion stayed clean (38 bulk responses, "failed":0); non-cluster hosts hot-reloaded on every field change.

AI assistance: Written with Claude Code (Claude Fable 5); design decisions, review and cluster validation by the author.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

api/v1/zz_generated.deepcopy.go: Generated file

+	if v := compressValue(c.cfg.LogCollector.Spec.IngestionCompression); v != "" {
+		out["compress"] = v


+	if instance.Spec.IngestionCompression == nil {
+		// The CRD default only applies on write, so materialize it for CRs
+		// created before the field existed.
+		compression := operatorv1.IngestionCompressionZstd
+		instance.Spec.IngestionCompression = &compression
+		modifiedFields = append(modifiedFields, "IngestionCompression")


+	if instance.Spec.IngestionCompression == nil {
+		// Hosts read this CR directly and treat an unset field as None, and
+		// the CRD default only applies on write, so materialize the default
+		// for CRs created before the field existed.
+		preDefaultPatchFrom := client.MergeFrom(instance.DeepCopy())


+	// Default: Zstd
+	// +optional
+	// +kubebuilder:default=Zstd
+	IngestionCompression *IngestionCompressionOption `json:"ingestionCompression,omitempty"`


Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

api/v1/zz_generated.deepcopy.go: Generated file

Suppressed comments (2)

pkg/enterprise/render/logcollector/pipeline.go:432

Enabling compression here as soon as this operator runs can lose logs during an upgrade. The LogCollector controller rolls Fluent Bit independently of the Linseed controller; Linseed's rolling deployment can still route requests to old ready pods, and managed clusters may target a separately upgraded management cluster. Because an unsupported encoding returns a 4xx and out_http drops that chunk, shipping both changes in one release does not provide the required ordering. Gate this on verified receiver compatibility, or retain an uncompressed default until compatibility is guaranteed.

	if v := compressValue(c.cfg.LogCollector.Spec.IngestionCompression); v != "" {
		out["compress"] = v

api/v1/logcollector_types.go:51

The PR description calls this a config-only change and says compress is the only addition, but this line introduces a new public LogCollector API field; the PR also adds a shared enum, a NonClusterHost field, controller defaulting, and generated CRD changes. Update the description and API-change checklist so reviewers and release tooling account for the actual public API scope.

	// +kubebuilder:default=Zstd
	IngestionCompression *IngestionCompressionOption `json:"ingestionCompression,omitempty"`

+	// +kubebuilder:default=Zstd
+	IngestionCompression *IngestionCompressionOption `json:"ingestionCompression,omitempty"`


The linseed http outputs post plain NDJSON. Setting compress makes fluent-bit zstd-encode each batch, which cuts ingest bandwidth by roughly 12x on real traffic. Both codecs are already compiled into the fluent-bit image, so this is config only. Requires the Linseed-side decompression from calico-private (EV-6973); those must ship together, since fluent-bit drops a chunk on a 4xx rather than retrying it. EV-6973

Compression for the fluent-bit to Linseed ingestion path was hardcoded to zstd in the rendered config, with no API to turn it off. Both resources gain ingestionCompression: None | Gzip | Zstd, defaulting to Zstd through the CRD, which fills the value on every read. LogCollector drives the in-cluster outputs: the rendered compress option follows the field, and None omits it so out_http posts plain NDJSON. NonClusterHost drives non-cluster hosts, which read the resource directly and hot-reload the value; on a cluster whose CRD predates the field they fall back to gzip, which every release's ingestion listener inflates. The field is named for the ingestion path and "in transit" deliberately: felix already zstd-compresses rotated log files on disk, and the additional stores (S3, Syslog, Splunk) and OpenTelemetry exporters are unaffected. EV-6973.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

hjiawei · 2026-09-02T21:51:11Z

Closing: this repo is no longer where operator changes land. tigera/operator has been imported into the Calico monorepos (tigera/calico-private#13606, projectcalico/calico#13674), both at operator/.

Both commits here have moved to tigera/calico-private#13367, which already carried the Linseed, host-plugin and Go-client sides of EV-6973 — so the change that had to ship together is now one PR in one repo, rather than two PRs with an ordering constraint between them.

The port is unchanged in substance; paths mapped as api/... → operator/api/..., pkg/... → operator/pkg/..., and pkg/imports/crds/operator/*.yaml → operator/pkg/crds/operator/*.yaml (the one directory the import relocated). Two things follow from the monorepo layout that did not apply here: the CRDs now reach charts/ through symlinks into operator/pkg/crds/operator, so no chart copy is needed, and manifests/{operator-crds,v1_crd_projectcalico_org,v3_projectcalico_org,v3_projectcalico_org-v1beta1}.yaml are generated copies that the CRD change regenerates in the same PR — replacing the old post-merge make get-operator-crds flow.

Nothing goes to projectcalico/calico: LogCollector and NonClusterHost are Enterprise resources, and OSS ships neither CRD outside operator/pkg/crds/operator.

Review continues on tigera/calico-private#13367.

Copilot AI lite review requested due to automatic review settings August 19, 2026 23:22

hjiawei requested a review from a team as a code owner August 19, 2026 23:22

marvin-tigera added this to the v1.45.0 milestone Aug 19, 2026

marvin-tigera added docs-pr-required release-note-required labels Aug 19, 2026

Copilot AI reviewed Aug 19, 2026

hjiawei added the docs-not-required label Aug 19, 2026

marvin-tigera removed the docs-pr-required label Aug 19, 2026

hjiawei closed this Aug 19, 2026

hjiawei reopened this Aug 19, 2026

marvin-tigera added docs-pr-required release-note-required and removed release-note-required docs-not-required labels Aug 19, 2026

hjiawei requested a balanced review from Copilot August 20, 2026 05:01

Copilot AI reviewed Aug 20, 2026

hjiawei force-pushed the linseed-ingest-compression branch from 1cb24e7 to da81c9f Compare August 20, 2026 17:28

Copilot AI review requested due to automatic review settings August 20, 2026 17:28

hjiawei requested a review from marvin-tigera as a code owner August 20, 2026 17:28

Copilot AI reviewed Aug 20, 2026

Copilot AI review requested due to automatic review settings September 1, 2026 19:19

hjiawei force-pushed the linseed-ingest-compression branch from da81c9f to e7a1c7c Compare September 1, 2026 19:19

Copilot started reviewing on behalf of hjiawei September 1, 2026 19:20 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 1, 2026 21:18

hjiawei force-pushed the linseed-ingest-compression branch from e7a1c7c to 2317d83 Compare September 1, 2026 21:18

Copilot started reviewing on behalf of hjiawei September 1, 2026 21:19 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread api/v1/nonclusterhost_types.go

Comment on lines +37 to +38

// +kubebuilder:default=Zstd

IngestionCompression *IngestionCompressionOption `json:"ingestionCompression,omitempty"`

hjiawei added 2 commits September 2, 2026 12:54

Copilot AI review requested due to automatic review settings September 2, 2026 19:58

hjiawei force-pushed the linseed-ingest-compression branch from 2317d83 to 5647600 Compare September 2, 2026 19:58

Copilot AI reviewed Sep 2, 2026

hjiawei changed the title ~~Compress fluent-bit log batches sent to Linseed~~ Add ingestionCompression to LogCollector and NonClusterHost, and compress fluent-bit log ingestion to Linseed Sep 2, 2026

hjiawei closed this Sep 2, 2026

marvin-tigera removed docs-pr-required release-note-required labels Sep 2, 2026

hjiawei deleted the linseed-ingest-compression branch September 2, 2026 21:55

Navigation Menu

Uh oh!

Add ingestionCompression to LogCollector and NonClusterHost, and compress fluent-bit log ingestion to Linseed - #5224

Add ingestionCompression to LogCollector and NonClusterHost, and compress fluent-bit log ingestion to Linseed#5224
hjiawei wants to merge 2 commits into
tigera:masterfrom
hjiawei:linseed-ingest-compression

hjiawei commented Aug 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

hjiawei commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if v := compressValue(c.cfg.LogCollector.Spec.IngestionCompression); v != "" {
		out["compress"] = v

		// +kubebuilder:default=Zstd
		IngestionCompression *IngestionCompressionOption `json:"ingestionCompression,omitempty"`

Uh oh!

Conversation

hjiawei commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Requires the Linseed-side change

Upgrade order

CPU cost

Release Note

For PR author

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

hjiawei commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hjiawei commented Aug 19, 2026 •

edited

Loading