Description
Following an upgrade of embedded Kubernetes/apimachinery libraries to v0.36.0+, kapp-controller's custom API server fails to handle certain watch requests targeting the packaging.carvel.dev group. The requests are rejected by the API aggregation machinery with a 422 Unprocessable Entity error.
Root Cause Analysis
The Kubernetes v0.36.0 toolchain upgrade brings native WatchList features (GA) enabled by default. Under this new API contract, when a client initializes a watch request with watch=1&resourceVersion=0, the API machinery injects default list options, enforcing ResourceVersionMatch=NotOlderThan.
The downstream API server validation logic (metav1.ValidateListOptions) now strictly mandates that if ResourceVersionMatch is set to NotOlderThan, the SendInitialEvents field must be non-nil.
However, inside kapp-controller, the custom REST storage translators map incoming options to external options using internalToMetaListOpts. This mapping logic was omitting the SendInitialEvents field:
// Affected files:
// - pkg/apiserver/registry/datapackaging/package_crd_rest.go
// - pkg/apiserver/registry/datapackaging/package_metadata_crd_rest.go
func (r *PackageCRDREST) internalToMetaListOpts(options internalversion.ListOptions) metav1.ListOptions {
lo := metav1.ListOptions{
TypeMeta: options.TypeMeta,
Watch: options.Watch,
AllowWatchBookmarks: options.AllowWatchBookmarks,
ResourceVersion: options.ResourceVersion,
ResourceVersionMatch: options.ResourceVersionMatch,
TimeoutSeconds: options.TimeoutSeconds,
Limit: options.Limit,
Continue: options.Continue,
// SendInitialEvents pointer is silently dropped here
}
// ...
return lo
}
Because the pointer is stripped and sent as nil, it violates the API contract expected by kube-apiserver, triggering the 422 validation failure.
Expected Behavior
The internalToMetaListOpts conversion utility should explicitly map and propagate the SendInitialEvents pointer from internalversion.ListOptions to metav1.ListOptions to maintain compatibility with modern Kubernetes control planes.
Proposed Fix
Pass the SendInitialEvents property explicitly through the struct initialization block inside both PackageCRDREST and PackageMetadataCRDREST.
(Note: I have already opened a pull request to resolve this issue and added corresponding contract unit tests to verify property propagation).
Description
Following an upgrade of embedded Kubernetes/apimachinery libraries to v0.36.0+, kapp-controller's custom API server fails to handle certain watch requests targeting the packaging.carvel.dev group. The requests are rejected by the API aggregation machinery with a 422 Unprocessable Entity error.
Root Cause Analysis
The Kubernetes v0.36.0 toolchain upgrade brings native WatchList features (GA) enabled by default. Under this new API contract, when a client initializes a watch request with watch=1&resourceVersion=0, the API machinery injects default list options, enforcing ResourceVersionMatch=NotOlderThan.
The downstream API server validation logic (metav1.ValidateListOptions) now strictly mandates that if ResourceVersionMatch is set to NotOlderThan, the SendInitialEvents field must be non-nil.
However, inside kapp-controller, the custom REST storage translators map incoming options to external options using internalToMetaListOpts. This mapping logic was omitting the SendInitialEvents field:
Because the pointer is stripped and sent as nil, it violates the API contract expected by kube-apiserver, triggering the 422 validation failure.
Expected Behavior
The internalToMetaListOpts conversion utility should explicitly map and propagate the SendInitialEvents pointer from internalversion.ListOptions to metav1.ListOptions to maintain compatibility with modern Kubernetes control planes.
Proposed Fix
Pass the SendInitialEvents property explicitly through the struct initialization block inside both PackageCRDREST and PackageMetadataCRDREST.
(Note: I have already opened a pull request to resolve this issue and added corresponding contract unit tests to verify property propagation).