fix: read the external resource cache under the event source monitor - #3524
fix: read the external resource cache under the event source monitor#3524csviri wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes a concurrency correctness issue in ExternalResourceCachingEventSource by ensuring cache read paths take a synchronized snapshot of per-primary cached resources (the nested maps are HashMaps and are mutated under the event source monitor). This avoids races that could lead to corrupted reads or TOCTOU NullPointerExceptions.
Changes:
- Added a
cachedResourcesFor(ResourceID)helper that synchronizes on the event source and returns a snapshot copy of cached secondary resources. - Routed
getSecondaryResources(ResourceID)and the relevant overrides in polling/inbound event sources through the snapshot helper. - Documented that
getCache()returns a live view and that iterating nested maps requires synchronizing on the event source; added a regression test asserting snapshot behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/ExternalResourceCachingEventSource.java | Introduces synchronized snapshot helper for cache reads and documents thread-safety expectations of getCache(). |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PerResourcePollingEventSource.java | Uses cachedResourcesFor to safely snapshot cached resources when registering tasks and when serving getSecondaryResources. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/inbound/CachingInboundEventSource.java | Uses cachedResourcesFor to safely snapshot cached resources in getSecondaryResources. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/event/source/ExternalResourceCachingEventSourceTest.java | Adds a test verifying getSecondaryResources returns a snapshot, not a live view. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 36 changed files in this pull request and generated no new comments.
Suppressed comments (3)
sample-operators/pom.xml:39
- The PR title/description focus on synchronizing reads in
ExternalResourceCachingEventSource, but this change also adds a newsample-operators/kotlin-operatormodule (and updates CI to run it) along with several unrelated framework fixes/version bumps in other files. Consider splitting these concerns into separate PRs or updating the PR title/description to reflect the broader scope.
operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/ExternalResourceCachingEventSource.java:166 - Minor typo: the helper method name
acceptedByFilerappears to be meant asacceptedByFilter. Since this code is being touched, consider renaming the method and its call site to avoid perpetuating the typo.
.anyMatch(r -> acceptedByGenericFilter(r) && acceptedByOnAddFilter(r));
operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PerResourcePollingEventSource.java:126
ResourceFetcher.fetchDelayexplicitly distinguishesnull(no fetch happened yet) from an empty set (fetch happened but no resources were found).cachedResourcesFor(primaryID)returns an empty set for both "no cache entry" and "cached empty", and the currentcachedResources.isEmpty() ? null : cachedResourcescollapses the two cases, potentially changing polling backoff behavior for implementations that rely on the distinction. Consider using cache presence (under the same monitor) to decide whether to passnullvs an empty set.
var cachedResources = cachedResourcesFor(primaryID);
var actualResources = cachedResources.isEmpty() ? null : cachedResources;
// note that there is a delay, to not do two fetches when the resources first appeared
`ExternalResourceCachingEventSource` mutates its cache from `synchronized`
methods (`handleResources`, `handleDelete`,
`handleRecentResourceCreate/Update`), but the read paths were not
synchronized. The outer map is a `ConcurrentHashMap`; the nested
per-primary maps are plain `HashMap`s that `handleDelete` mutates in
place, so a reconciler thread reading them while a poll or informer
thread writes can observe a corrupted map or throw.
`getSecondaryResources(ResourceID)` additionally looked the primary up
twice:
var cachedValues = cache.get(primaryID);
if (cachedValues == null) { return Collections.emptySet(); }
else { return new HashSet<>(cache.get(primaryID).values()); }
If a concurrent `handleDelete` removes the entry between the two calls,
the second `get` returns null and this throws a NullPointerException.
Adds a `cachedResourcesFor` helper that snapshots the cached resources
while holding the monitor, and routes `getSecondaryResources` plus the
`PerResourcePollingEventSource` and `CachingInboundEventSource` overrides
(and `checkAndRegisterTask`) through it. The helper only copies, so the
potentially slow `ResourceFetcher` calls in those overrides still run
outside the lock and cannot block the informer or poll threads.
`getCache()` still returns a live view for backwards compatibility, but
now documents that iterating the nested maps requires synchronizing on the
event source.
Adds a test asserting `getSecondaryResources` returns a snapshot rather
than a live view.
…c filter is set (operator-framework#3518) `acceptedByFiler` guards each of its three filter branches with `onXFilter != null || genericFilter != null`, but the branch body dereferences `onXFilter` unconditionally: if (onAddFilter != null || genericFilter != null) { ... .anyMatch(r -> acceptedByGenericFiler(r) && onAddFilter.accept(r)); So configuring only a generic filter via `setGenericFilter(...)` throws a NullPointerException as soon as a resource is added, deleted or updated. All three branches (add / delete / update) are affected, which means `PollingEventSource`, `PerResourcePollingEventSource` and `CachingInboundEventSource` all break when used with a generic filter only. The existing `genericFilteringEvents` test missed this because it uses a filter that returns `false`: `&&` short-circuits before the null dereference. Only a generic filter that accepts a resource reaches the NPE. Each filter check is now null-safe (an absent filter accepts), which preserves the previous behaviour whenever the specific filter is set. Adds three regression tests, one per branch; they fail with NullPointerException without this change. Signed-off-by: Attila Mészáros <a_meszaros@apple.com> # Conflicts: # operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/event/source/ExternalResourceCachingEventSourceTest.java
b1a6772 to
5b9b6cd
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (2)
operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PerResourcePollingEventSource.java:126
- In
checkAndRegisterTask, convertingcachedResourcesFor(primaryID)tonullwhenisEmpty()loses the distinction between “no cached entry” and “cached but empty”. Previously, an existing empty per-primary cache map resulted in an emptySet(meaning “fetch happened but no resources were found”), but now it becomesnull(meaning “no fetch happened”), which can changeResourceFetcher.fetchDelaybehavior and scheduling.
if (scheduledFutures.get(primaryID) == null
&& (registerPredicate == null || registerPredicate.test(resource))) {
var cachedResources = cachedResourcesFor(primaryID);
var actualResources = cachedResources.isEmpty() ? null : cachedResources;
operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/event/source/ExternalResourceCachingEventSourceTest.java:223
- The new test
getSecondaryResourcesReturnsASnapshotNotALiveViewlikely passes even without this fix becausegetSecondaryResources(ResourceID)already returned a copiedHashSetbefore (so it was already a snapshot). To regress the actual TOCTOU/NPE scenario this PR fixes, consider a deterministic test that fails pre-fix by makingcache.get(primaryID)return non-null once andnullon the second call (simulating a delete between two lookups).
void getSecondaryResourcesReturnsASnapshotNotALiveView() {
source.handleResources(primaryID1(), Set.of(testResource1()));
var snapshot = source.getSecondaryResources(primaryID1());
source.handleDelete(primaryID1());
assertThat(snapshot).containsExactly(testResource1());
assertThat(source.getSecondaryResources(primaryID1())).isEmpty();
}
ExternalResourceCachingEventSourcemutates its cache fromsynchronizedmethods (
handleResources,handleDelete,handleRecentResourceCreate/Update), but the read paths were notsynchronized. The outer map is a
ConcurrentHashMap; the nestedper-primary maps are plain
HashMaps thathandleDeletemutates inplace, so a reconciler thread reading them while a poll or informer
thread writes can observe a corrupted map or throw.
getSecondaryResources(ResourceID)additionally looked the primary uptwice:
If a concurrent
handleDeleteremoves the entry between the two calls,the second
getreturns null and this throws a NullPointerException.Adds a
cachedResourcesForhelper that snapshots the cached resourceswhile holding the monitor, and routes
getSecondaryResourcesplus thePerResourcePollingEventSourceandCachingInboundEventSourceoverrides(and
checkAndRegisterTask) through it. The helper only copies, so thepotentially slow
ResourceFetchercalls in those overrides still runoutside the lock and cannot block the informer or poll threads.
getCache()still returns a live view for backwards compatibility, butnow documents that iterating the nested maps requires synchronizing on the
event source.
Adds a test asserting
getSecondaryResourcesreturns a snapshot ratherthan a live view.
Part of #3517