Repository navigation
Filtering ONLY own events in read-after-write-consitency #3398
Description
When doing updates with read-after-write-consistency we are now filtering events that happened before our update. This is causing mainly now a practical issue only for primary resource:
quarkiverse/quarkus-operator-sdk#1353
however, for primary resource it could happen that while we update the status some changes the spec, and that we would like to reconcile. So for primary we should just filter only changes for our updates. But similar issue(s) scenarios might happen for the secondary resources see below.
At the end we will be on safe side to propagate intermediate events for changes that happen during our update. Thus only filter the event for our resource change, where the new resource version is the same as the resource version received after our update request.
Note that this is a bit harder than it seems, since relist might happen in informer for that we might need a small patch to fabric8 client, if that happens we need to propagate events also for our update since an event might be lost.
The problematic cases:
Primary resource spec update during status update
When we update the status at the end of the reconciliation, we might receive an update just before for the spec, that spec change now will be filtered out, so reconciliation won't be triggered for it.
Workaround
For now users can simply use the client to patch the status of the primary resource or update update secondary resources:
context.getClient().resource(webPage).patchStatus();That will not filter our the events.
Or downgrade to v 5.2.x
Dependency between managed resources
Let's say we have a primary P that manages a resource A and there are other controllers on the cluster, that patches that resource A also. Assuming that in the controller we reconcile the resource A and that we should take into account properties of A that are managed by an other controller (think SSA). So in this case we have similar situation as with primary resource (above), that if we do an update on resource A and before that it just get updated (but after we calculated the target state) we don't get triggered by again for that update, however that is needed to get into desired state.
Edge case with re-list
Note that if a a 3rd party did an update on the resource just before our update but informer was down and had to re-list we might not see that event. To fix this we would need and additional callback before re-list and propagate events for all changed resources since the web socket if informer was down until the most recent resource version from list.
Notes
- Note that both of these scenarios might be handled currently by the user with some (if user uses the the caching and filtering update directly not using
UpdateControl) since user sees the up-to-date resource after updates. But for that user might require to check the resource after the update; what is quite strong requirement and it goes against the current mindset in the operator pattern.
Activity
csviri commentedon Jun 7, 2026
cc @shawkins (if you are available please join the community meeting tomorrow)
robobario commentedon Jun 8, 2026
Hi, just chiming in to say I think we're experience this in our Operator using v5.3.4, we observe that some primary reconciliations aren't triggered when we patch the status of our resource with SSA, while concurrently some third party is updating the resource. We end up with the latest generation not triggering a reconciliation.
Pinning down to 5.2.4 looks like it should prevent this in the meanwhile, is that correct? edit: nope test still failing at 5.2.4.
Downgrade JOSDK to 5.2.4
fix(operator): force reschedule to guard against missed generation up…
csviri commentedon Jun 8, 2026
@robobario a workaround for now would be to use the kubernetes client directly to update the resource not
UpdateControl.patchStatusor downgrade to5.2.x. But will provide a fix for this soon, that will solve the issue, some edge case will require a tweak to fabric8 client too, will let you know on updated on this issue.metacosm commentedon Jun 8, 2026
Maybe we're trying to be too smart for our own good…
csviri commentedon Jun 8, 2026
PR needed cover the edge case for re-list:
https://github.com/fabric8io/kubernetes-client/pull/7899/changes
csviri commentedon Jun 8, 2026
During the initial design actually we discussed this briefly with @shawkins, but we failed to see this scenario. An alternative would be to simply turn off the event filtering; However this way it could be solved relatively nicely, and IMO this is otherwise a great performance optimization.
csviri commentedon Jun 8, 2026
@robobario regarding:
That is probably a different issue, since this feature was released in
5.3.0. Could you please provide some more information regarding that?robobario commentedon Jun 8, 2026
Yep I'll continue debugging and update if I find out more. A bit tricky to share something minimal/reproducible as it's happening in our heaviest system tests with multiple reconcilers and resources playing together.
robobario commentedon Jun 9, 2026
okay on that, I noticed that when I downgraded to 5.2.4 the build succeeded even though I didn't revert
operator-framework-junit-5tooperator-framework-junit(or whichever way it changed, I might have that mixed up), a bit suspicious since it looks like there was only a single class modification in 5.2.4I'm wondering if something has happened like 5.2.4 got released off main? Looking at the core jar I can see classes that aren't in the v5.2.4 tag
edit: I'll lift to a new issue, #3402
csviri commentedon Jun 9, 2026
Yes, there was I PR backported on top of
5.2.3(health check improvements) that resulted in5.2.4off the main release. @metacosm can tell more in case.it does not matter that much imo, it was renamed because of forward compatibility naming wise for junit framework versions.
csviri commentedon Jun 9, 2026
We are further discussing the new algorithm from here: #3399 (comment)