Repository navigation
where resources already exist but requests are generated by CREATE #2290
Description
Bug Report
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.23.0.1:443/apis/policy/v1/poddisruptionbudgets/os-osfarm-cluster-manager-pdb?fieldManager=opensearchreconciler&force=true. Message: Operation cannot be fulfilled on poddisruptionbudgets.policy "os-osfarm-cluster-manager-pdb": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=policy, kind=poddisruptionbudgets, name=os-osfarm-cluster-manager-pdb, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on poddisruptionbudgets.policy "os-osfarm-cluster-manager-pdb": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:713)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:232)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:237)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:252)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:1132)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:92)
at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.create(KubernetesDependentResource.java:140)
at io.javaoperatorsdk.operator.processing.dependent.kubernetes.CRUDKubernetesDependentResource.create(CRUDKubernetesDependentResource.java:16)
at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.handleCreate(AbstractDependentResource.java:114)
at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.handleCreate(KubernetesDependentResource.java:111)
at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.handleCreate(KubernetesDependentResource.java:32)
at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:62)
at io.javaoperatorsdk.operator.processing.dependent.SingleDependentResourceReconciler.reconcile(SingleDependentResourceReconciler.java:19)
at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:52)
at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileExecutor$NodeReconcileExecutor.doRun(WorkflowReconcileExecutor.java:115)
at io.javaoperatorsdk.operator.processing.dependent.workflow.NodeExecutor.run(NodeExecutor.java:22)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.handleCreate(AbstractDependentResource.java:114)
What did you do?
Creating PDB with DependentResource
However, even though resources are already created
k get PodDisruptionBudget ✔ 10357 17:43:26
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
os-osfarm-cluster-manager-pdb N/A 1 1 88m
When running additional 'reconcile' loop
The request was created as Create rather than Update
If there is a secondary resource, shouldn't the request go to update right?
Why is it created as create?
409 errors continue to occur, which is a problem
What did you expect to see?
What did you see instead? Under which circumstances?
no
Environment
Kubernetes cluster type:
Kubernetes
$ Mention java-operator-sdk version from pom.xml file
v4.4.4
$ java -version
jdk21
$ kubectl version
v1.23.15
Possible Solution
no
Additional context
no
Activity
csviri commentedon Mar 15, 2024
Hi @10000-ki there were improvements regarding to this in newer versions, pls upgrade to the newest version
10000-ki commentedon Mar 15, 2024
@csviri what version??
I raised it to 4.6.0, but there was the same issue
csviri commentedon Mar 15, 2024
pls try the newest (4.8.1), if there will be still issues, we will take a look. Note that k8s 1.23 is also a bit old too, there were some issues with event sending in that version (what should be fixed now also on fabric8 client (restarting watches) ). But if the problem persist we can take a look, and try to make a reproducer. Probably the issue will be rather in the fabric8 client, since we are reading the last resource from the informer cache.
10000-ki commentedon Mar 15, 2024
I'm using in-house k8s cluster in my company but
v1.24is maximum version :(will that be a problem?
10000-ki commentedon Mar 15, 2024
First, I'll increase the
josdkversion to the maximumv4.8.1csviri commentedon Mar 15, 2024
AFAIK, it should not be, there were known issues, that k8s stopped sending events at some point (if remember correctly still present in 1.23) therefore the watch had to be restarted in informers periodically. But that was done in fabric8, so now it should be good.
10000-ki commentedon Mar 15, 2024
Is there anything else I need to check?
After posting the v4.8.1 version
There are a lot of events then before
and Create issues are still occurring
csviri commentedon Mar 15, 2024
ok, could you pls create a minimal open source reproducert project? Where we can run it on top of minikube and take a look on it, based on logs you provided it is impossible to figure out.
csviri commentedon Mar 15, 2024
Also what you can you can check if this is a problem on the newer versions of k8s.
10000-ki commentedon Mar 15, 2024
okay i will try
10000-ki commentedon Mar 18, 2024
@csviri
In the k8s 1.29 version, it seems to run smoothly (i tested in my local)
By the way, can you tell me exactly which version had the issue related to the
k8s eventand up to what version the impact is?csviri commentedon Mar 18, 2024
Pls take a look on these:
fabric8io/kubernetes-client#4781
kubernetes/kubernetes#102464
but there might be some newer ones. Maybe @shawkins knows more.
shawkins commentedon Mar 18, 2024
I'm not aware of newer definitive underlying kube issues. There were several later client changes to limit any potential issues with the api server not sending events - we'll restart the watches at most every 10 minutes.
10000-ki commentedon Mar 19, 2024
@csviri @shawkins
I don't know if it's related to the problem I raised.
The problems I defined are as follows.
mayBeActualfield is null, even though resources are in k8s serverIs it because the operator did not receive the resource creation event from k8s?
csviri commentedon Mar 19, 2024
yes, that means it is not in the cache of the informer, so no event received regarding that resource.
15 remaining items
shawkins commentedon Mar 20, 2024
This log shows 5 updates of 5 different resources by the operator sdk. The operator then skips propogating the update event for all of them, so this appears to be expected behavior. Can you find a situation in your logs where "Propagating event for ..." appears - the surrounding logs should help determine if that resourceVersion of that resource was the result of the operator in which case there would be an event cycle.
10000-ki commentedon Mar 22, 2024
10000-ki commentedon Mar 22, 2024
@shawkins
i think
The spec is the same, but only the resourceVersion keeps changing
The event seems to keep coming in without skipping.
10000-ki commentedon Mar 22, 2024
fix: ignore server managed fields for SSA matching
shawkins commentedon Mar 22, 2024
resourceVersions change for any modification, not just the spec. It's the generation that typically increments with spec changes.
The primary problem is the one captured here: #2249 - if the matcher fails to recognize that the resource is the same, then it adds the new annotation value, which due to a kube bug will increment the Deployment generation, and the cycle repeats.
The SSA logic is not trying to be generation aware in this case, but sees the generation change as something you are requesting in the desired state - which is not correct.
#2309 will address this specific case by ignoring all server managed fields in the comparison - this is similar to what the kubernetes client does in the PatchUtils class when generating a patch.
@csviri not sure if that is sufficient to fully close #2249 as the doc changes would be good too.
What do you mean?
10000-ki commentedon Mar 25, 2024
@shawkins oh i see..
thank you for explaining
https://javaoperatorsdk.io/docs/dependent-resources#comparing-desired-and-actual-state-matching
i found this guide, and turned
SSAoffbut it wasn't work well
csviri commentedon Mar 25, 2024
still thinking of revisiting the other algorithm with event recording, thus providing both, but I will get back to that a bit later.
10000-ki commentedon Mar 28, 2024
@shawkins @csviri hello
After �i upgrade to v4.8.2
javaoperatorsdk.io/previouslabel value keeps changingIs the k8s version bug the reason
resourceversionkeeps increasing?What's interesting is that this issue does not occur in josdk version v4.4.4 (under v4.5.0)
csviri commentedon Mar 28, 2024
@10000-ki can you create a simple reproducer pls?
( Ideally with 1 deployment dependent )
shawkins commentedon Mar 28, 2024
@csviri @10000-ki with the full yaml it's clear that there's another matching issue in play. The SSA matching logic sees the n3r.navercorp.com annotations as part of the desired state, but does not think that they are owned by the operator, thus it's trying the operation again.
shawkins commentedon Mar 28, 2024
@10000-ki I should have realized with the previous incarnation of this issue that you were creating the desired state on top of the existing state - are you doing something like new StatefulSetBuilder(existing)... ?
That is why the desired state had the generation set. Here with the state from the other field managers populated you see a similar issue.
The SSA matching logic expects you to only provide your managed desired state when creating the entry - and nothing else.
@csviri is it worth guarding against this more?
10000-ki commentedon Mar 29, 2024
shawkins commentedon Mar 29, 2024
Please don't just remove the annotations, construct your desired state starting with a fresh object and don't copy anything from one obtained by the server - the point of SSA is that the server will figure out which manager is managing which fields.
If the problem still persists after that it would be good to determine where the matching is failing. At the debug level, you should see logs like:
That will show what the matcher thinks your desired state is vs what is on the api server. Any descrepency between the two will cause another reconciliation. We can determine from there if there is SSA / Kube issue in play (that requires a workaround in the operator sdk) or if it's a problem with the matching logic.
That is expected. This optimization is based upon the matching logic working as intended. As mentioned on #2249 it is probably fine to just default this to false and allow users to opt in if they want greater performance.