Avoid looping with two mutually exclusive desired state #1288

Given a CRD that allows to set a value on a given key in a config map and an operator using informer as event source for the secondary resource (configmap)

When those two CRs

spec:
  configmap: my-config
  key: fruit
  value: banana

spec:
  configmap: my-config
  key: fruit
  value: apple

are accidentally deployed, the operator loops and reconciliations of each CR are fighting to set their value. This cause an heavy load on the k8s api.

The current version of my operator is based on an old version of the framework and basically use a timed reconciliation with withReschedule. The targetted config map was only flickering every minutes or so.

I would like to refactor and use informer, but need a circuit breaker when thing goes wrong. For eg, limiting the number of reconciliation after a burst is detected for a given period of time.

This should also ensure a reonciliation will always occured at the end of this "peace" period of time even if no new event happened, This is because if I removed one of the conflicting CR, the remaining one may not be the one that win the last fight.

While it should be possible to implement skiping in the reconcile method and have a reconciliationMaxInterval, I think that maybe #1281 could help, or maybe have a throttled event source wrapping the informer.

Any idea ?