Leader election #411

Activity

1. jmrodri

   added

   kind/featureCategorizes issue or PR as related to a new feature.Categorizes issue or PR as related to a new feature.

   feature

   on Sep 2, 2021
2. csviri

   self-assigned this

   on Jan 10, 2022
3. csviri commented on Feb 2, 2022

   

   csviri

   on Feb 2, 2022

   Collaborator

   More actions

   see https://kubernetes.slack.com/archives/CAW0GV7A5/p1643798302258639?thread_ts=1643796946.554699&cid=CAW0GV7A5
4. csviri

   added this to the 3.3 milestone on Jun 8, 2022
5. csviri commented on Jun 8, 2022

   

   csviri

   on Jun 8, 2022

   Last edited by csviri

   Collaborator

   More actions

   @shawkins I added this to 3.3 milestone for now.

   To summarize it to my understanding there are two cases when running multiple instances of operators is happening and/or desirable, and leader election makes sure only one of them is actively reconciling, thus other than leader instances don't execute reconcilers:

   1. minimize downtime in following cases:
      • There is an updated version of operator being released, and deployment first creates the new version of the operator pod then stops the old one. (For now to handle this scenario use recreate deployment strategy)
      • Minimise downtime of an operator crash, so there are multiple instances running all the time. However there are multiple strategies are this. So if an operator not the leader, should it populate the caches? and just not reconcile the events.
   2. make sure fail-over operators are be provisioned on the cluster, so multiple instance are provisioned, therefore in case the active operator's pod is crashed, it cannot happen that a new instance is not provision because of cluster resources are not available.

   In summary there is one design question:

   • Should the non-leader operator instances activate event sources, and just don't trigger reconciliation until elected as leader? Or just basically start the operator when it is elected as leader. Both has pros and cons, when event sources are activated will consume resources (polling in some cases possibly, cache resources in memory), on the other hand will minimize downtime, in case of the syncing the caches on startup takes long time.
6. metacosm commented on Jun 8, 2022

   

   metacosm

   on Jun 8, 2022

   Collaborator

   More actions

   Maybe the strategy should be configurable i.e. the framework would support event replication but let users activate or deactivate it depending on their needs?

   👍React with 👍1Reacted by Attila Mészáros
7. csviri commented on Jun 9, 2022

   

   csviri

   on Jun 9, 2022

   Collaborator

   More actions

   Maybe the strategy should be configurable i.e. the framework would support event replication but let users activate or deactivate it depending on their needs?

   Yes, a feature flag would be nice for that, agree.
8. csviri commented on Jun 9, 2022

   

   csviri

   on Jun 9, 2022

   Last edited by csviri

   Collaborator

   More actions

   Just one more note, in both cases, if an operator becomes the leader will need reconcile all the resources anyways. Since there is no info how long the other "leader before" was down.
9. csviri

   modified the milestones: 3.3, 3.2 on Jun 20, 2022
10. csviri

    linked a pull request that will close this issueLeader Election #1358on Jul 21, 2022
11. metacosm

    added a commit that references this issue on Aug 24, 2022

    feat: support leader Election (#1358)

    ...

    This issue will close once commit e5c4928 is merged into the 'main' branch.

    e5c4928
12. csviri

    added a commit that references this issue on Aug 25, 2022

    feat: support leader Election (#1358)

    ...

    This issue will close once commit 6e874fc is merged into the 'main' branch.

    6e874fc
13. csviri

    closed this as completedon Aug 26, 2022
14. csviri

    added 2 commits that reference this issue on Aug 30, 2022

    feat: support leader Election (#1358)

    ...

    This issue will close once commit 4a560fe is merged into the 'main' branch.

    4a560fe

    feat: support leader Election (#1358)

    ...

    This issue will close once commit 4676ce6 is merged into the 'main' branch.

    4676ce6