Skip to content

Syncing Event Source on Startup Before Event Processing Starts #788

Closed

Description

@csviri

Consider the following scenario (from the mysql sample):
When there is already a custom resource that represents a schema and the previous reconciliation already created the schema, just the operator crashed after, so neither the status or observed generation were updated. Now: the operator restarts, we check that if the schema is there within the EventSource cache, but the cache was not filled yet since the resource event was processed before. Therefore it tries to create the schema already exists.

To cover such race out of sync situations we should wait until with event processing until the event sources which are managing a cache are filled the cache on startup with the related resources.

In controller runtime there is an analogy called SyncingSource: https://github.com/kubernetes-sigs/controller-runtime/blob/e52a8b1fba620f42bf6be7c6904c77a6da342898/pkg/source/source.go#L64

See usage here: https://github.com/kubernetes-sigs/controller-runtime/blob/e52a8b1fba620f42bf6be7c6904c77a6da342898/pkg/internal/controller/controller.go#L188-L211

So the workflow should be similar in our case, initiate a start on all event sources, wait after all of the is "synced" and start processing events after this happened.

Nota that the syncing is not that simple in some cases, the PerResourcePollingEventSource might want to for all the resources in ControllerResourceEventSource and populate the cache, before it starts.

An other approach (a quick work around?) is to try to get the resources from the suppliers in event sources if not present in the cache, see:
https://github.com/java-operator-sdk/java-operator-sdk/blob/12f6125024d15c9a3963ba4fab2b75687d30d992/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PerResourcePollingEventSource.java#L126

Activity

  1. added
    kind/featureCategorizes issue or PR as related to a new feature.
    on Jan 4, 2022
  2. self-assigned this
    on Jan 4, 2022
  3. csviri commented on Jan 6, 2022

    @csviri
    CollaboratorAuthor

    See #794 , it basically solves the problem. But would be better to have an explicit SyncingEventSource, that would be quite big change for now.
    Also we could improve the current one to start the ControllerResourceEventSource to start first, then parallel way start other event sources. Would be more efficient startup, if there are lot's of event sources..

  4. csviri commented on Feb 4, 2022

    @csviri
    CollaboratorAuthor

    Changed the PollingEventSource to also sync on start.
    The only exception is PerResourcePollingEventSource (and inbound event source, but those cannot be solved this way), it does not sync, instead reads the resource if it's not polled yet. So no syncing is required there.

    Will close this issue for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/featureCategorizes issue or PR as related to a new feature.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions