Repository navigation
Depends On and Conditions to describe workflows Dependent Resources #850
Description
Problem statement
- If for a resource we need an input what is an output (or status in K8S of the other resource). (Ideally these would be detected automatically? probably that is not feasable)
- When we want to make sure that something happens after an other resource is ready. Like if database is deployed, and want to create a specific default schema.
Solution
Both of these issues, can be addressed by with notions.
dependsOn- If a dependent resourceBdepends on an other dependentA, this will ensure thatBwill be reconciled afterAis ready (see optionalReadyCondition).ReconcilePreconditionis optional for a dependent, if available it is evaluated if a resource should be reconciled or not.ReadyPostconditionis optional, is provided the dependent is considered reconciled if the certain state reached. The condition check is async.DeletePostcondition- on cleanup (backwards workflow) checks if the resource is deleted successfully (or delete completed) , or the cleanup execution should be re-scheduled (the finalizer not removed).
@ControllerConfiguration( ... ,
dependents = {
@Dependent(name="dep1", type = DeploymentDependentResource.class, readyPostcondition = ReadyCondition.class )
@Dependent(name="ser1", type = ServiceDependentResource.class, reconcilePrecondition = MyCondition.class),
@Dependent(type = OtherResource.class, reconcilePrecondition = OtherCondition.class,
deletePostcondition = DeleteCondition.class , dependsOn = {"dep1","ser1"}
})
})
public class MyResourceReconciler implements Reconciler<MyResource>, EventSourceInitializer<MyResource> {
....
}Behavior Details
- If a dependent resource
Bdepends onA, butAhas aReconcilePreconditionis evaluated as false, thenBwill not be reconciled either. - If a
ReconcilePreconditionis evaluated false onAdependent resource it is aDeleter, delete should be called on it. And all the resources which are dependent onAand are deleters should be deleted. This should work in a transitive manner too ( in reveres order, so the resource with reconcile condition closer to the top is deleted later). (See: https://github.com/java-operator-sdk/java-operator-sdk/blob/4d63e1260efeb70ac7550d52e48117bf8982fabb/sample-operators/webpage/src/main/java/io/javaoperatorsdk/operator/sample/WebPageStandaloneDependentsReconciler.java#L68-L72 ) - if
ReadyPostconditioncan define anUpdateControlto return if the condition is now reached, this means that effectively wait is always async. It can also defined time delay to reschedule the reconciliation. - For cleanup (TBD) - we could dry run the actual workflow (evaulating the conditions), and execute deleters backwards, also taking in mind delete post conditions (this might be an issue if dependent changes just parallel with the CR deletion). - Alternatively, what might a more bulletproof way is just based on the backwards order red the state of all the resources defined, and delete them if present.
- Eventually a possible concurrency option could be implemented (maybe think of it from early design). Not dependent resources could be reconciled concurrently.
Sync and Async Waiting Condition
Note that waiting for a state condition can happen synchronously or asynchronously, thus if we know that we will wait for deployment quite a long time to be ready, the scheduling algirthm can just exit the reonciliation and schedule a new reconciliation for a specified delay (UpdateControl.noUpdate().rescheduleAfter(2, TimeUnit.MINUTES) if the condition not holds at the moment.
UPDATE: we discussed, for now and sync wait will be out of scope
Notes
- Not that without cycles the result is a DAG, what can be nicely scheduled for execution, more precisely a set of DAGs.
- The cycle detection will be needed, to help developers detect cycles as early as possible.
- This is an implementation detail, but 1 thing to think about is actually how to manage state, so if it's not in the status but a ConfigMap/Secret/CR the execution of two dependent resource reconciliation in the DAG cannot be parallel. Or at least not of those which update the state, so there are no conflicts. Maybe those should be marked explicitly as an improvement later?
- DeletePostcondition makes sense only for
Deleterdependent resources.
To Discuss
DeletePostconditioncould be replaced by returning a boolean fromdeleteofDeleter.
Reacted by Andrea Peruffo
Activity
metacosm commentedon Jan 20, 2022
How could they be detected automatically? This processing needs to happen at build time so unless you perform static analysis I don't see how it would be possible…
csviri commentedon Jan 20, 2022
yep, don't think it's practically possible, without explicitly stating them anyways. So the best what can be easily done is the depends on. Will update the description.
andreaTP commentedon Jan 21, 2022
This is a good starting point to model state transitions!
Something that I'm not sure about is if it should be modelled as an
Annotationat the Reconciler class level, are you going to not run the reconciling loop if the condition is not met?This approach can probably lead to a number of edge cases ...
Another comment is around
Conditions, it would be ideal if they can be more abstract than just "K8s conditions" but even data/content changes (for example monitoring the change of therevisionor of the actual content).I think that a valid almost real-world use case to test this feature against could be:
SecretDeploymentdoesn't start until theSecretis created (shows a message in the status?)SecretDeploymentSecretSecretchanged and performs a rolling restart onceNOTE1: I understand that the first
Deploymentcan be created independently from theSecret, but this serves as a test bench for this featureNOTE2: Part of this feature is offered by some popular operators (e.g. Reloader) but is an arguably common task/pattern for operators and it would be nice if the SDK can take care of those details.
I hope this comment doesn't go too much beyond the original scope of this issue.
csviri commentedon Jan 21, 2022
I guess we won't, that mean the resource A is somehow make sense to reconcile if the resource B is in a certain condition.
Agree, probably needs to be the target resource but maybe the whole context accessible from a condition API.
This use case would be perfectly covered with this design. We can create an e2e test for this.
The use case you mentioned on community meeting, that there might be a job that is created and executed only once, and deleted is I think completely covered also with this: #851
But we can discuss that further.
Than you very much for feedback @andreaTP!
andreaTP commentedon Jan 21, 2022
I might be missing something, but I think that resource A reconciler should be triggered anyhow, with appropriate parameter/s(
ready = falseor something), even if resource B haven't reached the specific condition.This will enable:
Statusfield)andreaTP commentedon Jan 21, 2022
This would be great ❤️
csviri commentedon Apr 3, 2022
Updated the design also added description. There can be now a
Conditionon any resources. ThedependsOnwill mean that the reconciliation is happening after the reconciliation of other resources - also with possibleWaitCondition.(
Note that this is now consistent with the model in Terraform (or CloudFormation and others) adjusted for Kubernetes needs, see https://www.terraform.io/language/meta-arguments/depends_on
https://stackoverflow.com/questions/60231309/terraform-conditional-creation-of-a-resource-based-on-a-variable-in-tfvars
)
[-]`dependsOn` Construct for Dependent Resources[/-][+]`dependsOn` and `Conditions` to describe workflows Dependent Resources[/+][-]`dependsOn` and `Conditions` to describe workflows Dependent Resources[/-][+]Depends On and Conditions to describe workflows Dependent Resources[/+]csviri commentedon Apr 13, 2022
Regarding exception handling but also for waitCondition and cleanupCondition, there is an interesting design option: being
fail fastor not.In other works if a reconciliation of a resource fails, but other independent resources might be still reconciled, should we proceed with them?
Same for wait and cleanup conditions, if a wait condition does not met, now reconciler can exit immediately (of course wait for already running reconciliations to finish - since execution is parallel). Or still reconcile the the independent resources. And just return with the update control that is specified with the first wait condition not met.
In the long terms, since the workflow can be executed concurrently (for independent resources), it might make sense to not fail fails. So the time to the target state is shorter. Assuming that some resources (like Deployments) might take time to get into ready state.
csviri commentedon Apr 13, 2022
An additional detail described here:
#1150
So at the end it seems all dependent resources will provide an event source, more precisely a
ResourceEventSource.That also beneficial here, because for the
Deletersit can be decided locally if a delete (so the actual API call) should be made or not based on local cache of the event source. What in a repeated delete makes it much more efficient.csviri commentedon Apr 13, 2022
UPDATE:
WaitConditionis removed, instead addedReadyConditionon resource.