Repository navigation
Split test jobs to avoid duplicate work #196
Description
Currently we have some test jobs that unnecessarily carry out duplicate work:
- the master-fmi and the master-fmi-fmpy jobs each build the FMUs before running them, but that's not necessary, the FMUs could just be built once and then ran twice, once with OMSimulator and once with FMPy
- the gbode task currrently runs once with the default parameters - in the future, we could try different gbode settings (e.g. different algorithms, possibly including multi-rate) for which there is no need to re-compile the simulation code.
We should probably split some of the test jobs to avoid this issue.
Metadata
Metadata
Assignees
Labels
No labels
Activity
casella commentedon Feb 17, 2025
For the FMI jobs, it seems that the majority of time is spent building the code, so splitting the test job would lead to a saving of almost 50% of the time.
AnHeuermann commentedon Sep 29, 2025
I took some time to think about locations where we can save some artifact and re-use it multiple times. Every location you see a yellow box we could reuse a model for multiple tests.
How to store around 400,000 models ($numModels \times numVersions \times artifacts = 189,29 \times 3 \times 7 = 397509$ ) in a fast way? Do we even need to store them or de we simply rearange the test pipeline so that after building a model always all possible tests are run?
Than we also need to check when do we want to run a test? No point in re-running a test with a release version of OMC if that didn't change and the library to test didn't change. I think we already do this with the SQLite data base.
casella commentedon Sep 29, 2025
This seems much more reasonable to me, I can't see the point of (ab) using mass storage the other way.
cbu-hw commentedon Sep 30, 2025
Does Jenkins have an "artifacts" feature for caching data?
In Github Actions, I would try to use the Matrix feature and upoad/download artifact action to describe the steps and their dependencies, and let the CI runner handle the caching/expiry of artifacts. Basically like this: https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/run-job-variations#using-an-output-to-define-two-matrices
Note that the up/download actions allow you to finetune caching options, so you can also cache locally. I've used this to pass freshly-built docker images between jobs running on a single machine (one step builds an image if necessary, the second uses it to run tests)
Note: AFAIUI, artifact caching only makes sense when there are multiple downstream dependents (as those will separately need the same artifact, presumably while being free to mutate it/add more files/...), so I don't get why there is an artifact box between BaseModelica export and BaseModelica.jl, and between C++ and CVODE? 🤔
AnHeuermann commentedon Sep 30, 2025
It would enable a different parallelization model than we currently use. We could use more and smaller containerized nodes for doing the steps. Splitting up our servers into a pool of e.g. 4 CPU + 32 GB virtual (insert DevOps buzzword here) servers and smooth the utilization.
Testing one library version on one version of OpenModelica with one version of importing tool / simulation setting at a time. So first one node is busy building all the models of that library, than a bunch of other nodes will simulate the models and a third set of nodes might validate the results.
But no node is used longer than for half an hour or so. In that case we could combine our available servers for pull requests and library tests into one large pool.
Uploading and downloading artifacts would of course consume time and storage.
AnHeuermann commentedon Sep 30, 2025
Yes, but for something of this size a separate and fast data base might be better suited.
They could be of course skipped. Depends on how much we test. With the C++ runtime we could test more runtimes. For the Base Modelica export there is not much of a point other than that fitted better in the parallelization model I had in mind, not the one we currently use.
cbu-hw commentedon Sep 30, 2025
what sizes are you talking about? I'd probably benchmark this a little before assuming too much. For an OOM estimate, with my machine-local cache, restoring a ~1GB compressed cache adds 8s to the CI time. I could not quickly find a recent job that updated that cache (versions seldomly change), so I can't offer a number for cache writing.