Repository navigation
Always Be Saving #42
Description
Working on datasets in Countess I find myself adding "Save CSV" plugins scattered all over the place, to store intermediate results, which obscures the flow of data quite a lot.
For example, compare these:
| Before | After |
|---|---|
 |
 |
Maybe it'd be more sensible to have an automatic "Save As" at the bottom of every node to reduce clutter?
Plus (maybe) saving intermediate results would be an efficient way to handle regeneration of data?
Activity
[-]Always Saving[/-][+]Always Be Saving[/+]afrubin commentedon Mar 12, 2026
Yeah I think this is a good idea to add an "Output to CSV" checkbox for every node.
Rather than "Save As" that implies the user should provide their own file path/file name, we should automatically generate this file name and put it in a
csv/directory under the associated working directory for the pipeline (that is set globally for the whole run). The automatic file names would start with the datetime and then have a mutated version of the plugin name (kebab case? underscores?). Note that this may constrain what characters are allowed to exist in the plugin names, or certain invalid characters could just be stripped out.nickzoic commentedon Mar 13, 2026
Hmmm, interesting, any step which saves as CSV can of course be retrieved as CSV which could be used to restart partial computation but I wonder if we'd be better off saving these intermediate files as something more flexible & efficient like parquet, or just persisting data to a DuckDB database.
afrubin commentedon Mar 13, 2026
I see these as serving two very different purposes. The CSV is for people who want to inspect/plot an intermediate artefact, and parquet is for resuming workflows (which I agree is a good thing to have). So maybe the answer is both?