[TrystanLea]

Reworks how post processes read and write feeds. Most processes now share block-based feed access and a few base classes, so each one is only its calculation. Adds a test suite that needs no database or redis, and fixes a number of bugs the tests found, several of which silently corrupted output data.

Speed and memory

Measured on synthetic data, old vs new; outputs are identical apart from the bug fixes listed below.

Process	Data	Before	After
mergefeeds	1M points	1.03s	0.06s
importcalc	1M points	0.93s	0.07s
liquidorairflow_tokwh	1M points	1.33s	0.12s
basic_formula (f1-0.5*f2+max(f3-21,0))	1M points	3.14s	0.30s
scalefeed	3M points	0.40s	0.11s
carnot_cop_simulator	1 year of 10s data	7.8s, 51 MB	2.6s, 3 MB
batterysimulator	1 year of 10s data	6.2s, 228 MB	4.5s, 8 MB
solarbatterykwh	1 year of 10s data	1.8s, 3 MB	1.6s, 3 MB

Memory no longer grows with the amount of data processed. Previously batterysimulator held all 18 outputs in memory until the end, which would need over 1 GB to recalculate several years of data.

Structure

• FeedReader / FeedWriter (common.php): block reads and writes, resampling onto a common timeline, checked writes (a full disk is an error rather than silent truncation), meta written with the data, redis last time/value set to the last datapoint.
• resolve_start_time(): one implementation of recent / from / all.
• Base classes:
  • MapProcess (input → output) and InPlaceProcess (modify a feed in place): implement value($x).
  • CombineProcess (several inputs → output, optionally accumulating): implement row($a, $b, ...).
• ModelHelper keeps the same API, rebuilt on the reader/writer.
• New process: backfillfeed. Extends a feed that has stopped recording from another feed recording the same quantity. It only ever appends after the target's current end.

Bug fixes

• powertokwh / accumulator: a value outside the min/max limit was dropped instead of written, shifting all later output one interval earlier.
• basic_formula:
  • a formula using f1 and f12 never read feed 12;
  • "from" mode appended a second copy of the output;
  • division by zero gave 0 (now a missing value);
  • f1*0 was evaluated as f1*1.
• "from" mode past the end of the output left a 0.0 gap. "Reprocess all" left stale data beyond the end of the input.
• carnot_cop_simulator: the first run shifted every output one interval later; each later run counted the step it continued from twice.
• batterysimulator: each run counted the step it continued from twice (state of charge and all kWh totals), and off-peak charging stopped at every run boundary.
• solarbatterykwh: the 12 hour backcast counted one 10s step twice on every run.
• solarbatterykwh, batterysimulator, carnot_cop_simulator: a run starting inside a gap held 0 instead of the last valid value.
• Other:
  • solardirectcalc updated redis using a setting that doesn't exist;
  • constantflow / liquidorairflow ignored the process mode;
  • combine processes overwrote the output meta in recent mode;
  • to_signed always reported 0 datapoints modified.

Behaviour changes to be aware of

• Finer inputs are averaged, not sampled. Where a combine process has inputs with different intervals, each output interval gets the average of the finer feed's values in it, not one sample. Only complete output intervals are written.
• "Reprocess all" replaces an output feed whose start time or interval doesn't match. "recent" and "from" refuse with an error.
• Redis time is the timestamp of the last datapoint, not one interval after it.
• Existing outputs written by the old code keep the old errors until reprocessed: carnot_cop_simulator outputs are shifted, and basic_formula outputs from formulas that divide by zero or multiply by 0 have the old values. The simulators have no "reprocess all" mode, so their output feeds need clearing and the process running again.

Testing

php tests/run.php runs 86 tests against synthetic feeds in a temporary directory:

• Existing behaviour: each converted process was checked against the old code before conversion, and the intended changes above are covered by tests that fail on the old code.
• Simulator snapshots: single 100k-step runs (spanning a summer time change) recorded from the old code; the new code matches them exactly.
• Incremental runs: for the simulators, processing in several runs must give the same output as one run.
• basic_formula: compared byte for byte with a per-datapoint reference evaluation on random data with gaps and zeros.

Not yet converted: removenan, average, downsample, upsample.

🤖 Generated with Claude Code