Repository navigation
Closed
Labels
Description
Description
Four tasks or helpers fail on edge cases:
AdvancedStatCounterTasktestscounter % show_everybefore incrementing the counter: the first counted execution is never logged and the reported item count is one behind. Withshow_every: 1, 3 executions give: nothing, "1 items processed", "2 items processed".IterableBatchTaskonly accepts an integerbatch_count: thenull !== $batchCountbranch ("only flush at the end", as inSimpleBatchTask) is dead code, andbatch_count: ~throws.ConditionTrait::checkValue(),checkEmpty()andgetValue()are typedobject|array: as soon as a condition is set, a scalar input raises aTypeError. The documented''path, which targets the whole value, is therefore unusable on scalars, e.g.array_filteron a list of strings.ColumnAggregatorTaskusesisset()to detect missing columns, so a column whose value isnullcounts as missing and throws anUnexpectedValueException.
Reproduction
clever_age_process:
configurations:
demo.tasks_edge.advanced_stat_counter:
tasks:
entry:
service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask'
options: { output: [a, b, c] }
outputs: [ stats ]
stats:
service: '@CleverAge\ProcessBundle\Task\Reporting\AdvancedStatCounterTask'
options: { show_every: 1 }
demo.tasks_edge.iterable_batch_null:
tasks:
entry:
service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask'
options: { output: [a, b, c] }
outputs: [ batch ]
batch:
service: '@CleverAge\ProcessBundle\Task\IterableBatchTask'
options: { batch_count: ~ }
outputs: [ debug ]
debug:
service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask'
demo.tasks_edge.condition_scalar:
tasks:
entry:
service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask'
options: { output: [a, b] }
outputs: [ filter ]
filter:
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
options:
transformers:
array_filter:
condition: { match: { '': a } }
outputs: [ debug ]
debug:
service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask'
demo.tasks_edge.column_aggregator_null:
tasks:
entry:
service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask'
options: { output: { a: x, b: ~ } }
outputs: [ aggregate ]
aggregate:
service: '@CleverAge\ProcessBundle\Task\ColumnAggregatorTask'
options: { columns: [a, b] }
outputs: [ debug ]
debug:
service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask'Output on main (c1a2804, PHP 8.5, Symfony 7.4):
# advanced_stat_counter (-vv): 3 executions, only 2 logs, counts are off by one
INFO [cleverage_process_task] Last iteration 00:00:00 ago - n/a items/s - 1 items processed in 00:00:00
INFO [cleverage_process_task] Last iteration 00:00:00 ago - n/a items/s - 2 items processed in 00:00:00
# iterable_batch_null
CRITICAL [cleverage_process_task] The option "batch_count" with value null is expected to be of type "integer", but is of type "null".
# condition_scalar
CRITICAL [cleverage_process_task] Transformation 'array_filter' have failed: CleverAge\ProcessBundle\Transformer\Array\ArrayFilterTransformer::checkValue(): Argument #1 ($input) must be of type object|array, string given, called in .../src/Transformer/ConditionTrait.php on line 33
# column_aggregator_null
CRITICAL [cleverage_process_task] Missing columns [b] in input
Cause
AdvancedStatCounterTask::execute()tests$this->counter > 0 && 0 === $this->counter % show_everyand only increments the counter afterwards (L70).IterableBatchTask::configureOptions():setAllowedTypes('batch_count', 'integer')makes thenull !== $batchCounttest ofexecute()dead code.ConditionTrait::checkValue(),checkEmpty()andgetValue()are typedobject|array, althoughcheckCondition()acceptsmixedandgetValue()handles the''path (whole value).ColumnAggregatorTask::execute():!isset($input[$column])is true for anullvalue.
Proposed fix
AdvancedStatCounterTask: increment the counter before theshow_everytest, so statistics are logged on the N-th, 2N-th… counted execution, with the right item count.IterableBatchTask: allow['integer', 'null']forbatch_count;nullbuffers every input and outputs them one by one on flush, likeSimpleBatchTask.ConditionTrait: type$inputasmixed; the''path targets the whole value, even a scalar; any other path on a scalar givesnull, like a missing key.ColumnAggregatorTask: detect missing columns witharray_key_exists()for array inputs, sonullcolumns are aggregated (ArrayAccessinputs keep theisset()check).- Update the reference documentation (AdvancedStatCounterTask, IterableBatchTask, ColumnAggregatorTask, FilterTask, ConditionTrait, ArrayFilterTransformer), add unit tests.
Requirements
- Documentation updates
- Reference
- Cookbooks
- Changelog
- Unit tests
Breaking changes
None intended (bug fixes), but behaviour changes slightly:
AdvancedStatCounterTasknow logs one execution earlier (on the N-th counted execution instead of the (N+1)-th), with correct item counts.ColumnAggregatorTasknow aggregates rows whose column value isnullinstead of throwing (or only warning withignore_missing: true).ConditionTraitmethods now declare amixed$input: a class overridingcheckValue(),checkEmpty()orgetValue()with the oldobject|arraysignature must widen it.
Activity
Merge pull request #57 from cleverage/process-bundle-206
Merge pull request #217 from cleverage/206