Repository navigation
Closed
Labels
Description
Description
JsonStreamFile has two problems:
- Reader, scalar line: when a JSON Lines file contains a line that decodes to a scalar (
42,"foo",true…),JsonStreamReaderTaskcrashes with a PHPTypeErrorcoming from the?arrayreturn type, without a useful message nor line number. - Writer, missing directory:
JsonStreamWriterTaskfails when the parent directory offile_pathdoes not exist, whileCsvWriterTaskcreates it (seeCsvFile).
Reproduction
clever_age_process:
configurations:
demo.file.json_stream_reader_scalar_line:
tasks:
content:
service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask'
outputs: [ write ]
options:
output: "{\"a\":1}\n42\n"
write:
service: '@CleverAge\ProcessBundle\Task\File\FileWriterTask'
outputs: [ read ]
options:
filename: '%kernel.project_dir%/var/data/json_stream_scalar_line.jsonl'
read:
service: '@CleverAge\ProcessBundle\Task\File\JsonStream\JsonStreamReaderTask'
outputs: [ debug ]
debug:
service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask'
demo.file.json_stream_writer_missing_dir:
tasks:
entry:
service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask'
outputs: [ write ]
options:
output:
- column1: value1-1
- column1: value1-2
write:
service: '@CleverAge\ProcessBundle\Task\File\JsonStream\JsonStreamWriterTask'
outputs: [ debug ]
options:
file_path: '%kernel.project_dir%/var/data/json_stream_missing_dir/sub/out.jsonl'
debug:
service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask'$ bin/console cleverage:process:execute demo.file.json_stream_reader_scalar_line
array:1 [
"a" => 1
]
CRITICAL [cleverage_process_task] CleverAge\ProcessBundle\Filesystem\JsonStreamFile::readLine(): Return value must be of type ?array, int returned
$ bin/console cleverage:process:execute demo.file.json_stream_writer_missing_dir
CRITICAL [cleverage_process_task] SplFileObject::__construct(/var/www/var/data/json_stream_missing_dir/sub/out.jsonl): Failed to open stream: No such file or directory
Tested on main (c1a2804), PHP 8.5, Symfony 7.4.
Cause
JsonStreamFile::readLine()returns the result ofjson_decode()directly, which can be a scalar and breaks the?arrayreturn type.JsonStreamFile::__construct(), used byJsonStreamWriterTask::execute(), opens the file withSplFileObjectwithout creating its directory, unlikeCsvFile::__construct().
Proposed fix
JsonStreamFile::readLine()checks the decoded value and throws an explicit\UnexpectedValueExceptionnaming the line number, the file path and the type found. The signature is unchanged andnulllines are still skipped.JsonStreamFile::__construct()creates the parent directory recursively, asCsvFiledoes, for non-read modes only (standard streams are skipped).JsonStreamWriterTask(and the--outputoption ofcleverage:process:executewith thejson-streamformat) then work with a path whose directory does not exist yet.- Update the reference documentation, add unit tests.
Requirements
- Documentation updates
- Reference
- Cookbooks
- Changelog
- Unit tests
Breaking changes
None. A scalar line now throws an \UnexpectedValueException instead of a \TypeError.
Activity
Merge pull request #58 from cleverage/process-bundle-207
Merge pull request #218 from cleverage/207