Skip to content

test(workflow-operator): add test helper for FileScanUtilsSpec - #9217

Open
aglinxinyuan wants to merge 1 commit into
apache:mainfrom
aglinxinyuan:test/file-scan-utils-spec
Open

aglinxinyuan wants to merge 1 commit into
apache:mainfrom
aglinxinyuan:test/file-scan-utils-spec

Conversation

@aglinxinyuan

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

In common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtilsSpec.scala:

  1. Add a scan(...) test helper providing canonical default arguments (displayFileName = "d", attributeType = FileAttributeType.STRING, fileEncoding = FileDecodingMethod.UTF_8, extract = false, outputFileName = false, fileScanOffset = None, fileScanLimit = None) around FileScanUtils.createTuplesFromFile.
  2. Deduplicate 11 createTuplesFromFile test cases by adopting the scan(...) helper instead of repeating lengthy 8-argument parameter lists.
  3. Add a memoryFailureMessage(...) test helper to encapsulate the safeToByteArray error interception across 5 memory guard test cases.

All 18 test cases, docstrings, assertions, and edge-case verifications are preserved untouched.

Any related issues, documentation, discussions?

Closes #9216

How was this PR tested?

Verified with sbt unit tests, operator metadata tests, and linters:

sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.file.FileScanUtilsSpec"
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.metadata.*"
sbt "WorkflowOperator/scalafmtCheck" "WorkflowOperator/Test/scalafmtCheck" "WorkflowOperator/scalafix --check" "WorkflowOperator/Test/scalafix --check"

All 18 tests in FileScanUtilsSpec passed, all 229 operator metadata tests passed, and all scalafmt / scalafix checks passed.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Antigravity (Gemini 2.5 Pro)

Copilot AI balanced review requested due to automatic review settings October 29, 2026 08:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • No candidates found from git blame history.

@codecov-commenter

codecov-commenter commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.04%. Comparing base (f673554) to head (e383366).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9217      +/-   ##
============================================
- Coverage     92.05%   92.04%   -0.01%     
+ Complexity     5041     5038       -3     
============================================
  Files          1286     1286              
  Lines         55911    55911              
  Branches       7137     7137              
============================================
- Hits          51469    51465       -4     
- Misses         2692     2693       +1     
- Partials       1750     1753       +3     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.16% <ø> (ø) Carriedforward from f673554
amber 86.55% <ø> (-0.03%) ⬇️
computing-unit-managing-service 64.02% <ø> (ø)
config-service 87.61% <ø> (ø)
file-service 82.29% <ø> (ø)
frontend 96.48% <ø> (ø) Carriedforward from f673554
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.54% <ø> (ø) Carriedforward from f673554
workflow-compiling-service 74.25% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 10 worse · ⚪ 5 noise (<±5%) · 0 without baseline

Compared against main f673554 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 356 0.217 27,466/37,625/37,625 us 🔴 +21.0% / 🔴 +168.1%
🔴 bs=100 sw=10 sl=64 783 0.478 126,294/152,086/152,086 us 🔴 +10.0% / 🔴 +58.0%
⚪ bs=1000 sw=10 sl=64 950 0.58 1,047,213/1,145,714/1,145,714 us ⚪ within ±5% / 🔴 +24.6%
Baseline details

Latest main f673554 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 356 tuples/sec 420 tuples/sec 905.73 tuples/sec -15.2% -60.7%
bs=10 sw=10 sl=64 MB/s 0.217 MB/s 0.257 MB/s 0.553 MB/s -15.6% -60.7%
bs=10 sw=10 sl=64 p50 27,466 us 22,691 us 11,331 us +21.0% +142.4%
bs=10 sw=10 sl=64 p95 37,625 us 33,374 us 14,032 us +12.7% +168.1%
bs=10 sw=10 sl=64 p99 37,625 us 33,374 us 17,338 us +12.7% +117.0%
bs=100 sw=10 sl=64 throughput 783 tuples/sec 853 tuples/sec 1,205 tuples/sec -8.2% -35.0%
bs=100 sw=10 sl=64 MB/s 0.478 MB/s 0.521 MB/s 0.735 MB/s -8.3% -35.0%
bs=100 sw=10 sl=64 p50 126,294 us 114,832 us 89,237 us +10.0% +41.5%
bs=100 sw=10 sl=64 p95 152,086 us 144,599 us 96,280 us +5.2% +58.0%
bs=100 sw=10 sl=64 p99 152,086 us 144,599 us 111,344 us +5.2% +36.6%
bs=1000 sw=10 sl=64 throughput 950 tuples/sec 959 tuples/sec 1,239 tuples/sec -0.9% -23.3%
bs=1000 sw=10 sl=64 MB/s 0.58 MB/s 0.585 MB/s 0.756 MB/s -0.9% -23.3%
bs=1000 sw=10 sl=64 p50 1,047,213 us 1,041,252 us 877,826 us +0.6% +19.3%
bs=1000 sw=10 sl=64 p95 1,145,714 us 1,101,149 us 919,412 us +4.0% +24.6%
bs=1000 sw=10 sl=64 p99 1,145,714 us 1,101,149 us 963,000 us +4.0% +19.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,562.28,200,128000,356,0.217,27465.68,37624.83,37624.83
1,100,10,64,20,2555.31,2000,1280000,783,0.478,126293.52,152086.45,152086.45
2,1000,10,64,20,21048.93,20000,12800000,950,0.580,1047212.81,1145713.68,1145713.68

@aglinxinyuan
aglinxinyuan requested a review from kunwp1 October 29, 2026 22:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add default-argument test helper for FileScanUtilsSpec

3 participants