Skip to content

fix: remove XML-invalid characters from plain text - #320

Open
the8thbit wants to merge 1 commit into
ReadAlongs:mainfrom
the8thbit:fix/filter-invalid-xml-characters-279
Open

the8thbit wants to merge 1 commit into
ReadAlongs:mainfrom
the8thbit:fix/filter-invalid-xml-characters-279

Conversation

@the8thbit

Copy link
Copy Markdown
Contributor

PR Goal?

Remove XML 1.0-invalid characters from plain-text lines before building ReadAlong XML. Escape characters such as U+001B no longer produce XML that the back-end rejects. Valid Unicode, XML escaping, tabs, line endings, and the existing page/paragraph conventions are preserved.

Fixes?

Fixes #279.

Tests added?

Parameterized cases cover forbidden control characters, surrogate code points, U+FFFE, and U+FFFF. A control case preserves Indigenous-language characters, combining marks, supplementary Unicode, tabs, and XML escaping.

How to test?

pytest tests/test_make_xml_cli.py -q

Independent review reran all 51 tests in this file successfully. Applicable formatting, lint, and type-check hooks pass.

Version change?

Patch; no version files changed.

@semanticdiff-com

semanticdiff-com Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Review changes with  SemanticDiff

Changed Files
File Status
  readalongs/align_utils.py  4% smaller
  tests/test_make_xml_cli.py  0% smaller

@joanise

joanise commented Oct 24, 2026

Copy link
Copy Markdown
Member

CI will fail for this one too. If you have a chance to rebase your open PRs onto origin/main that should let CI pass, with the image change I pushed.

@joanise joanise left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thank you! Thanks for finding what was causing the errors in the first place. The code looks good and CI passes (in a separate rebased branch). Next week I'll test the integration with Studio-Web to complete my review.

Happy to see these pull requests from you -- I'm curious, are you using the studio for some Indigenous language projects or for some other projects? We're always interested in finding out what our tools are used for if you're willing to share.

@the8thbit
the8thbit force-pushed the fix/filter-invalid-xml-characters-279 branch from 55627ab to 3a22d58 Compare October 24, 2026 03:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The presence of an escape character in a text file causes a 422 from the back-end

2 participants