feat: add orphan and group rejections to spark backend

[georgeRobertson]

TLDR of changes

add orphan and group rejections to spark backend.

Also, fixed some issues around spark xml stream reader and recursive delete issue in pipeline with hive partitioned parquets

What kind of changes does this PR introduce?

Tick all that apply

• fix: A bug fix. Correlates with PATCH in SemVer
• feat: A new feature. Correlates with MINOR in SemVer
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
• refactor: A code change that neither fixes a bug nor adds a feature
• perf: A code change that improves performance
• test: Adding missing or correcting existing tests
• build: Changes that affect the build system or external dependencies (example scopes: pip, docker, npm)
• ci: Changes to CI configuration files and scripts (example scopes: GitLabCI)

Please check if the PR fulfills these requirements

• I have read and followed the Contributing guidance
• Docs have been added / updated
• Tests and Linting in the CI are passing
• Changes have been reviewed and approved by a Project Maintainer

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud