Skip to content

build(tools): run the repository tooling on a locked uv Python 3.14 environment - #323

Merged
aghassemlouei merged 10 commits into
google:mainfrom
digitalresistor:build/tools-uv-python
Oct 21, 2026
Merged

aghassemlouei merged 10 commits into
google:mainfrom
digitalresistor:build/tools-uv-python

Conversation

@digitalresistor

@digitalresistor digitalresistor commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

This change runs the tools/ scripts and the root tests/ suite on a single pinned Python version and a locked package set. Contributors now install only uv, and CI installs the same environment that they use locally.

Fixes #322

Changes in the branch:

  • A new root pyproject.toml file defines the stellar-engine-tools project.
    • It sets requires-python = ">=3.14,<3.15", which matches the cli/ project, so the whole repository runs on one interpreter.
    • It sets [tool.uv] package = false and declares no uv workspace, so the cli/ project keeps its own independent lock.
    • The main dependencies cover everything that tools/requirements.txt listed. They also add pyyaml, which the tools import but the old file never declared.
    • The default dev group holds pytest and ruff. Ruff keeps the exact 0.17.1 pin that tools/requirements.txt had, so a new release cannot change formatting under CI.
    • The default gem4gov group holds the packages that the root tests need from the gem4gov-cli blueprint. The blueprint keeps its own requirements.txt file for blueprint users.
  • A new root uv.lock file locks 58 packages.
  • The tools/requirements.txt file is removed.
  • The tools/lint.sh script runs every Python check and both ruff steps through uv run --locked.
  • The tools/run_tests.sh script runs its Python steps through uv run --project "${REPO_ROOT}" --locked.
    • The cli/ exclusion and its outdated comment are gone, because both projects now pin Python 3.14.
    • As a result, the compile check now covers the cli/ sources too.
  • In the .github/workflows/ci.yml workflow, the python-tests job now uses astral-sh/setup-uv with uv sync --locked. It pins the same action SHA as the cli-tests job.
  • The python-tests job runs ruff check and ruff format --check through uv run --locked.
  • In the cli-tests job, the license header step now runs check_boilerplate.py through uv run --locked.
  • The disabled documentation jobs in .gitlab-ci.yml now use the python:3.14 image and run through uv from the repository root. This fixes the broken paths that the old cd tools step caused.
  • The root ruff.toml file targets py314, the version that the tooling now runs on.
    • The new target rewrites three except clauses in tools/ to the PEP 758 form without parentheses.
    • The blueprints/ code keeps the py311 target through per-file-target-version. Blueprint code runs on the interpreter of whoever deploys it, and the gem4gov-cli blueprint documents Python 3.6 or later, so ruff must not rewrite it to syntax that only Python 3.14 accepts.
  • The .gitignore file ignores .venv/ directories.
  • The Contributing section of the root README.md explains how to run the checks with uv.
  • A new tests/tools/test_uv_tooling.py test enforces five rules.
    • The root and cli/ projects share one requires-python range, and that range spans exactly one minor version.
    • The root project has package = false and declares no uv workspace.
    • The repository commits a uv.lock file.
    • Every Python call in the tools/*.sh scripts goes through uv run --locked.
    • The ruff target matches the pinned Python, and blueprints/ stays on py311.
  • The tools/check_names.py step in tools/lint.sh now checks fast/stages-aw.
    • The step pointed at fast/stages, which does not exist, so the script crashed on max() over an empty list.
    • The script now reports a usage error when a directory holds no names.
    • The script now exits non-zero on overlong names. It returned its status from the Click command before, and Click ignores that value, so overlong names never failed the check.
    • A new tests/tools/test_check_names.py test covers both exit codes, the usage error and the tools/lint.sh target.
  • The tests/test_label_taxonomy.py file is now formatted with ruff. It landed in the old 2-space style after the switch to ruff, so ruff check and ruff format --check fail on main and would fail this branch too. PR style(tests): format test_label_taxonomy.py with ruff #324 carries the same commit for main, and the next rebase drops it here once style(tests): format test_label_taxonomy.py with ruff #324 merges.
  • The tests/__init__.py file no longer mocks missing packages. The locked environment installs every package it mocked, so the mocks never triggered, and a missing package now fails the import instead of passing against a MagicMock.

Related work:

Breaking Changes

Local contributor setup changes. The tools/requirements.txt file no longer exists, and the tools/lint.sh and tools/run_tests.sh scripts now need uv on your PATH. Terraform deployments, variables and state are unaffected.

Install uv by following https://docs.astral.sh/uv/getting-started/installation/ and run `uv sync` from the repository root. Then run the checks with `uv run pytest`, `bash tools/run_tests.sh` and `bash tools/lint.sh`. You no longer need a pip virtual environment built from `tools/requirements.txt`, so you can delete it.

Merge order

These four pull requests share one merge order. Merge them in this sequence.

  1. style(tests): format test_label_taxonomy.py with ruff #324 makes the ruff check step pass on main again. Every other pull request needs it for a green CI run.
  2. build(tools): run the repository tooling on a locked uv Python 3.14 environment #323 adds the root uv.lock file. Rebase it onto main after style(tests): format test_label_taxonomy.py with ruff #324 merges, which drops its copy of the style(tests): format test_label_taxonomy.py with ruff #324 commit.
  3. ci(deps): open Dependabot pull requests for pinned versions #326 adds Dependabot. Its root uv entry needs the uv.lock file from build(tools): run the repository tooling on a locked uv Python 3.14 environment #323.
  4. chore(repo): point .git-blame-ignore-revs at the #302 squash commit #325 fixes .git-blame-ignore-revs. It only needs style(tests): format test_label_taxonomy.py with ruff #324 for a green CI run, so it can merge at any point after style(tests): format test_label_taxonomy.py with ruff #324.

This pull request is step 2.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

Deployment & Compliance Impact

  • Applicable Regimes:
    • US Region Restricted (e.g., Access Policy constraint)
    • FedRAMP Moderate
    • FedRAMP High
    • DoD IL4
    • DoD IL5
    • General / All
  • NIST 800-53r5 Controls: None. The change affects repository tooling and CI only, and deploys no resources.

Checklist

Code Quality & Reusability

  • My code adheres to the Maximize Reusability principle. I have not redefined common elements and have reused existing base configurations and modules where possible.
  • I have checked that no existing module or configuration in modules/ or fast/stages-aw/ can be leveraged for this change.
  • My code follows the established naming conventions outlined in docs/naming-convention.md.

Documentation

  • I have updated the README.md of the modified module or blueprint.
  • I have added/updated documentation for inputs (variables) and outputs.

Security

  • My change adheres to GCP security best practices and the principle of least privilege.
  • I have ensured compliance with the targeted regime (FedRAMP Moderate, FedRAMP High, IL4, IL5, etc.).

Testing

  • I have tested my changes locally.
  • I have included details of my testing in this PR.

Testing Performed

All commands ran locally from the branch.

Command Result
uv lock --check and uv sync --locked Pass
uv run --locked pytest 82 passed, including the 10 new tooling and check_names.py tests
uv run --locked ruff check . and uv run --locked ruff format --check . Clean
bash tools/run_tests.sh Pass, with 82 unittest tests, all shell tests passing and 56 compiled Python files
uv lock --check, uv sync --locked and uv run --locked pytest in cli/ 160 passed
uv run --locked python ../tools/check_boilerplate.py src tests in cli/ Clean
check_boilerplate.py on the new test files Clean
shellcheck 0.11.0 on tools/*.sh and the changed shell test scripts Clean
yaml.safe_load on .github/workflows/ci.yml Pass
terraform fmt -recursive -check Pass
ast.parse on the gem4gov-cli and analytics blueprint sources under Python 3.11 and 3.13 Pass, and git diff shows no change under blueprints/
check_names.py --prefix-length=10 --failed-only fast/stages-aw Pass, with no overlong names

Comparison with the old setup:

  • The old Python 3.11 pip setup was rebuilt from tools/requirements.txt and the gem4gov-cli requirements on the same tree. It passed the same tests as the new setup apart from the new ones. That comparison ran before the rebase onto the ruff switch.
  • Each Python step of tools/lint.sh ran under both setups and gave matching results.
    • validate_metadata.py passes on both setups.
    • check_boilerplate.py, check_documentation.py and check_links.py already fail on main with identical output on both setups. This change does not cause those failures.
    • check_names.py crashed on both setups before the fix in this branch.
  • The new tooling test fails as expected when a bare python3 call is added to tools/lint.sh.

Not verified:

  • tools/tflint.sh and tests/scripts/test_tflint.sh did not run, because tflint is not installed locally. The CI tflint job covers them.
  • shellcheck still reports 28 existing findings in tests/scripts/test_common_functions.sh. That count matches main, and this branch leaves the file untouched.
  • CI has not run yet.

Follow-ups outside this change:

  • An exclude-newer cooldown in both uv projects would keep lock refreshes in line with the repository's two-week rule for new releases.
  • Pin the uv install in .gitlab-ci.yml when that pipeline is enabled again.

@github-actions github-actions Bot added Documentation Improvements or additions to documentation Project Management labels Oct 20, 2026
Add a root pyproject.toml and uv.lock that pin the tools/ scripts and the
root tests/ suite to Python 3.14, the same interpreter the cli/ project
uses. The dependencies cover tools/requirements.txt, and dependency groups
cover the test runners and the gem4gov-cli packages the root tests import.
The root project declares no uv workspace, so cli/ keeps its own lock.
The lint.sh and run_tests.sh scripts now call uv run --locked instead of
the system python3 and yapf, so every machine uses the pinned Python 3.14
environment from the root uv.lock.

run_tests.sh no longer skips cli/ in its compile check, because cli/ and
the root project now pin the same Python version.
The python-tests job now sets up uv with the astral-sh/setup-uv commit
that the cli-tests job already pins, installs the root uv.lock with uv
sync --locked, and runs pyflakes and pytest through uv. This replaces
actions/setup-python on Python 3.11 and the unpinned pip installs.

The cli-tests job runs the license header check through uv as well.

Remove tools/requirements.txt, because the root pyproject.toml now lists
those dependencies.
The commented-out check-docs jobs in .gitlab-ci.yml now install the root
uv.lock and run tools/check_documentation.py from the repository root on
Python 3.14, so they match the GitHub workflow if you re-enable the
pipeline.
The Contributing section of the root README now lists the uv commands
that run the unit tests and the lint checks on the pinned Python version.
The new tests/tools/test_uv_tooling.py test fails when a tools/*.sh script
calls Python outside `uv run --locked`, when the root pyproject.toml
allows more than one Python minor version or drifts from cli/, when the
root project declares a uv workspace, or when uv.lock is missing.
…long names

The lint step pointed check_names.py at fast/stages, which does not exist, so the script crashed on max() over an empty list. The script also returned its status from the Click command, which Click ignores, so overlong names never failed the check. Point the step at fast/stages-aw, report a usage error when a directory holds no names, and exit non-zero on overlong names.
The locked uv environment installs every package that tests/__init__.py mocked, so the mocks never triggered. Without them, a missing package fails the import instead of letting tests pass against a MagicMock.
@digitalresistor

Copy link
Copy Markdown
Contributor Author

Working on rebasing this. Stand by.

The file landed in 2-space yapf style after the repository switched to ruff, so ruff check and ruff format --check fail on main.
The tools/ scripts and the root tests/ suite now run on the Python 3.14 that the root pyproject.toml pins, so ruff targets the same version. Blueprint code runs on the deployer's interpreter, and the gem4gov-cli blueprint documents Python 3.6 or later, so blueprints/ keeps the py311 target and ruff does not rewrite it to syntax such as PEP 758 except clauses. The new target reformats three except clauses in tools/.

@aghassemlouei aghassemlouei left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the root pyproject.toml and uv.lock configuration, the ruff.toml per-file target version settings, the tools/check_names.py exit status and fast/stages-aw path fixes, and the tests/tools/test_check_names.py and tests/tools/test_uv_tooling.py test suites.

@aghassemlouei
aghassemlouei merged commit 17de1fa into google:main Oct 21, 2026
16 checks passed
@digitalresistor
digitalresistor deleted the build/tools-uv-python branch October 21, 2026 00:11
@aghassemlouei aghassemlouei mentioned this pull request Oct 22, 2026
@aghassemlouei aghassemlouei added Dependencies - Python Pull requests that update Python and uv dependencies Enhancement New feature or request Level of Effort - Medium Moderate task requiring thought and testing; typically takes a couple of days to a week Priority - Medium Standard features and non-blocking bugs; important for the current milestone but not urgent labels Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Dependencies - Python Pull requests that update Python and uv dependencies Documentation Improvements or additions to documentation Enhancement New feature or request Level of Effort - Medium Moderate task requiring thought and testing; typically takes a couple of days to a week Priority - Medium Standard features and non-blocking bugs; important for the current milestone but not urgent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Run the repository tooling on one pinned Python version with uv

2 participants