proWES is a robust and scalable Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API gateway that may play a pivotal role in augmenting the capabilities of your GA4GH Cloud ecosystem by offering flexible middleware injection for effective workflow deployment across a range of federated workflow management system deployments.
proWES gateway may serve as a crucial component in federated compute networks based on the GA4GH Cloud ecosystem. Its primary purpose is to provide centralized features to a federated network of independently operated GA4GH WES instances. As such, it can serve, for example, as a compatibility layer, a load balancer workload distribution layer, a public entry point to an enclave of independent compute nodes, or a means of collecting telemetry.
When WES requests are received, proWES applies one or more configurable middlewares before forwarding the requests to appropriate WES instances in the network. A plugin system makes it easy to write and inject middlewares tailored to specific requirements, such as for access control, request/response processing or validation, or the selection of suitable endpoints considering data use restrictions and client preferences.
TBA
proWES is a Flask microservice that supports OAuth2-based authorization out of the box (bearer authentication) and stores information about incoming and outgoing tasks in a NoSQL database (MongoDB). Based on our FOCA microservice archetype, it is highly configurable in a declarative (YAML-based!) manner. Forwarded tasks are tracked asynchronously via a RabbitMQ broker and Celery workers that can be easily scaled up. Both a Helm chart and a Docker Compose configuration are provided for easy deployment in native cloud-based production and development environments, respectively.
For production-grade Kubernetes-based deployment, see separate instructions. For testing/development purposes, you can use the instructions described below.
Ensure you have the following software installed:
- Docker with Docker Compose 2.23.1 or later
- Git
Clone the repository and change into it:
git clone https://github.com/elixir-cloud-aai/prowes.git
cd prowesDocker Compose keeps the data of MongoDB, RabbitMQ and the runs in named
volumes (mongodb-data, rabbitmq-data, prowes-data); no data directories
need to be created.
The following user-configurable files are available:
The app configuration is a FOCA configuration (FOCA 0.22; see FOCA's configuration reference); the comments in the file describe each section. The settings that deployments usually change:
| Setting | Default | Description |
|---|---|---|
db.host, db.port |
mongodb, 27017 |
MongoDB; overridden by the environment variables MONGO_HOST and MONGO_PORT, with credentials from MONGO_USERNAME and MONGO_PASSWORD, the database name from MONGO_DBNAME (default: runStore), or the whole connection string from MONGO_URI |
jobs.host, jobs.port |
${RABBIT_HOST:-rabbitmq}, ${RABBIT_PORT:-5672} |
RabbitMQ broker of the Celery worker; the environment variable CELERY_BROKER_URL takes precedence (e.g., for brokers with credentials) |
service_info.service |
example values | Service info (GET /service-info), served by FOCA (see FOCA's service info reference): the fields of the GA4GH service info (id, name, type, organization, contactUrl, ...) and the WES fields (workflow_type_versions, supported_wes_versions, supported_filesystem_protocols, workflow_engine_versions, default_workflow_engine_parameters, auth_instructions_url, tags), checked on startup; version defaults to the version of the installed package (service_info.package); system_state_counts is computed for each request and can't be set |
server.debug |
False |
Flask's debug mode, which allows executing arbitrary code: only for local development |
exceptions |
pro_wes.exceptions.exceptions, application/json |
Problem details of errors (title, status, detail) and their media type |
custom.post_runs |
Storage path of runs (storage_path), characters and length of run identifiers (id_charset, id_length; the characters are taken as is), timeout of forwarding runs and reading run logs (timeout_post, 30 seconds), polling of the remote WES (polling_wait, polling_attempts) |
|
custom.defaults.timeout |
2 |
Timeout of other requests to the remote WES (status, cancel), in seconds |
Strings in the configuration may reference environment variables, as
${VAR} or, with a default, ${VAR:-default}; an unset variable without
default stops the app. Write $$ for a literal $. Further configuration
files can be merged over pro_wes/config.yaml with the environment variable
FOCA_CONFIG_OVERLAY (paths separated by :).
The app is served by Gunicorn with FOCA's settings for containers
(pro_wes/gunicorn.py: port 8080, or the environment
variable PORT; workers and threads from GUNICORN_PROCESSES and
GUNICORN_THREADS, default 1 and 4; other settings via GUNICORN_CMD_ARGS),
and the Celery worker is started with celery -A pro_wes.celery_worker worker.
Build the image and start proWES, the Celery worker, MongoDB and RabbitMQ, waiting until their health checks pass:
docker compose up -d --build --waitThen open the Swagger UI at http://localhost:8090/ga4gh/wes/v1/ui/. Only
proWES is published, on the host's loopback interface (127.0.0.1:8090); to
publish it elsewhere, override ports in a docker-compose.override.yaml.
The image reports version 0.0.0.dev0 unless you set
SETUPTOOLS_SCM_PRETEND_VERSION on the host (e.g.,
SETUPTOOLS_SCM_PRETEND_VERSION=0.20.0 docker compose up -d --build --wait).
RabbitMQ's credentials default to prowes:prowes; set RABBITMQ_DEFAULT_USER
and RABBITMQ_DEFAULT_PASS on the host to change them (on the first start, or
after docker compose down --volumes, which also removes the database and
the runs).
To monitor the Celery worker with Flower, at http://localhost:5565, add the
profile monitoring and set its basic authentication:
FLOWER_BASIC_AUTH=<user>:<password> docker compose --profile monitoring up -d --waitDocker Compose is for local development and testing; for deployments, see the Helm chart.
The Dockerfile builds the image on FOCA's image
(elixircloud/foca, with the FOCA release that uv.lock locks), pinned by
digest:
- Locked installs: the runtime dependencies and the build requirements
(dependency group
build-systemofpyproject.toml) are installed at the versions ofuv.lock, checked against their hashes, as wheels only; the app is built as a wheel in a separate stage, offline, and installed without its build tools, pip or uv. - User: the image runs as the non-root user
1000(group0); the app files are read-only for it. The only writable directory is/data, the storage of the runs (custom.post_runs.storage_path), owned by the user and writable for group0, for platforms that run images with an arbitrary user ID in group0(e.g., OpenShift). Volumes mounted at/datamust be writable for the user (the Helm chart setsfsGroup: 1000on Kubernetes); the app refuses to start if it can't write to the storage of the runs. - Command:
gunicorn --config python:pro_wes.gunicorn pro_wes.wsgi:app; the Celery worker runscelery -A pro_wes.celery_worker workerwith the same image. - Health check: probes the Swagger UI
(
http://localhost:8080/ga4gh/wes/v1/ui/); setHEALTHCHECK_URLif the port, the base path or the Swagger UI setting change. Containers of the Celery worker need their own health check (seedocker-compose.yaml). - Version: the build argument
SETUPTOOLS_SCM_PRETEND_VERSIONsets the version of the package, which the service info reports unlessservice_info.service.versionis set; without it,0.0.0.dev0. - Labels: OCI annotations (
org.opencontainers.image.*).
Secrets (e.g., passwords, tokens, private keys) are never part of the image:
pass them only at runtime, as environment variables or mounted secrets (e.g.,
Kubernetes Secrets), never as build arguments, which are recorded in the image
history. The image build receives only the files it needs (an allowlist in
.dockerignore). As a safety net, common secret files are
left out even within pro_wes/: .env files (except .env.example), key,
certificate and keystore files (e.g., *.pem, *.key, *.p12, *.jks), SSH
keys, .netrc, .pgpass, .aws/ and kubeconfig files. Don't keep other
secrets in the working tree either.
proWES 0.20.0 upgrades FOCA from 0.12 to 0.22. Operators need to check the following:
- MongoDB 4.4 or later: current PyMongo releases refuse to connect to
older servers; Docker Compose now runs MongoDB 7.0 (before: 3.2). The data
files of MongoDB 3.2 can't be opened by MongoDB 7.0: either start with an
empty database (
$PROWES_DATA_DIR/db; the runs stored so far are lost), or upgrade the database one major version at a time (3.2 → 3.4 → 3.6 → 4.0 → 4.2 → 4.4 → 5.0 → 6.0 → 7.0, withsetFeatureCompatibilityVersionafter each step), or export the runs withmongodumpbefore and import them withmongorestoreafter the upgrade. - Configuration: configuration files of earlier versions fail
validation, naming the keys that FOCA removed. Changed sections:
server(noenvironmentanduse_reloader;debugisFalse),security.auth(novalidation_methodsandvalidation_checks),exceptions(mappinginstead ofexceptions,required_membersandstatus_member), andcustom.post_runs.id_charset, whose characters are now taken as is (before: a Python expression such asstring.ascii_uppercase + string.digits). Start frompro_wes/config.yaml. - Run identifiers:
POST /runsreturns the run's own identifier, by whichGET /runs/{run_id}finds it (before: the identifier at the remote WES, which proWES didn't find). - Errors: error responses are problem details (
title,statusanddetail, asapplication/json) instead ofmessageandcode. Clients that readmessageorcodeneed to readdetailandstatusinstead. - Commands: the app is served with
gunicorn --config python:pro_wes.gunicorn pro_wes.wsgi:appand the worker started withcelery -A pro_wes.celery_worker worker, both from/app(before:gunicorn -c gunicorn.py wsgi:appandcelery -A celery_worker workerfrom/app/pro_wes), as indocker-compose.yamland the Helm chart (2.0.2 or later; deployments that set their owncommandorargsneed the same change). The app is created once per Gunicorn worker (no longer also by the Gunicorn master), and the Celery worker replaces its MongoDB client after forking (no moreMongoClient opened before forkwarnings). Gunicorn now runs one worker with 4 threads by default (GUNICORN_THREADS; before: 1). - Image: the image is built on FOCA's image and runs as the non-root
user
1000(group0; before: root); the app is installed from a wheel intosite-packages(before: editable, from/app/pro_wes, whoseapidirectory was world-writable). Only/datais writable: volumes mounted there must be writable for the user (the Helm chart, 2.0.4 or later, setsfsGroup: 1000on Kubernetes; OpenShift assigns user and group itself). Runs stored before by root may not be writable for the user. Upgrade the chart to 2.0.4 before pods pull the new image (e.g., charts withelixircloud/prowes:latestandimagePullPolicy: Always, on their next restart), or make the volume writable for user1000first; 2.0.4's init containers hand/dataitself (not its contents) to the user where they can, also on volumes whose ownership Kubernetes doesn't manage (e.g., NFS; withroot_squash, they only log a warning). The app now refuses to start if it can't write tocustom.post_runs.storage_path(before: onlyPOST /runsfailed). Run directories that root created before stay root's; to make them writable too, runchown -R 1000:1000 /dataas root on the volume (e.g., from a debug pod). The image now has a health check (the Swagger UI) and a default command (Gunicorn, see above). - Docker Compose:
docker-compose.yamlbuilds the imageprowes:local(before:prowes:latest) and keeps its data in named volumes instead ofPROWES_DATA_DIR(default../data/pro_wes), whose data isn't migrated. It runs MongoDB 7.0, RabbitMQ 4.3 with the credentialsprowes:prowes(configurable, see Deploy; before:guest) and Flower 2.0 only with the profilemonitoringand basic authentication. proWES (port 8090) and Flower (port 5565) are published on the host's loopback interface only; RabbitMQ (before: 5682) and MongoDB (before: 27027) are no longer published. - Broker:
RABBIT_HOSTandRABBIT_PORTare read by the app configuration (jobs);CELERY_BROKER_URLtakes precedence. - Logs: the Celery worker logs in the format of the app (section
log) instead of Celery's own. - Service info (breaking): the service info is served by FOCA from the
configuration, no longer from the database:
- Move
custom.service_infotoservice_info.service(the app refuses to start withcustom.service_info). The WES fields (workflow_type_versions, ...) stay in it and are checked on startup;contactUrlmust be a URI (e.g.,mailto:support@example.org, notsupport@example.org). POST /service-infois gone (405 Method Not Allowed): change the service info in the configuration and restart the app. Service info posted to or changed in the database before is ignored; the collectionservice_infois no longer used and can be dropped (db.service_info.drop()in the database of the runs).versionis the version of the installed package unlessservice_info.service.versionis set (before:0.18.0from the example configuration). Images built without--build-arg SETUPTOOLS_SCM_PRETEND_VERSION=<version>report0.0.0.dev0.- Fields that aren't set are left out instead of being served as
null, and values are served as configured (e.g., URLs are no longer normalized). Runs with states that proWES doesn't know are left out ofsystem_state_counts. Startup no longer fetches the GA4GH service info schema from GitHub. - The Helm chart (2.0.3 or later) no longer creates the collection
service_info. The chart can't pass a configuration overlay yet (FOCA_CONFIG_OVERLAY), so deployments with the chart serve the service info ofpro_wes/config.yamlin the image until it can (#38). - Runs that proWES accepted keep being tracked if their workflow type or
version is later removed from
workflow_type_versions(new runs are still checked). Workers of earlier versions still check stored runs against the service info in the database: stop them before the app of this version accepts runs (no rolling update across this upgrade).
- Move
- Timeouts: Gunicorn's threaded workers don't stop requests that hang,
so requests to the remote WES are limited: forwarding runs by
custom.post_runs.timeout_post(now 30 seconds; before: none), also for reading run logs, canceling runs bycustom.defaults.timeout. - Run logs: the Celery worker stores the outputs, the engine log and the task logs of the remote run log, each if it is valid (invalid ones and run logs that can't be read are logged as warnings); the run keeps its own identifier and state (before: the task failed before storing any). Other fields, e.g., of WES 1.1, are left out.
proWES is packaged with pyproject.toml; all dependencies
are locked in uv.lock. You need uv and
Git; uv also provides the Python version that proWES targets (see
requires-python in pyproject.toml, currently Python 3.13, the version of
the base image in the Dockerfile).
Clone the repository and install the app with its development tools (the
dev extra) from the lock into a virtual environment (.venv):
git clone https://github.com/elixir-cloud-aai/prowes.git
cd prowes
uv sync --locked --extra devInstall the pre-commit hooks, so that the checks run on every commit, and run them on all files once:
uv run pre-commit install
uv run pre-commit run --all-filesThe hooks (.pre-commit-config.yaml) format and
lint the code with ruff (Google style docstrings), check types with
mypy, check the project's semgrep (security) and ast-grep
(anti-patterns) rules, lint the GitHub Actions workflows with actionlint, scan
for hardcoded secrets with gitleaks and check that uv.lock matches
pyproject.toml. CI runs the same hooks. The rules, and how to suppress a
finding, are listed in the contributing guidelines.
Run the tests:
uv run pytestAfter changing dependencies in pyproject.toml, update the lock with
uv lock and commit both files. The version of the package comes from Git
tags (vX.Y.Z) via setuptools-scm; there is no version
to edit in the code.
If you use proWES, please cite it with the metadata in
CITATION.cff: on GitHub, "Cite this repository" in the
sidebar of the repository formats it as APA or BibTeX. A DOI follows with the
first release archived on Zenodo.
This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, ideas, fixes, or other code changes. Please read these guidelines if you want to contribute. And please mind the code of conduct for all interactions with the community.
The project adopts the semantic versioning scheme for versioning. Currently the service is in beta stage, so the API may change and even break without further notice. However, once we deem the service stable and "feature complete", the major, minor and patch version will shadow the supported WES version, with the build version representing proWES-internal updates.
This project is covered by the Apache License 2.0 also shipped with this repository.
proWES is part of ELIXIR Cloud & AAI, a multinational effort at establishing and implementing FAIR data sharing and promoting reproducible data analyses and responsible data handling in the life sciences.
If you have suggestions for or find issue with this app, please use the issue tracker. If you would like to reach out to us for anything else, you can join our Slack board, start a thread in our Q&A forum, or send us an email.


