proTES

A gateway for the GA4GH Task Execution Service (TES) API that federates your TES instances, brought to you by ELIXIR On-Cloud. Clients send their tasks to proTES as to any TES instance; proTES passes each task through a configurable stack of middlewares, which pick the TES instance to run it on (or change the request), forwards it there and tracks its state. Built-in middlewares distribute tasks randomly, for load balancing, or to the TES instance closest to the task's input data, to bring compute to the data; write your own for access control, request processing or validation, or the selection of instances by data use restrictions and client preferences. Run it with Docker Compose or deploy it to Kubernetes with its Helm chart.

Known deployments

None listed yet.

Running proTES yourself? Open a pull request that adds your deployment (URL, operator and notes) to this section of README.md.

Contents

• Getting started: run proTES with Docker Compose and send the first requests.
• Configuration: the base config, overlays, the TES instances, the middlewares, the task settings, the request size limit, the server and the service info.
• API: the endpoints, how tasks are forwarded, and errors.
• Middlewares: the built-in middlewares, how to write your own, and the middleware management API.
• Access control: authentication, roles, task ownership, the permissions API and credentials of TES instances.
• Deployment: the Helm chart for Kubernetes.
• Operations: the Celery workers, Flower, logs, the owners of tasks, and verifying images and charts.
• Upgrading: changes to existing deployments and data.
• Development: development setup, checks, documentation, versioning and releases.
• Contributing: how to contribute, the code of conduct, pull requests, commit messages and the Python guidelines.
• API reference: the code, generated from its docstrings.
• Changelog: release notes.

Architecture

proTES provides central features to a federated network of independently operated TES instances: it can serve, for example, as a compatibility layer, a load balancer, a public entry point to an enclave of independent compute nodes, or a means of collecting telemetry.

When proTES receives a task, it applies the configured middlewares before forwarding the task to a TES instance of the network. A plugin system makes it easy to write middlewares for specific requirements, such as access control, request or response processing or validation, or the selection of suitable TES instances considering data use restrictions and client preferences. Two middlewares are shipped with proTES, as proofs of concept for different ways of distributing tasks (see Built-in middlewares): randomly, for load balancing, and to the TES instances closest to the task's input data, to bring compute to the data.

proTES is a Flask app built with FOCA, configured declaratively in YAML. It stores incoming and forwarded tasks in a MongoDB database. Forwarded tasks are tracked asynchronously by Celery workers, which can be scaled up, via a RabbitMQ broker. A Helm chart and a Docker Compose setup are provided for deployments on Kubernetes and for development, respectively.

License

This project is covered by the Apache License 2.0 also shipped with this repository.

Contact

The project is a collaborative effort under the umbrella of ELIXIR On-Cloud. Follow the link to get in touch with us via chat or email. Please mention the name of this service for any inquiry, proposal, question etc.