Metadata-Version: 2.1
Name: streamflow
Version: 0.0.31
Summary: StreamFlow framework
Home-page: https://github.com/alpha-unito/streamflow
Author: Iacopo Colonnelli
Author-email: iacopo.colonnelli@unito.it
License: UNKNOWN
Download-URL: https://github.com/alpha-unito/streamflow/releases
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.8, <4
Description-Content-Type: text/markdown
License-File: LICENSE

# StreamFlow

[![Build Status](https://travis-ci.com/alpha-unito/streamflow.svg?branch=master)](https://travis-ci.com/alpha-unito/streamflow)

The StreamFlow framework is a container-native *Workflow Management System (WMS)* written in Python 3.
It has been designed around two main principles:
* Allow the execution of tasks in **multi-container environments**, in order to support concurrent execution
of multiple communicating tasks in a multi-agent ecosystem.
* Relax the requirement of a single shared data space, in order to allow for **hybrid workflow** executions on top of
multi-cloud or hybrid cloud/HPC infrastructures.

## Use StreamFlow

#### PyPI
 
The StreamFlow module is available on [PyPI](https://pypi.org/project/streamflow/), so you can install it using pip.

```bash
pip install streamflow
```

Please note that StreamFlow requires `python >= 3.7`. Then you can execute it directly from the CLI

```bash
streamflow /path/to/streamflow.yml
```

#### Docker

StreamFlow Docker images are available on [Docker Hub](https://hub.docker.com/r/alphaunito/streamflow). In order to run
a workflow inside the StreaFlow image
 - A StreamFlow project, containing a `streamflow.yml` file and all the other relevant dependencies (e.g. a CWL
   description of the workflow steps and a Helm description of the execution environment) need to be mounted as a volume
   inside the container, for example in the `/streamflow/project` folder
 - Workflow outputs, if any, will be stored in the `/streamflow/results` folder. Therefore, it is necessary to mount
   such location as a volume in order to persist the results
 - StreamFlow will save all its temporary files inside the `/tmp/streamflow` location. For debugging purposes, or in
   order to improve I/O performances in case of huge files, it could be useful to mount also such location as a volume
 - The path of the `streamflow.yml` file **inside the container** (e.g. `/streamflow/project/streamflow.yml`) must be
   passed as an argument to the Docker container

The script below gives an example of StreamFlow execution in a Docker container

```bash
docker run -d \
    --mount type=bind,source="$(pwd)"/my-project,target=/streamflow/project \
    --mount type=bind,source="$(pwd)"/results,target=/streamflow/results \
    --mount type=bind,source="$(pwd)"/tmp,target=/tmp/streamflow \
    alphaunito/streamflow \
    /streamflow/project/streamflow.yml
```

#### Kubernetes

It is also possible to execute the StreamFlow container as a `Job` in [Kubernetes](https://kubernetes.io/).
In this case, StreamFlow is able to deploy `Helm` models directly on the parent cluster through the
`ServiceAccount` credentials. In order to do that, the `inCluster` option must be set to `true` for each
involved module on the `streamflow.yml` file

```yaml
models:
  helm-model:
    type: helm
    config:
      inCluster: true
      ...
```

A `Helm` template of a StreamFlow `Job` can be found in the `helm/chart` folder.

Please note that, in case [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) is active on the
Kubernetes cluster, a proper `RoleBinding` must be attached to the `ServiceAccount` object, in order to give
StreamFlow the permissions to manage deployments of pods and executions of tasks.

## Contribute to StreamFlow

StreamFlow uses [pipenv](https://pipenv.kennethreitz.org/en/latest/) to guarantee deterministic builds.
Therefore, the recommended way to manage dependencies is by means of the `pipenv` command.

As a first step, get StreamFlow from [GitHub](https://github.com/alpha-unito/streamflow) 
```bash
git clone git@github.com:alpha-unito/streamflow.git
```

Then you can install all the requred packages using the `pipenv` command
```bash
pip install --user pipenv
cd streamflow
pipenv install
```

Finally, you can run StreamFlow in the generated virtual environment. In order for this to work, it is
necessary to add the streamflow project folder (the one generated by the `git clone` command) to your
`PYTHONPATH` list
```bash
pipenv run python -m streamflow
```

StreamFlow relies on [Travis CI](https://travis-ci.com/) for PyPI and Docker Hub distributions. Therefore, in order to publish a
new version of the software, you only have to augment the version number in `version.py` file.

## StreamFlow Team

Iacopo Colonnelli <iacopo.colonnelli@unito.it> (creator and maintainer)  
Barbara Cantalupo <barbara.cantalupo@unito.it> (maintainer)  
Marco Aldinucci <aldinuc@di.unito.it> (maintainer)

Gaetano Saitta <gaetano.saitta@edu.unito.it> (contributor)  
Alberto Mulone <alberto.mulone@edu.unito.it> (contributor)


