Pipeline for strong scaling experiments contained ~460000 tasks in all of the cases. The picture below shows the execution times of different task types in the pipeline. HyperLoom scalability for a pharmaceutical machine-learning pipeline running on 1, 8, 16 and 64 nodes (24 CPUs each). Deploy virtualized HyperLoom A Platform for Defining and Executing Scientific Pipelines in Distributed Environments},Īuthor=, ArchitectureĮxecute your first HyperLoom pipeline in 4 easy steps using Docker: 1.
Define loom full#
Define loom code#
Upload your code to `/home/ubuntu/web` on both `` and ``. You then need a `modules/roles/manifests/web.pp` that sets up `/etc/init/web.conf` to run your app in `/home/ubuntu/web`.ģ. You must also define a directory for your apps to live in with `env.app_root`.įor example, suppose this was your `fabfile.py`: **init**: The name of the Upstart script to start/restart after uploading. **post-upload**: A script to run on each server after uploading.
Define loom install#
to build static assets or install local dependencies). **build**: A script to run locally before uploading (e.g. **role** (required): The role that the app will be uploaded to.
**repo** (required): A Git URL of the repo that contains your app. It is a dictionary where the key is the name of the app and the value is a dictionary with these keys: It assumes you've set up a role for the app (e.g., "web"), and that role has all of the packages you require and an Upstart init script to start the app.Īpps in Loom are configured using `env.apps`. Loom includes a bunch of Fabric tasks for building and uploading code.
Define loom update#
Then you could use the included "all" task to update Puppet on all your hosts: Because this is just Fabric, you can write a task in `fabfile.py` to do it deploy_puppet(): Then install third party Puppet modules, upload your local modules, and apply them:Įvery time you make a change to your modules, you can run that command to apply them. First, bootstrap the host (in this example, the single db host you defined in `env.roledefs`): (If you've used Puppet before, this is a replacement for `node` definitions.)įor example, `modules/roles/manifests/db.pp` defines what the db role is: Roles are defined in a magic module called `roles` which contains manifests for each role. Your own modules are put in a directory called `modules/` in the same directory as `fabfile.py`. (This is for (), a tool for installing reusable Puppet modules. You can then define any third-party Puppet modules you want in a file called `Puppetfile`: It also includes some Fabric tasks for building and uploading app code – something that is particularly complex to do with Puppet.įirst of all, you create `fabfile.py` and define your hosts: It's useful for both serverless and master/agent Puppet installations. Loom does the stuff Puppet doesn't do well or at all: bootstrapping machines, giving them roles, deploying Puppet code and installing reusable Puppet modules.