Configuration File

By default, Spotty is looking for a spotty.yaml file in the root directory of the project. This file describes parameters of a remote instance and an environment for the project. Here is a basic example of such file for AWS:

project:
  name: my-project-name
  syncFilters:
    - exclude:
      - .git/*
      - .idea/*
      - '*/__pycache__/*'

container:
  projectDir: /workspace/project
  image: tensorflow/tensorflow:latest-gpu-py3-jupyter
  ports: [6006, 8888]
  volumeMounts:
    - name: workspace
      mountPath: /workspace

instances:
  - name: i1
    provider: aws
    parameters:
      region: eu-west-1
      instanceType: p2.xlarge
      volumes:
        - name: workspace
          parameters:
            size: 10

For GCP, project and container configurations are exactly the same, but an instance configuration is slightly different, you can find the specification here.

Available Parameters

Configuration file consists of 4 sections: project, container, instances and scripts.

project section:

  • name - the name of your project. It will be used to create an S3 bucket and a CloudFormation stack to run an instance.

  • syncFilters (optional) - filters to skip some directories or files during synchronization. By default, all project files will be synced with the instance. Example:

      syncFilters:
        - exclude:
            - .idea/*
            - .git/*
            - data/*
        - include:
            - data/test/*
        - exclude:
            - data/test/dump.json
    

    It will skip “.idea/”, “.git/” and “data/” directories except the “data/test/” directory. All files from the “data/test/” directory will be synced with the instance except the “data/test/dump.json” file.

    You can read more about filters here: Use of Exclude and Include Filter.

container section:

  • projectDir - a directory inside the container where the local project will be copied. If it’s a subdirectory of a container volume, the project will be located on that volume, otherwise, the data will be lost once the instance is terminated.

  • image (optional) - the name of the Docker image that contains the environment for your project. For example, you could use TensorFlow image for GPU (tensorflow/tensorflow:latest-gpu-py3-jupyter). It already contains NumPy, SciPy, scikit-learn, pandas, Jupyter Notebook and TensorFlow itself. If you need to use your own image, you can specify the path to your Dockerfile in the file parameter (see below), or push your image to the Docker Hub.

  • file (optional) - relative path to your custom Dockerfile.

    Note: make sure that the build context for the Dockerfile doesn’t contain gigabytes of training data or some other heavy data (keep the Dockerfile in a separate directory or use the .dockerignore file). Otherwise, you would get an out-of-space error, because Docker copies the entire build context to the Docker daemon during the build. Read more here: “docker build” command.

    Example: if you use TensorFlow and need to download your dataset from S3, you could install AWS CLI on top of the original TensorFlow image. Just create the Dockerfile file in the docker/ directory of your project:

      FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
        
      RUN pip install --upgrade awscli
    

    Then set the file parameter to the docker/Dockerfile value.

  • volumeMounts (optional) - where to mount instance volumes into the container’s filesystem. Each element of a list has the following parameters:
    • name - this must match the name of an instance volume.
    • mountPath - a path within the container at which the volume should be mounted.
  • workingDir (optional) - working directory for your custom scripts (see “scripts” section below),

  • commands (optional) - commands which should be performed once your container is started. For example, you could download your datasets from an S3 bucket to the project directory (see “project” section):
      commands: |
        aws s3 sync s3://my-bucket/datasets/my-dataset /workspace/project/data
    
  • ports (optional) - list of ports to open. For example:
      ports: [6006, 8888]
    

    It will open ports 6006 for Jupyter Notebook and 8008 for TensorBoard.

  • runtimeParameters (optional) - a list of additional parameters for the container runtime. For example:
      runtimeParameters: ['--privileged', '--shm-size', '2G']
    

instances section:

This section contains a list of instances. Each instance is described with the following parameters:

  • name - a name of the instance. Use this name to manage the instance with the commands like “spotty start” or “spotty stop”. Also Spotty uses this name in the names of AWS resources.

  • provider - a provider for the instance. At the moment Spotty supports “aws” (Amazon Web Services). and “gcp” (Google Cloud Platform) providers.

  • parameters - parameters of the instance. These parameters are different for different providers:

scripts section (optional):

This section contains customs scripts which can be run using the spotty run <SCRIPT_NAME> command. The following example defines scripts train, jupyter and tensorflow:

train: |
  PYTHONPATH=/workspace/project
  python /workspace/project/model/train.py --model-name {{MODEL}}
jupyter: |
  jupyter notebook --allow-root --notebook-dir=/workspace/project
tensorboard: |
  tensorboard --logdir /workspace/outputs

For example, training can be started using the following command:

spotty run train -p MODEL=my-model

Use the Ctrl + b, then d combination of keys to be detached from the SSH session - the script will keep running. Then use the same command again to be reattached to the running script.

Note: don’t forget to use the “|” character for multi-line scripts, otherwise the YAML parser will merge multiple lines together.