Skip to main content
Transformer Lab

Task YAML Structure

This guide explains how to format YAML files for creating tasks in Transformer Lab. Tasks define jobs that run on compute providers and can include training scripts, evaluation scripts, or any other computational workloads.

Note: For detailed information about defining task parameters with validation and custom UI, see the Task Parameters guide.

Basic Structure​

The basic structure of a task YAML file includes the following sections:

name: task-name
resources:
  compute_provider: provider-name-in-your-transformerlab-workspace
  cpus: 2
  memory: 4
minutes_requested: 60
envs:
  KEY: value
setup: "command"
run: "command"
github_repo_url: "url"
github_repo_dir: "dir"
github_repo_branch: "branch"
parameters: { ... }

The file is validated against a strict schema before the task is created: unknown top-level keys and unknown keys under resources: are rejected, and the error names the offending key. A typo such as runs: or accelerator: fails the submission rather than being silently ignored, so there is no need to guess whether a key took effect.

For backwards compatibility the whole document may also be nested under a single top-level task: key. New files should not do this.

Required Fields​

Two fields are required: name, and run — the command that runs the task, documented in full under Commands. Everything else is optional.

name​

The task name. This will be sanitized to create a safe filename and name of the cluster on the compute provider.

Type: String

Example:

name: my-training-task

Resources Configuration​

The resources section defines the compute resources required for the task.

resources.compute_provider​

The name of the compute provider to use. This should match a configured provider name in your workspace.

Type: String

Example:

resources:
  compute_provider: skypilot-provider

Note: If not specified, the system will use the first available provider as a fallback.

resources.cpus​

Number of CPUs to allocate.

Type: Integer or String

Example:

resources:
  cpus: 4

resources.memory​

Amount of memory to allocate (in GB).

Type: Integer or String

Example:

resources:
  memory: 16

resources.disk_space​

Amount of disk space to allocate (in GB).

Type: Integer or String

Example:

resources:
  disk_space: 100

resources.accelerators​

Accelerator specification (e.g., GPU type and count). Format depends on the provider. To look at supported formats in Skypilot, refer to their accelerator documentation and for Slurm, refer to their GPU documentation.

Type: String

Example:

resources:
  accelerators: "H100:8"

resources.num_nodes​

Number of nodes for distributed training.

Type: Integer

Example:

resources:
  num_nodes: 2

resources.fleet_name​

Name of an existing dstack fleet to run on, for workspaces whose provider is dstack. Ignored by every other provider.

Type: String

Example:

resources:
  fleet_name: "my-h100-fleet"

Complete Resources Example:

resources:
  compute_provider: aws-ec2
  cpus: 8
  memory: 32
  disk_space: 200
  accelerators: "1xA100"
  num_nodes: 1

Also accepted: instance_type, cloud, region, zone, use_spot and image_id. These exist so a SkyPilot-style resources block validates instead of erroring, but task creation does not currently act on them — set the machine you want through accelerators, cpus, memory and the provider's own configuration. They are listed here so you know a file carrying them is valid, not so you reach for them.

Commands​

setup​

Command(s) to run before the main task execution. This is typically used for installing dependencies, setting up the environment, or downloading data.

Type: String

Example:

setup: "pip install -r requirements.txt"

Multi-line Setup:

setup: |
  pip install -r requirements.txt
  apt-get update
  apt-get install -y git
  python download_data.py

run​

The main command to execute for the task. This is the primary script or command that performs the actual work. Required.

Type: String

Example:

run: "python train.py"

With Arguments:

run: "python train.py --epochs 10 --batch-size 32"

Multi-line Run:

run: |
  python train.py \
    --epochs 10 \
    --batch-size 32 \
    --learning-rate 2e-5

Environment Variables​

envs​

Environment variables to set for the task execution. These are passed as key-value pairs.

Type: Dictionary (key-value pairs)

Example:

envs:
  CUDA_VISIBLE_DEVICES: "0"
  OPENAI_API_KEY: "{{secrets.OPENAI_API_KEY}}"
  INTERNAL_API_BASE: "https://api.internal.example.com"

Referencing secrets​

Do not paste credentials into task.yaml — the file is stored, versioned and shared like any other part of the task. Write {{secrets.NAME}} instead ({{secret.NAME}}, singular, works too) and the value is substituted when the job launches, from your team's secrets merged with your own — user secrets win where both define a name. Manage them with lab team secret (add --user for a secret that is yours rather than the whole team's).

Placeholders resolve in envs:, parameters:, setup: and run:. A name that has no matching secret is left as-is rather than blanked, so an unresolved {{secrets.OPENAI_API_KEY}} reaching your script is the signal that the secret was never set. Values are redacted from job logs; the unresolved placeholder is not.

Four names are managed separately because the platform sets them up for you — _HF_TOKEN, _WANDB_API_KEY, _GITHUB_PAT_TOKEN and _NGROK_AUTH_TOKEN. They live alongside your other secrets, so reference them the same way, keeping the leading underscore: HF_TOKEN: "{{secrets._HF_TOKEN}}". Set them from Settings or with lab team secret set _HF_TOKEN, which recognises the reserved name and stores it through the right path for you.

From Python you can skip the indirection entirely and call lab.get_secret(), which reads the same store.

Quota Tracking​

minutes_requested​

Estimated number of minutes the task will run. This is used for quota tracking and resource allocation. When specified, a quota hold is created to reserve the estimated compute time.

Type: Integer

Example:

minutes_requested: 60

Note: This is an optional field but recommended for tasks running on remote compute providers to enable quota tracking and better resource management.

GitHub Integration​

github_repo_url​

GitHub repository URL to clone before running the task. The repository will be cloned to the working directory.

Type: String

Example:

github_repo_url: "https://github.com/username/repo.git"

github_repo_dir​

Subdirectory within the GitHub repository to use as the working directory. Useful when the repository contains multiple projects.

Type: String

Example:

github_repo_url: "https://github.com/username/multi-project-repo.git"
github_repo_dir: "project1"

github_repo_branch​

Branch of the GitHub repository to clone. Defaults to the default branch if not specified. Type: String Example:

github_repo_url: "https://github.com/username/multi-project-repo.git"
github_repo_dir: "project1"
github_repo_branch: "main"

Note: The final path where the cloned folder would be available is either: ~/github_repo_dir or ~/github_repo_name (if no directory is specified).

Complete GitHub Example:

github_repo_url: "https://github.com/transformerlab/examples.git"
github_repo_dir: "training/llm-finetuning"
setup: "pip install -r requirements.txt"
run: "python train.py"

Parameters​

parameters​

Task parameters (hyperparameters, configuration, etc.) that will be accessible via lab.get_config() in your scripts. These are passed to the job and can be used to configure the training or evaluation process.

Detailed documentation on this field is on its own page

Type: Dictionary (any JSON-serializable values)

Example:

parameters:
  model_name: "gpt2"
  learning_rate: 2e-5
  batch_size: 8
  num_epochs: 3
  max_seq_length: 512
  warmup_ratio: 0.03
  weight_decay: 0.01

Nested Parameters:

parameters:
  model:
    name: "gpt2"
    architecture: "GPT2LMHeadModel"
  training:
    learning_rate: 2e-5
    batch_size: 8
    num_epochs: 3
  data:
    dataset_name: "wikitext"
    max_seq_length: 512

Note: Parameters can be accessed in your Python scripts using the Lab SDK:

from lab import lab

lab.init()
config = lab.get_config()
learning_rate = config.get("learning_rate")
model_name = config.get("model_name")

📖 For comprehensive parameter documentation, including:

  • Parameter types (int, float, bool, enum, string, json, model, dataset)
  • Schema validation (min, max, multipleOf)
  • UI customization (ui_widget options)
  • Special model and dataset selectors
  • Complete examples

See the Task Parameters guide.

Complete Examples​

Example 1: Simple Training Task​

name: simple-training
resources:
  compute_provider: local
  cpus: 4
  memory: 8
minutes_requested: 30
setup: "pip install transformers torch"
run: "python train.py"
parameters:
  model_name: "gpt2"
  learning_rate: 2e-5
  batch_size: 8
  num_epochs: 3

Example 2: Training Task with GitHub Repository​

name: finetune-llm
resources:
  compute_provider: skypilot-provider
  cpus: 8
  memory: 32
  accelerators: "H100:1"
minutes_requested: 120
github_repo_url: "https://github.com/username/llm-training.git"
github_repo_dir: "finetuning"
setup: |
  pip install -r requirements.txt
  pip install wandb
envs:
  WANDB_API_KEY: "{{secrets._WANDB_API_KEY}}"
  HF_TOKEN: "{{secrets._HF_TOKEN}}"
run: "python train.py"
parameters:
  model_name: "meta-llama/Llama-2-7b-hf"
  dataset_name: "wikitext-2"
  learning_rate: 2e-5
  batch_size: 4
  gradient_accumulation_steps: 8
  num_epochs: 3
  max_seq_length: 512
  warmup_ratio: 0.03
  weight_decay: 0.01

Example 3: Evaluation Task​

name: evaluate-model
resources:
  compute_provider: local
  cpus: 2
  memory: 4
setup: "pip install transformers datasets"
run: "python evaluate.py"
parameters:
  model_name: "gpt2"
  dataset_name: "wikitext"
  batch_size: 16
  max_samples: 1000

Best Practices​

  1. Use Descriptive Names: Choose clear, descriptive task names that indicate what the task does.

    name: finetune-gpt2-wikitext  # Good
    name: task1                    # Bad

  2. Specify Resources Appropriately: Match resources to your workload. Don't request more than you need, but ensure you have enough for the task.

    # For small models
    resources:
      cpus: 4
      memory: 8

    # For large models
    resources:
      cpus: 16
      memory: 64
      accelerators: "1xA100"

  3. Use Setup for Dependencies: Install dependencies in the setup command rather than in the run command.

    setup: "pip install -r requirements.txt" # Good
    run: "python train.py"

  4. Store Sensitive Data Securely: Don't hardcode API keys or tokens in YAML files. Reference a secret and let the platform substitute it at launch (see Referencing secrets). Shell-style ${VAR} is not expanded by the platform — it reaches the job as the literal text.

    # Good - resolved from your team's secrets when the job launches
    envs:
      WANDB_API_KEY: "{{secrets._WANDB_API_KEY}}"

    # Bad - hardcoded, and stored with the task
    envs:
      WANDB_API_KEY: "abc123xyz"

  5. Use Parameters for Configuration: Store hyperparameters and configuration in the parameters section so they're accessible via lab.get_config().

    parameters:
      learning_rate: 2e-5
      batch_size: 8

  6. Use GitHub for Code: Store your code in a GitHub repository and reference it with github_repo_url, github_repo_dir, and github_repo_branch rather than uploading files manually.

    github_repo_url: "https://github.com/username/my-project.git"
    github_repo_dir: "training"
    github_repo_branch: "main"

  7. Test Locally First: Test your task configuration locally before running on expensive cloud resources.

    resources:
      compute_provider: local # Test locally first

  8. Use Multi-line Strings for Long Commands: Use YAML's | or > syntax for multi-line commands.

    setup: |
      pip install -r requirements.txt
      python download_data.py
      python preprocess_data.py

  9. Validate YAML Syntax: Ensure your YAML is valid before submitting. Use a YAML validator or linter.

Common Issues and Solutions​

Issue: YAML Parsing Errors​

Problem: Invalid YAML syntax causes parsing errors.

Solution: Validate your YAML syntax. Common issues:

  • Missing colons after keys
  • Incorrect indentation (use spaces, not tabs)
  • Unquoted strings with special characters

Issue: Parameters Not Accessible​

Problem: Parameters defined in YAML are not accessible via lab.get_config().

Solution: Ensure parameters are at the root level under parameters: key:

parameters:
  learning_rate: 2e-5 # Correct

Not:

config:
  parameters:
    learning_rate: 2e-5 # Wrong

Issue: Provider Not Found​

Problem: compute_provider name doesn't match any configured provider.

Solution: Check the exact provider name in your workspace. The system will use the first available provider as a fallback, but it's better to specify the correct name.