Run Jobs

Troubleshooting

Fix common Run Job problems, from jobs that never start to failed commands, time limits and missing output.

This page lists common problems with Run Jobs, grouped by what you were doing, each with its cause and fix.

Waiting to start

A Run Job waits and never starts

Open it on the Run Jobs page to see what it's waiting for:

  • Compute: no compute of the type it asked for is free. It starts when some frees up, or submit it with another GPU type.
  • Your organization's limits: it starts when other instances or Run Jobs finish. See Check your compute limits.
  • Credit: your credit doesn't cover its time limit. Add credit, or submit it with a shorter --timeout.

A Run Job takes a long time to start running

Before the command runs, the job downloads its image. The first run of a large image takes a few minutes; later runs are faster. Billing starts only when the container starts.

Command failures

A Run Job fails right away

Its status turns Failed within seconds of starting. Read its output and the message shown with the job. Common causes:

  • The command isn't in the image. Check the image, or install what it needs as part of the command.
  • The command uses shell syntax. Commands run without a shell, so &&, pipes and $VARIABLES aren't interpreted. Wrap them in a shell:
featherless-cloud exec --gpu-count 1 -- bash -c "pip install -r /workspace/requirements.txt && python /workspace/train.py"

The job can't find my files

Your local files aren't uploaded with the job. Put them on a network volume and mount it, or build them into the image:

featherless-cloud volume cp ./train.py vol://my-data/
featherless-cloud exec --gpu-count 1 --volume my-data:/workspace -- python /workspace/train.py

The GPU runs out of memory

Your framework reports that GPU memory is exhausted, for example HIP out of memory or CUDA out of memory in PyTorch. The model, its activations and the batch don't fit. Lower the batch size, load the model in a lower precision such as bfloat16, or submit with more GPUs.

Every job spends minutes installing packages

Each job starts from a clean container disk. Build the packages into a custom image and run the job from it, or save the setup as a template.

My secret isn't in the environment

Secret variables are files under $FEATHERLESS_SECRETS_PATH, not environment variables. Read them in your code, or export one in the command:

featherless-cloud exec --gpu-count 1 --secret HF_TOKEN=hf_... -- bash -c 'export HF_TOKEN="$(cat "$FEATHERLESS_SECRETS_PATH/HF_TOKEN")" && python train.py'

Time limits and interruptions

A Run Job timed out

Its status is Timed out: it reached its time limit. Raise --timeout, up to 24 hours, or save checkpoints to a network volume and continue in a new job.

The work needs more than 24 hours

A Run Job runs for at most 24 hours. Save checkpoints to a network volume, and submit a new job that resumes from the latest one when the previous one ends.

A Run Job was interrupted

Its status is Interrupted: it was stopped to make room for other work. Submit with --preemption-retries <n> so it starts again, and write checkpoints to a network volume so a retry can resume instead of starting over. See When a Run Job is interrupted.

A Run Job stopped when credit ran out

Running Run Jobs stop and fail when your balance reaches zero, and waiting ones keep waiting. Add credit on the Billing page and submit the job again.

Following and output

I pressed Ctrl-C and the Run Job is still running

Ctrl-C detaches exec from the job; it doesn't cancel it. Follow it again with featherless-cloud runjob logs <job-id> --follow, cancel it with featherless-cloud runjob cancel <job-id>, or submit with --cancel-on-interrupt so Ctrl-C cancels the job.

Part of the output is missing

For each attempt, the first and last 50 MB of output are kept, and output is kept for 30 days. A job that started again after an interruption has one output per attempt; read an earlier one with featherless-cloud runjob logs <job-id> --attempt 1. Write anything you need to keep to a network volume.

A Run Job's files are gone

Its container disk is erased when it ends. Mount a network volume with --volume <name>:<path> and write results there.

Getting help

If you're still stuck, ask us on Discord. Include:

  • the Run Job's ID;
  • the command you submitted, without any secret values;
  • its status and the end of its output, from featherless-cloud runjob logs <job-id>.