This page lists common problems with Run Jobs, grouped by what you were doing, each with its cause and fix.
Open it on the Run Jobs page to see what it's waiting for:
--timeout.Before the command runs, the job downloads its image. The first run of a large image takes a few minutes; later runs are faster. Billing starts only when the container starts.
Its status turns Failed within seconds of starting. Read its output and the message shown with the job. Common causes:
&&, pipes and $VARIABLES aren't interpreted. Wrap them in a shell:featherless-cloud exec --gpu-count 1 -- bash -c "pip install -r /workspace/requirements.txt && python /workspace/train.py"
Your local files aren't uploaded with the job. Put them on a network volume and mount it, or build them into the image:
featherless-cloud volume cp ./train.py vol://my-data/
featherless-cloud exec --gpu-count 1 --volume my-data:/workspace -- python /workspace/train.py
Your framework reports that GPU memory is exhausted, for example HIP out of memory or CUDA out of memory in PyTorch. The model, its activations and the batch don't fit. Lower the batch size, load the model in a lower precision such as bfloat16, or submit with more GPUs.
Each job starts from a clean container disk. Build the packages into a custom image and run the job from it, or save the setup as a template.
Secret variables are files under $FEATHERLESS_SECRETS_PATH, not environment variables. Read them in your code, or export one in the command:
featherless-cloud exec --gpu-count 1 --secret HF_TOKEN=hf_... -- bash -c 'export HF_TOKEN="$(cat "$FEATHERLESS_SECRETS_PATH/HF_TOKEN")" && python train.py'
Its status is Timed out: it reached its time limit. Raise --timeout, up to 24 hours, or save checkpoints to a network volume and continue in a new job.
A Run Job runs for at most 24 hours. Save checkpoints to a network volume, and submit a new job that resumes from the latest one when the previous one ends.
Its status is Interrupted: it was stopped to make room for other work. Submit with --preemption-retries <n> so it starts again, and write checkpoints to a network volume so a retry can resume instead of starting over. See When a Run Job is interrupted.
Running Run Jobs stop and fail when your balance reaches zero, and waiting ones keep waiting. Add credit on the Billing page and submit the job again.
Ctrl-C detaches exec from the job; it doesn't cancel it. Follow it again with featherless-cloud runjob logs <job-id> --follow, cancel it with featherless-cloud runjob cancel <job-id>, or submit with --cancel-on-interrupt so Ctrl-C cancels the job.
For each attempt, the first and last 50 MB of output are kept, and output is kept for 30 days. A job that started again after an interruption has one output per attempt; read an earlier one with featherless-cloud runjob logs <job-id> --attempt 1. Write anything you need to keep to a network volume.
Its container disk is erased when it ends. Mount a network volume with --volume <name>:<path> and write results there.
If you're still stuck, ask us on Discord. Include:
featherless-cloud runjob logs <job-id>.