[{"data":1,"prerenderedAt":671},["ShallowReactive",2],{"page-\u002Finstances\u002Ftroubleshooting":3,"surround-\u002Finstances\u002Ftroubleshooting":662},{"id":4,"title":5,"body":6,"description":654,"extension":655,"meta":656,"navigation":657,"path":658,"seo":659,"stem":660,"__hash__":661},"docs\u002Finstances\u002Ftroubleshooting.md","Troubleshooting",{"type":7,"value":8,"toc":617},"minimark",[9,13,18,23,26,30,33,37,45,85,88,92,99,125,129,144,148,151,155,159,170,174,193,197,200,204,211,215,222,263,267,278,282,293,297,304,308,312,357,361,372,380,384,395,399,406,410,413,417,421,428,457,471,475,484,544,548,551,555,566,570,580,584,593,613],[10,11,12],"p",{},"This page lists common problems with instances, grouped by what you were doing, each with its cause and fix.",[14,15,17],"h2",{"id":16},"starting-and-stopping","Starting and stopping",[19,20,22],"h3",{"id":21},"the-instance-takes-a-long-time-to-start","The instance takes a long time to start",[10,24,25],{},"The first start of an image downloads it, which takes a few minutes for a large image. Later starts of the same image are faster. Billing starts only when the container starts.",[19,27,29],{"id":28},"the-instance-wont-start-again","The instance won't start again",[10,31,32],{},"Starting needs a free GPU of the same model and credit for one hour of the instance's rate. If GPUs of that model are busy, try again later, or deploy a new instance on another GPU model.",[19,34,36],{"id":35},"the-instance-keeps-restarting","The instance keeps restarting",[10,38,39,40,44],{},"Its main process exits. Open the ",[41,42,43],"strong",{},"Logs"," tab for the error. Common causes:",[46,47,48,64,70],"ul",{},[49,50,51,54,55,59,60,63],"li",{},[41,52,53],{},"The start command uses shell features"," such as ",[56,57,58],"code",{},"&&",", pipes or ",[56,61,62],{},"$VARIABLES",". It runs without a shell; put the commands in a script and run the script.",[49,65,66,69],{},[41,67,68],{},"The start command finishes."," An instance runs as long as its main process does. Start a long-running server, or leave the start command empty to keep the image's own.",[49,71,72,79,80,78],{},[41,73,74,75,78],{},"The image isn't built for ",[56,76,77],{},"linux\u002Famd64","."," See ",[81,82,84],"a",{"href":83},"\u002Ftemplates\u002Fcustom-images#check-that-your-image-will-run","Check that your image will run",[10,86,87],{},"An instance that crashes 10 times in a row, each time within 10 minutes of starting, is stopped for you.",[19,89,91],{"id":90},"the-instance-stopped-on-its-own","The instance stopped on its own",[10,93,94,95,98],{},"Open its ",[41,96,97],{},"Activity"," tab to see why:",[46,100,101,116],{},[49,102,103,106,107,115],{},[41,104,105],{},"Credit ran out:"," add credit on the ",[81,108,114],{"href":109,"rel":110,"target":113},"https:\u002F\u002Fconsole.cloud.featherless.ai\u002Fbilling",[111,112],"noopener","noreferrer","_blank","Billing page",", then start the instance again.",[49,117,118,121,122,78],{},[41,119,120],{},"It kept crashing:"," see ",[81,123,36],{"href":124},"#the-instance-keeps-restarting",[19,126,128],{"id":127},"my-files-or-installed-packages-are-gone-after-a-stop-or-restart","My files or installed packages are gone after a stop or restart",[10,130,131,132,135,136,139,140,78],{},"They were on the container disk, which is erased when an instance stops or restarts, including the restart that applies an edit. Keep files on a network volume mounted at ",[56,133,134],{},"\u002Fworkspace",", and keep a ",[56,137,138],{},"requirements.txt"," on the volume so reinstalling packages is one command. See ",[81,141,143],{"href":142},"\u002Fstorage#choose-where-to-keep-each-file","Choose where to keep each file",[19,145,147],{"id":146},"i-cant-see-an-instance-a-teammate-deployed","I can't see an instance a teammate deployed",[10,149,150],{},"It's in another organization. Switch organizations from the menu at the top of the console's sidebar.",[14,152,154],{"id":153},"running-code","Running code",[19,156,158],{"id":157},"my-code-cant-find-the-gpu","My code can't find the GPU",[10,160,161,162,165,166,169],{},"Your framework reports no GPU, for example ",[56,163,164],{},"torch.cuda.is_available()"," returns ",[56,167,168],{},"False"," in PyTorch. Either the instance runs on CPU compute, or the image is built for the other GPU vendor: ROCm images need AMD GPUs, and CUDA images need NVIDIA GPUs. Deploy again with a matching template and GPU.",[19,171,173],{"id":172},"the-gpu-runs-out-of-memory","The GPU runs out of memory",[10,175,176,177,180,181,184,185,188,189,192],{},"Your framework reports that GPU memory is exhausted, for example ",[56,178,179],{},"HIP out of memory"," or ",[56,182,183],{},"CUDA out of memory"," in PyTorch. The model, its activations and the batch don't fit. Lower the batch size, load the model in a lower precision such as ",[56,186,187],{},"bfloat16",", or deploy with more GPUs and split the model across them. The ",[41,190,191],{},"Metrics"," tab shows each GPU's memory over time.",[19,194,196],{"id":195},"a-program-prints-killed","A program prints Killed",[10,198,199],{},"The instance ran out of memory, and the program using the most was stopped. Load less data at once, or deploy a larger size: each GPU comes with its own share of memory, and CPU sizes list theirs.",[19,201,203],{"id":202},"downloads-or-installs-fail-with-no-space-left-on-device","Downloads or installs fail with No space left on device",[10,205,206,207,78],{},"The container disk or a network volume is full. See ",[81,208,210],{"href":209},"\u002Fstorage\u002Ftroubleshooting#downloads-or-installs-fail-with-no-space-left-on-device","Storage troubleshooting",[19,212,214],{"id":213},"my-secret-isnt-in-the-environment","My secret isn't in the environment",[10,216,217,218,221],{},"Secret variables aren't environment variables. Each one is a file under ",[56,219,220],{},"$FEATHERLESS_SECRETS_PATH",". Read it in your code, or export it for tools that expect an environment variable:",[223,224,229],"pre",{"className":225,"code":226,"language":227,"meta":228,"style":228},"language-bash shiki shiki-themes github-dark-default","export HF_TOKEN=\"$(cat \"$FEATHERLESS_SECRETS_PATH\u002FHF_TOKEN\")\"\n","bash","",[56,230,231],{"__ignoreMap":228},[232,233,236,240,244,247,251,255,258,260],"span",{"class":234,"line":235},"line",1,[232,237,239],{"class":238},"suJrU","export",[232,241,243],{"class":242},"sZEs4"," HF_TOKEN",[232,245,246],{"class":238},"=",[232,248,250],{"class":249},"s9uIt","\"$(",[232,252,254],{"class":253},"sQhOw","cat",[232,256,257],{"class":249}," \"",[232,259,220],{"class":242},[232,261,262],{"class":249},"\u002FHF_TOKEN\")\"\n",[19,264,266],{"id":265},"a-gated-hugging-face-model-wont-download","A gated Hugging Face model won't download",[10,268,269,270,273,274,277],{},"Hugging Face needs your token for gated models. Add it as a secret variable named ",[56,271,272],{},"HF_TOKEN"," when you deploy, export it as shown in ",[81,275,214],{"href":276},"#my-secret-isnt-in-the-environment",", and accept the model's license on its Hugging Face page.",[19,279,281],{"id":280},"models-download-again-every-time-the-instance-starts","Models download again every time the instance starts",[10,283,284,285,288,289,292],{},"Hugging Face caches models on the container disk, which a stop erases. Set the environment variable ",[56,286,287],{},"HF_HOME"," to a folder on a network volume, such as ",[56,290,291],{},"\u002Fworkspace\u002Fhf",", so models download once.",[19,294,296],{"id":295},"my-programs-output-isnt-in-the-logs","My program's output isn't in the logs",[10,298,299,300,78],{},"Logs show only the instance's main process. Output from programs you start in the terminal or over SSH stays in that terminal. See ",[81,301,303],{"href":302},"\u002Finstances\u002Flogs-and-metrics#send-your-programs-output-to-the-logs","Send your program's output to the logs",[14,305,307],{"id":306},"connecting","Connecting",[19,309,311],{"id":310},"ssh-says-permission-denied-publickey","SSH says Permission denied (publickey)",[46,313,314,326,339,349],{},[49,315,316,319,320,325],{},[41,317,318],{},"The key isn't added:"," your organization's SSH keys are on the ",[81,321,324],{"href":322,"rel":323,"target":113},"https:\u002F\u002Fconsole.cloud.featherless.ai\u002Fcredentials\u002Fssh-keys",[111,112],"Credentials page",". Add the public key of the key you connect with.",[49,327,328,331,332,335,336,78],{},[41,329,330],{},"Your SSH client offers a different key:"," name it with ",[56,333,334],{},"-i",", for example ",[56,337,338],{},"ssh -i ~\u002F.ssh\u002Fid_ed25519 \u003Cinstance-id>@ssh.cloud.featherless.ai",[49,340,341,344,345,348],{},[41,342,343],{},"The instance isn't running, or the ID is wrong:"," copy the command from the instance's ",[41,346,347],{},"Connect"," tab.",[49,350,351,121,354,356],{},[41,352,353],{},"The instance runs a custom image without SSH:",[81,355,84],{"href":83},", or use the browser terminal.",[19,358,360],{"id":359},"vs-code-or-cursor-cant-connect","VS Code or Cursor can't connect",[10,362,363,364,367,368,371],{},"Remote-SSH connects to a host from your SSH config. Add an entry for the instance to ",[56,365,366],{},"~\u002F.ssh\u002Fconfig",", then connect to ",[56,369,370],{},"my-instance",":",[223,373,378],{"className":374,"code":376,"language":377,"meta":228},[375],"language-text","Host my-instance\n  HostName ssh.cloud.featherless.ai\n  User \u003Cinstance-id>\n  IdentityFile ~\u002F.ssh\u002Fid_ed25519\n","text",[56,379,376],{"__ignoreMap":228},[19,381,383],{"id":382},"a-program-stopped-when-i-disconnected","A program stopped when I disconnected",[10,385,386,387,390,391,78],{},"It ran in a plain SSH session, which ends its programs when it closes. Start long-running work in tmux, from the browser terminal or with ",[56,388,389],{},"featherless-cloud instance tmux my-instance",", and it keeps running after you disconnect. See ",[81,392,394],{"href":393},"\u002Finstances\u002Fconnect#keep-programs-running-with-tmux","Keep programs running with tmux",[19,396,398],{"id":397},"the-browser-terminal-closed","The browser terminal closed",[10,400,401,402,405],{},"It closes after 30 minutes without activity. Programs in its tmux session keep running; open the ",[41,403,404],{},"Terminal"," tab again to get back to them.",[19,407,409],{"id":408},"my-tmux-sessions-are-gone","My tmux sessions are gone",[10,411,412],{},"tmux sessions end when the instance stops or restarts. Restart the programs, or make the main one the instance's start command so it starts with the instance.",[14,414,416],{"id":415},"public-ports","Public ports",[19,418,420],{"id":419},"a-public-port-returns-503-with-connection-refused","A public port returns 503 with Connection refused",[10,422,423,424,427],{},"Nothing is accepting connections on that port inside the instance. Either no service is running, it listens only on ",[56,425,426],{},"127.0.0.1",", or it stopped when the session that started it ended. Check from the instance's terminal:",[223,429,431],{"className":225,"code":430,"language":227,"meta":228,"style":228},"curl -s -o \u002Fdev\u002Fnull -w \"%{http_code}\\n\" localhost:8000\n",[56,432,433],{"__ignoreMap":228},[232,434,435,438,442,445,448,451,454],{"class":234,"line":235},[232,436,437],{"class":253},"curl",[232,439,441],{"class":440},"sFSAA"," -s",[232,443,444],{"class":440}," -o",[232,446,447],{"class":249}," \u002Fdev\u002Fnull",[232,449,450],{"class":440}," -w",[232,452,453],{"class":249}," \"%{http_code}\\n\"",[232,455,456],{"class":249}," localhost:8000\n",[10,458,459,460,463,464,467,468,78],{},"If that prints ",[56,461,462],{},"200"," but the address still returns ",[56,465,466],{},"503",", restart the service listening on ",[56,469,470],{},"0.0.0.0",[19,472,474],{"id":473},"my-web-app-only-listens-on-localhost","My web app only listens on localhost",[10,476,477,478,480,481,483],{},"Many servers listen on ",[56,479,426],{}," unless you tell them otherwise, which makes the public address return ",[56,482,466],{},". Start them on all interfaces:",[485,486,487,500],"table",{},[488,489,490],"thead",{},[491,492,493,497],"tr",{},[494,495,496],"th",{},"Server",[494,498,499],{},"Listen on all interfaces",[501,502,503,514,524,534],"tbody",{},[491,504,505,509],{},[506,507,508],"td",{},"Uvicorn (FastAPI)",[506,510,511],{},[56,512,513],{},"uvicorn app:app --host 0.0.0.0 --port 8000",[491,515,516,519],{},[506,517,518],{},"Flask",[506,520,521],{},[56,522,523],{},"flask run --host 0.0.0.0 --port 8000",[491,525,526,529],{},[506,527,528],{},"Gradio",[506,530,531],{},[56,532,533],{},"demo.launch(server_name=\"0.0.0.0\", server_port=8000)",[491,535,536,539],{},[506,537,538],{},"JupyterLab",[506,540,541],{},[56,542,543],{},"jupyter lab --ip 0.0.0.0 --port 8888",[19,545,547],{"id":546},"a-connection-to-a-public-port-drops-after-5-minutes","A connection to a public port drops after 5 minutes",[10,549,550],{},"A connection with no traffic for 300 seconds is closed. Send keepalive messages, such as WebSocket pings, more often than that.",[19,552,554],{"id":553},"a-vllm-server-returns-401","A vLLM server returns 401",[10,556,557,558,561,562,78],{},"vLLM started with an API key rejects requests without it. Send the key as a bearer token: ",[56,559,560],{},"-H \"Authorization: Bearer $VLLM_API_KEY\"",". See ",[81,563,565],{"href":564},"\u002Fexamples\u002Fvllm-server","Serve an LLM with vLLM",[19,567,569],{"id":568},"a-model-servers-address-doesnt-answer-yet","A model server's address doesn't answer yet",[10,571,572,573,575,576,579],{},"The server is still downloading or loading the model, which takes minutes for large models. Follow its progress on the ",[41,574,43],{}," tab; vLLM prints ",[56,577,578],{},"Application startup complete."," when it's ready.",[14,581,583],{"id":582},"getting-help","Getting help",[10,585,586,587,592],{},"If you're still stuck, ask us on ",[81,588,591],{"href":589,"rel":590,"target":113},"https:\u002F\u002Fdiscord.com\u002Finvite\u002F7gybCMPjVA",[111,112],"Discord",". Include:",[46,594,595,602,610],{},[49,596,597,598,601],{},"the instance ID, from the instance's page or ",[56,599,600],{},"featherless-cloud instance get \u003Cname>",";",[49,603,604,605,180,607,609],{},"what you did, and the exact error from the ",[41,606,43],{},[41,608,97],{}," tab;",[49,611,612],{},"when it happened.",[614,615,616],"style",{},"html pre.shiki code .suJrU, html code.shiki .suJrU{--shiki-default:#FF7B72}html pre.shiki code .sZEs4, html code.shiki .sZEs4{--shiki-default:#E6EDF3}html pre.shiki code .s9uIt, html code.shiki .s9uIt{--shiki-default:#A5D6FF}html pre.shiki code .sQhOw, html code.shiki .sQhOw{--shiki-default:#FFA657}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sFSAA, html code.shiki .sFSAA{--shiki-default:#79C0FF}",{"title":228,"searchDepth":618,"depth":618,"links":619},2,[620,629,639,646,653],{"id":16,"depth":618,"text":17,"children":621},[622,624,625,626,627,628],{"id":21,"depth":623,"text":22},3,{"id":28,"depth":623,"text":29},{"id":35,"depth":623,"text":36},{"id":90,"depth":623,"text":91},{"id":127,"depth":623,"text":128},{"id":146,"depth":623,"text":147},{"id":153,"depth":618,"text":154,"children":630},[631,632,633,634,635,636,637,638],{"id":157,"depth":623,"text":158},{"id":172,"depth":623,"text":173},{"id":195,"depth":623,"text":196},{"id":202,"depth":623,"text":203},{"id":213,"depth":623,"text":214},{"id":265,"depth":623,"text":266},{"id":280,"depth":623,"text":281},{"id":295,"depth":623,"text":296},{"id":306,"depth":618,"text":307,"children":640},[641,642,643,644,645],{"id":310,"depth":623,"text":311},{"id":359,"depth":623,"text":360},{"id":382,"depth":623,"text":383},{"id":397,"depth":623,"text":398},{"id":408,"depth":623,"text":409},{"id":415,"depth":618,"text":416,"children":647},[648,649,650,651,652],{"id":419,"depth":623,"text":420},{"id":473,"depth":623,"text":474},{"id":546,"depth":623,"text":547},{"id":553,"depth":623,"text":554},{"id":568,"depth":623,"text":569},{"id":582,"depth":618,"text":583},"Fix common instance problems, from SSH errors and unreachable ports to GPUs your code can't see.","md",{},true,"\u002Finstances\u002Ftroubleshooting",{"title":5,"description":654},"instances\u002Ftroubleshooting","qYRxaaJ2Gb4tfG7zykpojKILjYX98BvRVvqevs6-Ye8",[663,667],{"title":664,"path":665,"description":666},"Use the CLI in an instance","\u002Finstances\u002Fcli-access","Script work across your organization from inside an instance, with the CLI already signed in.",{"title":668,"path":669,"description":670},"Run Jobs overview","\u002Frun-jobs","Train, evaluate and process data on GPUs you pay for only while the work runs.",1794932598298]