Examples

Serve an LLM with vLLM

Serve an open model behind an OpenAI-compatible API on your own GPU.

This example deploys the vLLM template on one GPU, serves Qwen/Qwen2.5-7B-Instruct at a public HTTPS address with an API key, and calls it with the OpenAI chat completions API. The model files are cached on a network volume, so later starts skip the download.

What you need

  • An API key for your server, for example from openssl rand -hex 24. Clients send it to call the model.

Step 1: Create a network volume for the model cache

Open the Storage page, click Create volume, name it models, set Capacity to 100, and click Create volume.

Step 2: Deploy the server

Choose the template

Open the Instances page and click Deploy instance. Name the instance vllm. Under Template, search for vLLM and select vLLM (ROCm).

Choose compute

Under Compute, choose a GPU model and 1 GPU.

Mount the volume

Under Network volumes, click Add network volume, choose models, and keep the mount path /workspace.

Expose the API port

Under Public ports, enter 8000.

Start vLLM

Under Start command, enter:

vllm serve Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 8000

Under Environment variables, add:

NameValue
HF_HOME/workspace/hf
VLLM_API_KEYYour API key

Launch

Click Review launch, then Launch instance.

HF_HOME puts the downloaded model on the volume. VLLM_API_KEY makes vLLM reject requests without the key, because anyone can reach the public address.

Step 3: Wait for the model to load

The first start downloads the image and the model, which takes several minutes. Open the instance's Logs tab, or follow them from the CLI, until vLLM reports that it's ready:

featherless-cloud instance logs vllm --follow
(APIServer pid=44) INFO:     Application startup complete.

Step 4: Call the model

Copy the address for port 8000 from the instance's Connect tab, or from featherless-cloud instance get vllm. Then send a chat completion:

curl https://<address>/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Say hello in five words."}],
    "max_tokens": 32
  }'

The response follows the OpenAI format:

{
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello there, friend!" },
      "finish_reason": "stop"
    }
  ]
}

Any OpenAI-compatible client works: set its base URL to https://<address>/v1 and its API key to your VLLM_API_KEY.

Clean up

Stop the instance when you don't need the server; the model stays cached on the volume for the next start. Delete both when you're done:

featherless-cloud instance delete vllm
featherless-cloud volume delete models

Next steps

Expose ports

Limits and security for public addresses.

Save it as a template

Deploy the same server again in one step.