Serverless

Overview

Pay-as-you-go compute for AI models and compute-intensive workloads. Review configuration and operations guidance for InstantGPU Serverless.

InstantGPU Serverless is a cloud computing platform that lets you serve AI models for inferenceInference is the process of using a trained machine learning model to make predictions or generate outputs from new input data. and run other compute-intensive workloads without managing servers. You only pay for the actual compute time you use, with no idle costs when your application isn't processing requests.

Get started #

Quickstart

Write a handler function, build a worker image, create an endpoint, and send your first request.

Generate images with ComfyUI

Deploy a ComfyUI worker and generate images using text prompts and API workflows.

Fork a worker repository

Use InstantGPU's worker templates on GitHub as a starting point for custom containerized AI services.

Call an Endpoint in Code #

Send an inference request to an active serverless endpoint using your language of choice:

import instantgpu

# Configure your API key
instantgpu.api_key = "igpu_live_9a8b7c6d5e4f3a2b1c"

# Run inference on serverless endpoint
endpoint = instantgpu.Endpoint("vllm-llama-3")

run_request = endpoint.run({
    "prompt": "Explain quantum computing in one sentence.",
    "max_tokens": 100,
    "temperature": 0.7
})

# Stream or await response
print("Response:", run_request.output)

curl -X POST "https://api.instantgpu.ai/v2/vllm-llama-3/runsync" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer igpu_live_9a8b7c6d5e4f3a2b1c" \
  -d '{
    "input": {
      "prompt": "Explain quantum computing in one sentence.",
      "max_tokens": 100,
      "temperature": 0.7
    }
  }'

import { InstantGPU } from "@instantgpu/sdk";

const igpu = new InstantGPU({ apiKey: process.env.INSTANTGPU_API_KEY });

async function main() {
  const response = await igpu.endpoint("vllm-llama-3").runSync({
    input: {
      prompt: "Explain quantum computing in one sentence.",
      max_tokens: 100
    }
  });

  console.log(response.output);
}

main();

Core concepts #

InstantGPU Serverless combines container isolation with dedicated enterprise GPUs to give you instant autoscaling:

Endpoints

An endpoint acts as the single HTTP/REST interface for your service. It automatically handles queuing, load distribution, rate limiting, and request retries.

Workers & Scaling

Workers are lightweight Docker containers that process jobs from your endpoint queue. Workers scale up dynamically as demand increases, and scale to 0 when idle.

Pay-per-second Billing

Unlike traditional cloud VMs that bill by the hour or month even when idling, Serverless bills only for the exact milliseconds your GPU spends executing code.

Flash Boot & Volumes

Attach high-speed network storage volumes to share model weights (like LLaMA, Mistral, Flux) across all workers with sub-second cold start times.