Troubleshooting
Use this section to identify and resolve common GPU, deployment, runtime, and API issues.
Hardware acceleration, distributed model execution, and low-latency serving involve intricate interactions between CUDA runtimes, GPU memory, network ingress, and container environments. Follow the checklists below to quickly diagnose and fix common roadblocks.
GPU Out of Memory (OOM) #
If your workload exceeds available GPU memory (e.g. CUDA out of memory. Tried to allocate...), follow these solutions:
Choose a GPU with more VRAM
Upgrade to higher tier cards. For example, switch from 16GB/24GB GPUs (RTX 4090) to 80GB enterprise GPUs (NVIDIA A100 or H100 SXM5).
Reduce batch size
Lower the micro-batch size in your inference or training script (e.g. reduce batch from 16 to 4 or 1) to lower activation memory footprints.
Reduce model size
Switch to smaller parameter variants (e.g., 8B instead of 70B parameters) or adopt distilled foundation architectures.
Use lower precision where supported
Load models using FP16, BF16, or 4-bit/8-bit quantization schemes (AWQ, GPTQ, BitsAndBytes) to cut memory usage by 50% to 75%.
Optimize your model configuration
Enable FlashAttention-2, gradient checkpointing, CPU offloading, or configure PagedAttention KV cache parameters in vLLM.
Model Fails to Load #
When containers crash on startup or models fail to initialize, check the following checklist:
1. Model files
Verify safe tensors or bin weights are not corrupted or partially downloaded. Ensure adequate disk space on attached volumes.
2. Dependencies
Ensure required Python libraries (e.g. accelerate, transformers, sentencepiece) match model versions.
3. CUDA compatibility
Check that PyTorch/TensorFlow build was compiled against the exact CUDA toolkit version (e.g. CUDA 12.1 vs 12.4).
4. Environment variables
Ensure HF_TOKEN is supplied for gated repositories (Llama 3, Gemma) and CUDA_VISIBLE_DEVICES=0 is correctly exposed.
5. GPU memory
Confirm the model fits into device VRAM during initial weights allocation before serving requests.
6. Container configuration
Set shared memory --ipc=host or shm-size to at least 16GB to prevent PyTorch DataLoader crashes.
Slow Inference #
If request latency is higher than expected or token throughput is subpar, check:
GPU utilization & Throttling
Run nvidia-smi to check GPU compute load, clock rates, and power limits. Low utilization often signals a CPU-bound bottleneck or slow data loading.
Model size & Architecture
Large parameter counts require substantial memory bandwidth. Consider switching to specialized architectures or applying TensorRT optimizations.
Input size & Prompt length
Long token contexts incur quadratic or high KV-cache attention costs. Use chunked prefill or context window limits where feasible.
Batch size & Concurrency
Batch sizes that are too small underutilize Tensor Cores; batch sizes that are too large cause queue latency spikes.
Model loading time & Cold Starts
Cache model weights on network volumes or keep minimum active replicas in Serverless to eliminate cold start delays.
Runtime configuration
Use modern serving engines such as vLLM or TensorRT-LLM instead of raw Hugging Face pipeline() calls.
API Request Fails #
If API calls to your endpoints or InstantGPU serverless gateways return errors, verify:
API key
Confirm the API key is active, has adequate account credits, and is not expired or revoked.
Endpoint URL
Ensure the URL format matches https://api.instantgpu.ai/v1/... with no trailing typos.
HTTP method
Confirm you are using POST for inference requests and GET for status checking.
Request parameters
Check parameter names, types, and values against the model's schema requirements.
Content type
Always include Content-Type: application/json in your HTTP request headers.
Authorization header
Verify the header syntax: Authorization: Bearer YOUR_API_KEY with a space after Bearer.
Deployment Fails #
Review deployment logs in the InstantGPU console and verify:
GPU availability
Verify that the selected accelerator model (e.g. H100 SXM5, RTX 4090) has active capacity in your selected region.
Container configuration
Ensure the container image name and tag are publicly pullable or valid registry pull secrets are supplied.
Dependencies & Environment
Review startup logs for fatal Python tracebacks or missing system packages during entrypoint execution.
Startup command
Ensure your container's startup command binds to host 0.0.0.0 (not 127.0.0.1) so external ingress can route traffic.
Connection Problems #
If you cannot connect to your GPU instance or pod, verify:
Instance status
Ensure the instance is marked Running in the console dashboard.
Connection credentials
Confirm the private SSH key file matches the public key uploaded to your profile.
SSH configuration
Specify the mapped external SSH port flag (e.g. ssh -p 22421 root@<ip>).
Network configuration
Check corporate firewall rules and VPN settings that may block custom non-standard TCP ports.
Exposed ports
Ensure web server ports (e.g. 8888 for JupyterLab, 7860 for Gradio, 8000 for FastAPI) are exposed in the pod settings.