How to Deploy Your Own Text-to-Image API on a Budget VPS (No GPU Required)

Stable Diffusion has a reputation problem: most tutorials assume you own a GPU with 12GB+ of VRAM. If you've ever looked into self-hosting image generation and closed the tab after seeing "requires NVIDIA GPU" in the requirements, you're not the only one. The reality is that CPU-only image generation has quietly become viable, and you can now run a production-ready text-to-image API on a $20/month VPS with no graphics card at all.

This guide walks through exactly how to do that using vps-diffusion-api, an open-source project built specifically to make CPU-native AI image generation practical on ordinary server hardware. By the end, you'll have a self-hosted image generation API running on your own VPS, reachable from anywhere, with full control over cost, uptime, and data.

Why Run Your Own Self-Hosted Image Generation API?

Before getting into setup, it's worth understanding what problem this actually solves.

No GPU rental costs. Cloud GPU instances for Stable Diffusion often run $0.50–$3+ per hour. A CPU-only VPS with 4 vCPUs and 8GB of RAM typically costs a fraction of that per month, running continuously.

Predictable, private infrastructure. Every image generation API call happens on hardware you control. No third-party service sees your prompts, and there's no per-image billing from an external provider.

A real, documented REST API. Rather than wiring together a Python notebook, you get a FastAPI service with interactive Swagger docs, ready to be called from any application, script, or frontend.

Built for constrained hardware, not despite it. This isn't a hacked-together workaround — the project is purpose-built around Intel's OpenVINO toolkit and SD-Turbo, a distilled Stable Diffusion model designed for single-step inference. That combination is what makes CPU-only generation fast enough to be usable at all.

How It Works Under the Hood

Two technical choices make this project different from just "running Stable Diffusion on a CPU and hoping for the best":

• OpenVINO handles graph optimization and thread affinity, essentially recompiling the model's execution graph to take advantage of your specific CPU's instruction set (AVX-512 support on Intel chips gives the biggest boost).
• SD-Turbo replaces the usual 20–50 step generation process with single-step inference. Instead of denoising an image over dozens of iterations, it's calibrated to produce a full image in one pass, which is what makes multi-second (rather than multi-minute) CPU generation realistic.

On top of that, the project layers in smart memory management — lazy loading the model only when needed and auto-unloading it after idle periods — plus a built-in request queue that prevents concurrent generations from fighting over CPU cores and crashing the container.

Step 1: Check Your VPS Meets the Minimum Requirements

This isn't a "throw it on any $5 droplet" situation — the model compilation step is memory-hungry, and underprovisioned hosts will hit the Linux Out-Of-Memory killer mid-setup. Before you deploy, confirm your VPS meets these minimums:

• CPU: 4 dedicated vCPUs, x86 architecture (Intel preferred, for AVX-512 support)
• RAM: 8 GB minimum — anything less risks an OOM crash during the initial model load
• Storage: 20 GB of NVMe SSD, for fast container boot and model weight caching

If you're choosing a provider specifically for this project, look for one that offers dedicated (not shared/burstable) vCPUs — burstable cores tend to throttle right when OpenVINO's graph compilation needs sustained CPU time.

Step 2: Install Docker and Docker Compose

The entire deployment is containerized, so the only real prerequisite on the server side is Docker with Compose support. Once your VPS is provisioned and you've SSH'd in, install Docker using your distro's standard method, then confirm both are available:

docker --version
docker compose version

If either command fails, install Docker Engine (which bundles the Compose plugin on modern versions) before continuing.

Step 3: Clone the Repository and Deploy

With Docker in place, deploying the API is a three-command process:

git clone https://github.com/TechMitten/vps-diffusion-api.git
cd vps-diffusion-api
docker compose up -d --build

That single docker compose up command does the heavy lifting: it pulls the container image, downloads the OpenVINO-optimized model weights, compiles the execution graph for your CPU's specific architecture, and starts the FastAPI service — all automatically.

The first build will take some time since it's downloading model weights and compiling the inference graph. Once it's done, your image generation API is live on port 8000.

Step 4: Test the API

You can confirm everything is working with a single curl request:

curl -X POST "http://<YOUR_VPS_IP>:8000/generate" \
     -H "Content-Type: application/json" \
     -d '{"prompt": "A retro futuristic sports car on a neon highway", "steps": 1, "guidance_scale": 0.0}' \
     --output result.jpg

Replace <YOUR_VPS_IP> with your server's actual address. If everything is configured correctly, this saves a generated JPEG named result.jpg to your current directory.

Prefer a visual check instead of the command line? Navigate to http://<YOUR_VPS_IP>:8000/docs in a browser. This loads an interactive Swagger UI where you can type in prompts, tweak parameters, and preview generated images directly — useful for quick testing without writing any client code.

Step 5: Use the Recommended SD-Turbo Settings

SD-Turbo isn't a drop-in replacement for standard Stable Diffusion settings — it's calibrated specifically for single-step generation, and using typical multi-step values will actually hurt your results. Stick to these parameters:

If you're integrating this into an application, hardcode these as your defaults rather than exposing them as free-form user settings — they're what makes the "fast" part of CPU inference actually possible.

Step 6: Understand Real-World Performance Expectations

Because the API includes lazy loading and auto-unloading to conserve RAM, response times vary a lot depending on whether the model is already warm. On a 4-vCPU / 8GB host, expect roughly:

• Cold start (first request, or after the 5-minute idle unloader clears memory): ~70 seconds — this includes model loading, OpenVINO graph compilation, and the actual inference/decode step.
• Graph warm-up (second request): ~47 seconds, as OpenVINO finishes finalizing execution optimizations for the dynamic shape graph.
• Warm pipeline (subsequent requests): ~19–21 seconds, with the actual denoising step taking only around 5.5 seconds — most of the remaining time is VAE decoding.

The practical takeaway: this setup is not built for sub-second, chatbot-style image generation. It's built for asynchronous or batch-style workloads — generating thumbnails, content assets, or images queued from a backend job — where a 20-second warm response time is perfectly acceptable.

Step 7: Tune Memory and CPU Limits for Your Host

The default docker-compose.yml ships with example resource limits, but these need to be adjusted to match your actual server specs — using the defaults on a differently-sized VPS risks either OOM crashes or leaving your host CPU-starved.

Adjusting memory:

Check your total RAM first:

free -m | awk '/^Mem:/{print $2}'

Then apply this sizing rule:

• 8GB+ total RAM → reserve 1GB for the host, allocate the rest to the container
• 4–8GB total RAM → reserve 512MB for the host
• Below 4GB → allocate about 85% of total RAM to the container

Update the memory line under deploy.resources.limits in docker-compose.yml accordingly — for example, on a 8GB host:

deploy:
  resources:
    limits:
      cpus: '3.8'
      memory: 6000M

Adjusting CPU allocation:

Check available cores:

nproc --all

Then apply a similar rule of thumb:

• 8+ vCPUs → reserve 1 full core for the host
• 4–8 vCPUs → reserve 0.5 core
• Below 4 vCPUs → allocate ~85% of total cores to the container

After editing either value, restart the stack for changes to take effect:

docker compose down && docker compose up -d --build

Common Pitfalls to Watch For

• Skipping the memory sizing step. The example values in docker-compose.yml are just that — examples. Running them unmodified on a smaller or larger host than they were written for is the most common cause of crashes during setup.
• Ignoring OOM kills during first load. If the container dies during the initial model compilation, it's almost always a memory issue. On genuinely low-RAM hosts, enabling a swap file can prevent allocation failures during that first compile.
• Changing steps or guidance_scale away from the recommended values. SD-Turbo's quality depends on staying within its calibrated single-step regime — bumping steps up "for better quality" the way you would with standard Stable Diffusion will produce worse results, not better ones.
• Not accounting for cold-start latency in your application. If you're calling this API from a user-facing app, build in a loading state for that first ~70-second cold request rather than assuming every call will return in 20 seconds.
• Choosing a provider with burstable, shared vCPUs. Graph compilation and inference both need sustained CPU access; throttled or shared cores will make performance wildly inconsistent. Check for steal time with top or vmstat 1 if generation feels unexpectedly slow.

Is This Setup Right for Your Project?

If you need real-time, sub-second image generation for an interactive product, CPU-only inference isn't the right tool — that's still a GPU's job. But for a huge range of use cases — batch content generation, background jobs, internal tools, low-traffic side projects, or anywhere a 20-second turnaround is acceptable — running your own OpenVINO and SD-Turbo powered API on a cheap VPS is a genuinely practical way to self-host image generation without paying for GPU infrastructure you'd barely use.

The entire stack — Docker, FastAPI, OpenVINO, and SD-Turbo — is open source and configurable, so once it's running, you own the whole pipeline: your prompts, your images, your uptime, and your costs.

Full source, Dockerfile, and configuration details are available in the vps-diffusion-api repository on GitHub.