Stable Diffusion Integration: Custom Image Generation Pipeline

Clients often complain that Midjourney offers no control over object poses, DALL·E becomes expensive at 5000 queries per day, and the license for generated images remains unclear. An open-source model like Stable Diffusion solves these problems: you get a self-hosted system with fine-tuning capabilities tailored to your business. Below is how we do it in practice and what a typical project includes.

What Problems Does Self-Hosted Stable Diffusion Solve?

We integrate ready-made LoRA models for specific art styles (anime, photorealism, 3D render) or custom fine-tuning on your dataset. We attach ControlNet for precise object positioning, and inpainting for local editing. For example, for an e‑commerce client we trained a LoRA on 500 product photos — the result: unified background generation in 2 seconds on an RTX 4090.

At 1000+ generations per day, cloud API costs become significant. For a client processing 5,000 images/day, switching to self-hosted SDXL saved $2,400/month on API costs. A self-hosted solution on an RTX 4090 pays for itself in 3–4 months, especially if you use quantized models (INT8) to reduce VRAM. The savings at scale can reach 60% compared to cloud services.

A task queue on Redis + Celery handles dozens of concurrent requests, while xFormers or Flash Attention 2 accelerate each generation by 20–30%.

How We Set Up the Generation Pipeline: Stack and Patterns

Our main tool is the diffusers library from Hugging Face. We use SDXL as the base model, attach a Refiner for final polish, and LoRA for styling. For comparison, SD 1.5 generates 512×512 with noticeably poorer detail, while SDXL produces 1024×1024 with quality close to Midjourney. This Stable Diffusion integration leverages latent diffusion and classifier-free guidance (CFG) to control output fidelity.

from diffusers import (
    StableDiffusionXLPipeline,
    StableDiffusionXLImg2ImgPipeline,
    StableDiffusionXLInpaintPipeline,
    DPMSolverMultistepScheduler
)
import torch
from PIL import Image
import io

class StableDiffusionService:
    def __init__(self, model_path: str = "stabilityai/stable-diffusion-xl-base-1.0"):
        self.pipe = StableDiffusionXLPipeline.from_pretrained(
            model_path,
            torch_dtype=torch.float16,
            use_safetensors=True,
            variant="fp16"
        )
        # Optimised sampler with Karras noise scheduling
        self.pipe.scheduler = DPMSolverMultistepScheduler.from_config(
            self.pipe.scheduler.config,
            use_karras_sigmas=True
        )
        self.pipe.to("cuda")

        # Optional VRAM optimizations
        self.pipe.enable_model_cpu_offload()
        self.pipe.enable_vae_tiling()

    def generate(
        self,
        prompt: str,
        negative_prompt: str = "nsfw, low quality, blurry, watermark, text",
        width: int = 1024,
        height: int = 1024,
        steps: int = 30,
        guidance_scale: float = 7.5,
        seed: int = None
    ) -> bytes:
        generator = torch.Generator("cuda").manual_seed(seed) if seed else None

        image = self.pipe(
            prompt=prompt,
            negative_prompt=negative_prompt,
            width=width,
            height=height,
            num_inference_steps=steps,
            guidance_scale=guidance_scale,
            generator=generator
        ).images[0]

        buf = io.BytesIO()
        image.save(buf, format="PNG")
        return buf.getvalue()

According to Hugging Face documentation, the DPMSolverMultistepScheduler with Karras sigmas provides faster convergence and quality.

SDXL Refiner for final polish:

from diffusers import StableDiffusionXLImg2ImgPipeline

refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0",
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16"
)
refiner.to("cuda")

def generate_with_refiner(prompt: str, steps: int = 40) -> bytes:
    # Base generates latent vector
    image = base_pipe(
        prompt=prompt,
        num_inference_steps=steps,
        denoising_end=0.8,
        output_type="latent"
    ).images

    # Refiner adds details
    image = refiner(
        prompt=prompt,
        num_inference_steps=steps,
        denoising_start=0.8,
        image=image
    ).images[0]

    buf = io.BytesIO()
    image.save(buf, format="PNG")
    return buf.getvalue()

Why LoRA Is Better Than Full Fine-Tuning?

Full fine-tuning requires 24+ GB VRAM even for SD 1.5 and takes hours. LoRA is a set of low-rank matrices (rank 16–64, 10–100 MB) that are attached on top of the pretrained model. We use peft to load multiple LoRAs simultaneously, combining styles with different weights. For example, 70% "anime" style + 30% "realistic textures".

# Load LoRA for a specific style
pipe.load_lora_weights("./loras/anime_style_v2.safetensors")
pipe.fuse_lora(lora_scale=0.8)

# Multiple LoRAs simultaneously
pipe.load_lora_weights("lora1.safetensors", adapter_name="style1")
pipe.load_lora_weights("lora2.safetensors", adapter_name="style2")
pipe.set_adapters(["style1", "style2"], adapter_weights=[0.7, 0.3])

Comparison of Approaches: LoRA vs Full Fine-Tuning

How We Accelerate Generation?

We use fp16 computation, enable_vae_tiling to reduce peak VRAM, and enable_model_cpu_offload to partially offload to CPU. For batch processing we apply torch.compile to optimize the graph. In production we set up a request balancer on RabbitMQ with multiple GPU workers.

Performance by GPU

xFormers or Flash Attention 2 speed up by 20–30% with the same VRAM.

Example REST API wrapper:

from fastapi import FastAPI, BackgroundTasks
from pydantic import BaseModel
import uuid

app = FastAPI()
sd_service = StableDiffusionService()

class GenerateRequest(BaseModel):
    prompt: str
    negative_prompt: str = ""
    width: int = 1024
    height: int = 1024
    steps: int = 30
    seed: int = None

@app.post("/generate")
async def generate(req: GenerateRequest, background_tasks: BackgroundTasks):
    job_id = str(uuid.uuid4())
    background_tasks.add_task(
        process_generation, job_id, req.dict()
    )
    return {"job_id": job_id}

@app.get("/result/{job_id}")
async def get_result(job_id: str):
    status = redis_client.get(f"job:{job_id}")
    return json.loads(status) if status else {"status": "not_found"}

Process of Work

1. Analysis. We gather requirements: number of generations, needed LoRA/ControlNet, GPU budget.
2. Design. We choose the stack (SDXL + Refiner, queue), design the API (REST/WebSocket) and storage (S3/MinIO).
3. Implementation. We write a FastAPI wrapper, configure LoRA and ControlNet, and set up monitoring (Prometheus + Grafana).
4. Testing. We run 100+ generations with varying parameters, measure p99 latency and throughput.
5. Deployment. On your server or cloud (AWS SageMaker / Google Vertex AI). We hand over documentation and auto-scaling scripts.

What's Included

• Ready-made REST API wrapper with /generate and /result endpoints (async queue).
• Support for LoRA, ControlNet, inpainting, img2img.
• Deployment documentation for your infrastructure.
• Training for your team (1-2 hours online).
• Compatibility guarantee with your database (via generation log table).

Estimated Timeframes

API wrapper over SDXL — from 3 to 5 days. Self-hosted service with queue and storage — from 1 to 2 weeks. The cost is calculated individually based on integration complexity and the need for fine-tuning. Contact us — we'll find the best configuration for your budget.

Our Stable Diffusion integration provides efficient image generation using SDXL, LoRA, and ControlNet. We have 10+ years of experience in AI/ML and 40+ implemented image generation pipelines for e‑commerce, game dev, and advertising. We guarantee 24/7 stable operation and provide post-launch support. Get in touch to discuss the details.