Stable Diffusion Tutorial: Learn AI Art from Scratch (2026)
Stable Diffusion democratized AI image generation by running on consumer GPUs. Unlike DALL-E or Midjourney, you can run it locally, fine-tune it on your own data, and understand exactly how it works. After training custom LoRAs and fine-tuning SDXL on architectural styles, I am still amazed that a 2 GB model can generate coherent images from text prompts.
This tutorial covers the diffusion process, prompt engineering for image models, ControlNet for spatial control, fine-tuning with LoRA, and deployment for production inference. All examples use the diffusers library from Hugging Face.
How Diffusion Models Work
Diffusion models learn to reverse a gradual noising process. During training, clean images are corrupted by adding Gaussian noise over many timesteps. The model learns to predict the added noise at each step. During generation, the model starts with pure noise and iteratively removes noise (denoises) to produce a clean image guided by a text prompt.
The U-Net architecture processes the noisy latent representation, while the text encoder (CLIP) embeds the prompt into the conditioning signal. Cross-attention layers let the image generation attend to specific words. The entire process happens in latent space (compressed 8x by a VAE), reducing computational cost dramatically.
from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained(
'runwayml/stable-diffusion-v1-5',
torch_dtype=torch.float16
).to('cuda')
prompt = 'a photograph of a futuristic city with flying cars, sunset lighting'
image = pipe(
prompt,
num_inference_steps=30,
guidance_scale=7.5,
generator=torch.Generator('cuda').manual_seed(42)
).images[0]
image.save('output.png')Prompt Crafting for Images
Prompt engineering for image models follows: subject + medium + style + lighting + composition + quality modifiers. Example: 'a majestic wolf howling at the moon, digital art, fantasy style, volumetric lighting, epic composition, highly detailed, 8K.' The order matters — subjects early in the prompt receive more attention.
Negative prompts are equally important: 'blurry, low quality, distorted, extra limbs, bad anatomy, watermark, text.' The guidance_scale parameter controls how strongly the model adheres to the prompt. Higher values (12-15) produce more literal results but can reduce diversity.
prompt = 'cinematic photo of a samurai standing in a bamboo forest at dawn'
negative_prompt = 'blurry, low quality, cartoon, anime, extra limbs, watermark'
image = pipe(
prompt,
negative_prompt=negative_prompt,
num_inference_steps=40,
guidance_scale=9.0,
height=768,
width=512
).images[0]ControlNet
ControlNet adds spatial conditioning inputs (edge maps, depth maps, pose skeletons) to guide the diffusion process. Instead of describing a pose in text, you provide an OpenPose skeleton and the model follows it precisely. This is a game-changer for character consistency and scene composition.
ControlNet models are loaded as separate subnetworks that connect to the main U-Net. Useful preprocessors: Canny (edges), HED (soft edges), Depth (MiDaS), OpenPose (poses), and Scribble (rough sketches). You can combine multiple ControlNets for fine-grained control.
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline
controlnet = ControlNetModel.from_pretrained('lllyasviel/sd-controlnet-canny')
pipe = StableDiffusionControlNetPipeline.from_pretrained(
'runwayml/stable-diffusion-v1-5',
controlnet=controlnet,
torch_dtype=torch.float16
).to('cuda')
image = cv2.Canny(np.array(load_image('input_photo.jpg')), 100, 200)
result = pipe(
prompt='a detailed pencil sketch of this scene',
image=image,
num_inference_steps=25
).images[0]Fine-Tuning with LoRA
LoRA (Low-Rank Adaptation) fine-tunes a diffusion model on a specific concept by training small rank-decomposition matrices attached to the cross-attention layers. The base model stays frozen; only the LoRA weights (typically 10-50 MB) are trained. This makes LoRA fast to train and easy to distribute.
Training requires a dataset of 10-100 images of your concept. Use captions that describe each image. The training loop uses the same noise-prediction objective as the base model, but only updates LoRA parameters. Inference loads the LoRA weights on top of the base model.
pipe = StableDiffusionPipeline.from_pretrained('stable-diffusion-v1-5', torch_dtype=torch.float16).to('cuda')
pipe.load_lora_weights('./my-style-lora', weight_name='pytorch_lora_weights.safetensors')
image = pipe(
'a portrait in my trained style',
cross_attention_kwargs={'scale': 0.8}
).images[0]Image-to-Image and Inpainting
Image-to-image generation starts from an existing image and modifies it according to a prompt while preserving the original composition. The strength parameter (0.0-1.0) controls how much of the original is kept. Inpainting is a special case that modifies only a masked region.
Inpainting models are trained with missing regions filled by noise. This is useful for removing objects, replacing backgrounds, or extending images beyond boundaries (outpainting). SDXL handles 1024x1024 resolution natively.
from diffusers import StableDiffusionImg2ImgPipeline
pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
'runwayml/stable-diffusion-v1-5', torch_dtype=torch.float16
).to('cuda')
init_image = load_image('sketch.png').resize((512, 512))
result = pipe(
prompt='turn this sketch into a detailed oil painting',
image=init_image,
strength=0.75,
guidance_scale=7.5,
num_inference_steps=40
).images[0]Production Deployment
Serving Stable Diffusion in production requires GPU inference with optimizations. Use torch.compile() for 20-40% speedup, fp16 to halve memory, and xformers for memory-efficient attention. A single A10G (24 GB VRAM) can serve SDXL at ~1 image/second with batch size 1.
For high throughput, use batching and request queuing via Redis + Celery or Triton Inference Server. Always add content moderation (NSFW filter) before returning images to users.
pipe = StableDiffusionXLPipeline.from_pretrained(
'stabilityai/stable-diffusion-xl-base-1.0',
torch_dtype=torch.float16,
variant='fp16',
use_safetensors=True
).to('cuda')
pipe.enable_xformers_memory_efficient_attention()
pipe.unet = torch.compile(pipe.unet, mode='reduce-overhead', fullgraph=True)
images = pipe(
['prompt 1', 'prompt 2'],
num_images_per_prompt=1,
num_inference_steps=25,
guidance_scale=7.5
).imagesFrequently Asked Questions
What hardware do I need to run Stable Diffusion locally?
SD 1.5 needs 4 GB VRAM minimum (runs on GTX 1060 6GB). SDXL needs 8 GB VRAM (RTX 3070 or better). Apple Silicon Macs with 16 GB unified memory run well via MPS backend. CPU-only generation is possible but takes 5-10 minutes per image.
How do I generate consistent characters across images?
Use a fixed seed for reproducibility. For truly consistent characters, train a LoRA on 10-20 images of the character's face from different angles. Then use the same LoRA + seed across generations.
What is the difference between SD 1.5, SDXL, and SD 3?
SD 1.5 (512x512) has the most community LoRA support. SDXL (1024x1024) has better composition. SD 3 (2024) uses a new MM DiT architecture with improved text rendering and hands. All use different VAE and text encoders.
How do I avoid NSFW content generation?
The safety checker in diffusers flags NSFW content. Add server-side moderation (AWS Rekognition, Sightengine, or CLIP-based classifiers) and require users to accept content policies. Never allow unfiltered generation in a public app.
Originally published on Ayodhyyya. Last updated June 1, 2026.