NanoDiffuser

One-step text-to-image diffusion, running locally in your browser.

NanoDiffuser uses a custom-modified version of Stable Diffusion, adapted and quantized to fit a tight sub-400 MiB model budget. Compact ONNX graphs, a JavaScript inference pipeline, and a minimal interface generate 512 × 512 images using the visitor's GPU through WebGPU. Hosting serves static files; image generation requires no inference server, API key, or server-side GPU.

The modifications cover model conversion, mixed-precision quantization, browser compatibility, local inference, and static deployment. No additional training or fine-tuning was performed for this package. The approximately 382 MiB budget covers the inference graphs, weights, tokenizer, and configuration; JavaScript and WebAssembly assets are additional. In decimal units, the model assets are approximately 400.5 MB.

At a glance

Property	Configuration
Base	Custom-modified Stable Diffusion adapted for a sub-400 MiB model budget
Output	One 512 × 512 RGB image, downloadable as PNG
Sampling	One denoising step, classifier-free guidance disabled
Scheduler	Exported one-step DEIS coefficients
Text encoder	CLIP, asymmetric Q4 MatMul/Gather weights
Denoiser	U-Net, Q4 matrix weights + per-channel INT8 convolution storage, with FP16 retained where needed
Decoder	TAESD / AutoencoderTiny, FP16
Model assets	Approximately 382 MiB, excluding JavaScript and WebAssembly runtime assets
Execution	ONNX Runtime Web with WebGPU; tokenization and orchestration in JavaScript
Hosting	Static HTML, CSS, JavaScript, WebAssembly, and model files

Getting started

Open the NanoDiffuser demo, use the ready-to-go prompt or enter your own, and select Generate image. Use a browser with WebGPU enabled.

How generation works

Prompt → CLIP tokenizer → Q4 CLIP encoder → FP16 text embeddings
Seed → Gaussian noise → one-step U-Net → scheduler update
                         ↓
                   FP16 TAESD → pixels → canvas → PNG

The tokenizer runs locally through Transformers.js, pads or truncates the prompt to 77 tokens, and supplies an INT32 [1, 77] tensor. CLIP produces [1, 77, 768] text embeddings, which the browser converts to FP16 for the U-Net.

A seeded JavaScript random generator and Box–Muller transform create Gaussian noise with shape [1, 4, 64, 64]. The U-Net receives the scaled noise, text embeddings, and timestep 999. Its prediction is combined with the initial latent using coefficients saved from the one-step DEIS scheduler:

initial_latent = noise × initNoiseSigma
model_input   = initial_latent × modelInputScale
denoised      = sampleCoefficient × initial_latent
              + outputCoefficient × noise_prediction

The current manifest contains:

{
  "timestep": 999.0,
  "initNoiseSigma": 1.0,
  "modelInputScale": 1.0,
  "sampleCoefficient": 14.642590522766113,
  "outputCoefficient": -14.579278945922852
}

These values are part of this export's sampling contract. The browser uses the recorded values directly. It does not reconstruct Diffusers' scheduler internals or apply an extra classifier-free guidance pass.

The denoised latent is converted to FP16 and passed to TAESD. The decoder wrapper maps its output into [0, 1]; JavaScript converts the channel-first output into 8-bit RGBA pixels for the canvas and PNG export.

Fixed seeds make the JavaScript noise sequence repeatable. They do not imply pixel-identical results to PyTorch, which uses a different random generator, or across different browser/GPU implementations.

Conversion and quantization

The supplied sdxs512_webgpu_colab.ipynb exports three separate graphs using PyTorch's ONNX exporter, evaluation mode, constant folding, and fixed dimensions. Export starts with ONNX opset 17; the U-Net is subsequently set to opset 21 for its FP16 dequantization operations.

Component	Conversion	Graph + external weights
CLIP	FP32 export; asymmetric Q4 weight-only quantization of eligible MatMul and Gather operations	63.36 MiB
U-Net	FP16 export; mixed Q4 matrix weights and per-output-channel INT8 convolution storage	312.71 MiB
TAESD	FP16 decoder export	2.36 MiB

Q4 quantization uses ONNX Runtime's weight-only quantizer with block size 128, asymmetric zero-points, QOperator format, and accuracy_level=2. Quantized matrix multiplication uses packed MatMulNBits weights. The graph checks require packed UINT8 zero-points.

U-Net convolution weights are quantized per output channel to signed INT8, with FP16 scales and a DequantizeLinear node before convolution. INT8 reduces weight storage; the convolution still executes in floating point. Input/output convolutions and timestep-projection MatMuls are excluded from their respective quantization passes and retain FP16 weights.

The U-Net uses Diffusers' explicit AttnProcessor during export. TAESD is loaded from the source checkpoint's vae subfolder; the export includes its decoder path.

Compatibility handling includes:

• Keeping CLIP export and LayerNormalization parameters in FP32, then converting final embeddings to FP16 at the U-Net boundary.
• A repair utility for older exports with mixed FP32/FP16 LayerNormalization parameters: test-site/scripts/repair_text_encoder_layernorm.py.
• Normalizing reduction axes into attributes or inputs according to the graph's opset.
• Using U-Net opset 21 so FP16 dequantization scales produce weights compatible with FP16 convolution inputs.
• Removing unreferenced initializers and consolidating external weights into one .data file per graph.
• Saving the tokenizer, fixed tensor shapes, filenames, and measured scheduler coefficients alongside the weights.

The Colab notebook is configured for a CUDA GPU, such as a T4. Its installation cell requests a session restart before export. Conversion can use several gigabytes of temporary disk space; the final release is much smaller. This conversion introduces no new training dataset.

Execution, memory, and caching

The browser pipeline uses onnxruntime-web/webgpu, NCHW layout, and ONNX Runtime graph optimization set to all. The current JavaScript dependency versions are ONNX Runtime Web 1.29.0 and Transformers.js 4.2.0. The source checkout's lockfile records the complete dependency versions.

Both the capability check and inference initialization request powerPreference: "high-performance". This asks the browser to prefer a discrete GPU, such as an NVIDIA RTX, on systems with integrated and discrete graphics. It is a preference; the browser and operating system retain the final selection. See the WebGPU adapter selection reference.

CLIP, U-Net, and TAESD sessions are created, run, and released sequentially to reduce peak GPU memory. Intermediate tensors are disposed when no longer needed. Model files can remain in browser memory, while GPU sessions are recreated on subsequent generations. This trades repeated initialization work for lower simultaneous GPU residency; the download size is not a VRAM requirement.

WebAssembly threading is set to one. The application does not require cross-origin isolation headers for WASM threads. Serve it over HTTPS or localhost so the browser can expose WebGPU; opening index.html directly as a local file is not the supported launch method. See ONNX Runtime's environment options.

The loader streams model downloads, checks file lengths, and stores assets in the browser's Cache API when available. A build-generated SHA-256 revision identifies the asset set and versions cache requests. Concurrent requests share the same pending model load; failed downloads can be retried, and truncated cached files are removed so retries can recover. Storage denial or quota errors do not prevent an uncached download.

Cache availability depends on browser policy, private browsing, available disk space, and eviction. Cached model files can avoid repeated network transfers, but they do not remove session initialization costs. No service worker is registered, and a fully offline page reload is not guaranteed.

Example prompts

Alpine lake (default)

vivid impressionist mountain landscape, crystalline turquoise lake surrounded by alpine meadows covered in blue purple yellow and red wildflowers, distant snow mountains softened by blue atmospheric haze, dramatic fluffy clouds catching warm sunlight, loose painterly brushwork, bright broken color, impasto texture, simplified forms, expansive peaceful composition

Sunlit garden

sunlit garden full of blooming flowers, stone path, small pond reflecting the sky, warm afternoon light, impressionist painting, oil on canvas, visible brushstrokes, soft broken color, luminous atmosphere, delicate painterly texture, elegant composition

Rainy Paris

rainy Paris street, pedestrians with umbrellas, reflections on wet pavement, glowing shop windows, soft evening light, impressionist painting, loose brushwork, atmospheric perspective, oil on canvas, muted colors, romantic mood, painterly texture

Wildflower meadow

woman walking through a meadow of wildflowers, flowing dress, summer breeze, bright sunlight, impressionist painting, oil on canvas, visible brushstrokes, luminous color, soft edges, natural light, poetic atmosphere

Flower bouquet

bouquet of wildflowers in a ceramic vase, scattered petals on a wooden table, warm window light, impressionist still life painting, oil on canvas, visible brushstrokes, rich painterly texture, beautiful color harmony

Hillside village

old European village on a hillside, stone cottages, narrow road, flowering trees, distant mountains, golden evening light, impressionist painting, oil on canvas, textured brushwork, warm colors, atmospheric perspective, timeless pastoral mood

Coastal sunset

quiet Mediterranean harbor at sunset, colorful fishing boats reflected in shimmering water, pastel houses along the shore, warm peach and golden sky, impressionist painting, oil on canvas, loose brushstrokes, bright broken color, rich painterly texture, peaceful composition

Autumn river

winding river through an autumn forest, golden birch trees and crimson leaves reflected in clear water, soft morning mist, sunlight filtering through branches, impressionist painting, oil on canvas, visible brushstrokes, luminous color, impasto texture, tranquil atmosphere

Performance and validation

NanoDiffuser generates an image in a single denoising step. The performance target is under 3 seconds on mid-tier consumer GPUs and capable mobile devices after model loading; this target has not been verified across devices.

The exporter includes ONNX structural checks, quantization checks, finite-output checks, and reference comparisons with minimum cosine similarity gates of 0.96 for CLIP embeddings and 0.90 for U-Net predictions. These are export acceptance thresholds, not measured benchmark scores or a general visual-quality rating.

Disclaimer

You're responsible for the content you generate. Do not use NanoDiffuser to create harmful or illegal content.

License

The converted weights retain their inherited OpenRAIL++ licensing requirements. Preserve applicable license and attribution notices when redistributing the files. Software dependencies retain their respective licenses.