Synapse

A local, GPU-constrained image intelligence system. Give it an image or a text prompt — it finds visually similar images from your collection and generates a new image from the semantic context. Everything runs offline on consumer hardware.

Target hardware: NVIDIA RTX 2050 (4 GB VRAM) · Python 3.14 · uv

---

How it works

Two pipeline modes depending on input:

Image mode (--image):

[Input Image]
      │
      ▼
  ┌─────────────────────────────────────────────────┐
  │  Stage 1 — Scry                                 │
  │  SmolVLM-500M-Instruct (1.04 GB VRAM)           │
  │  Image → Semantic Anchor (dense caption)        │
  └──────────────────────┬──────────────────────────┘
                         │  "A busy street market at dusk..."
                         ▼
  ┌─────────────────────────────────────────────────┐
  │  Stage 2 — Find                                 │
  │  CLIP ViT-B/32 (0.31 GB VRAM)                   │
  │  Anchor → 512-dim embedding → HNSW ANN search   │
  └──────────────────────┬──────────────────────────┘
                         ▼
  ┌─────────────────────────────────────────────────┐
  │  Stage 3 — Make                                 │
  │  Stable Diffusion Turbo (1.70 GB VRAM)          │
  │  Anchor → Generated image (1 denoising step)    │
  └──────────────────────┬──────────────────────────┘
                         ▼
               [outputs/make_<timestamp>.png]

Prompt mode (--prompt):

[Text Prompt]
      │
      ▼
  ┌─────────────────────────────────────────────────┐
  │  Stage 2 — Find                                 │
  │  CLIP ViT-B/32 (0.31 GB VRAM)                   │
  │  Prompt → 512-dim embedding → HNSW ANN search   │
  └──────────────────────┬──────────────────────────┘
                         ▼
  ┌─────────────────────────────────────────────────┐
  │  Stage 3 — Make                                 │
  │  Stable Diffusion Turbo (1.70 GB VRAM)          │
  │  Prompt → Generated image (1 denoising step)    │
  └──────────────────────┬──────────────────────────┘
                         ▼
               [outputs/make_<timestamp>.png]

Both modes can be combined: --image --prompt uses the image for retrieval context and the prompt to steer generation.

Pipeline latency on RTX 2050:

Mode	Time
--image	~12–14s (includes SmolVLM load + inference)
--prompt	~6–10s (Scryer skipped entirely)

Stages 1 and 2 are co-loaded (1.35 GB total), then unloaded before Stage 3 (1.70 GB) loads — fitting the full pipeline inside 4 GB VRAM.

---

Models

Stage	Model	VRAM	Notes
Scry	SmolVLM-500M-Instruct	1.04 GB	SigLIP vision encoder + SmolLM2 LM, bfloat16
Find	openai/clip-vit-base-patch32	0.31 GB	Shared 512-dim text+image embedding space, float16
Make	stabilityai/sd-turbo	1.70 GB	1-step adversarial distillation, guidance_scale=0, fp16

Models are stored locally and loaded fully offline (HF_HUB_OFFLINE=1).

---

File structure

synapse/
│
├── main.py                   # CLI entrypoint — subcommands: run, scry, find, make
│
├── brain/
│   ├── scry.py               # Stage 1: SmolVLM image captioning (Scryer class)
│   ├── find.py               # Stage 2: CLIP embedding + HNSW search (Finder class)
│   ├── make.py               # Stage 3: SD-turbo text-to-image (Maker class)
│   ├── index.py              # One-time HNSW index builder — run before first use
│   └── data/
│       ├── index.bin         # HNSW graph (binary, generated by index.py)
│       └── index.json        # int_id → image_path metadata (generated by index.py)
│
├── data/
│   └── raw_imgs/             # Your image library (indexed by index.py)
│
├── outputs/                  # Generated images (make_<timestamp>.png)
│
├── sd-turbo/                 # SD-turbo model weights (git-lfs clone, not in repo)
│
├── pyproject.toml            # uv project config + dependencies
└── uv.lock                   # Locked dependency tree

---

Setup

1. Install dependencies

uv sync

2. Download models

SmolVLM + CLIP — downloaded automatically on first run (cached to ~/.cache/huggingface/):

uv run python brain/scry.py brain/data/test_images/sample.jpg

SD-turbo — clone with git-lfs (fp16 weights only, ~1.7 GB):

# Install git-lfs if needed:
# Arch:   sudo pacman -S git-lfs
# Ubuntu: sudo apt install git-lfs

git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/stabilityai/sd-turbo
cd sd-turbo
git lfs pull --include="*.fp16.safetensors"
cd ..

3. Build the HNSW index

Point it at your image library. This encodes every image with CLIP once and saves the index to brain/data/. Run once; re-run only when your library changes.

uv run python brain/index.py --folder data/raw_imgs/

---

Usage

Full pipeline

# Image mode: scry → find → generate (~12–14s)
uv run python main.py run --image data/raw_imgs/000000101420.jpg

# Prompt mode: find → generate (~6–10s)
uv run python main.py run --prompt "a rainy street at night"

# Combined: image for retrieval context, prompt steers generation
uv run python main.py run --image photo.jpg --prompt "same scene but at dusk"

Shared options:

• --top-k N — number of similar images to retrieve (default: 5)
• --seed N — random seed for reproducible generation (default: random)
• --temperature 0.0–1.0 — generation variance (default: 0.0 = deterministic, 1.0 = max variation)

Individual stages

# Describe an image (Stage 1)
uv run python main.py scry --image path/to/photo.jpg

# Search your indexed library by text (Stage 2)
uv run python main.py find --query "a rainy street at night" --top-k 5

# Generate an image from a prompt (Stage 3)
uv run python main.py make --prompt "a busy street market at dusk" --seed 42 --temperature 0.3

Each brain module also runs standalone:

uv run python brain/scry.py photo.jpg
uv run python brain/scry.py photo.jpg --ask "What objects are on the table?"
uv run python brain/find.py --query "espresso machine" --folder data/raw_imgs/ --top-k 3
uv run python brain/find.py --query "a cat" --db        # fast HNSW path (requires index)
uv run python brain/make.py --prompt "neon Tokyo street at night" --seed 42

---

Architecture notes

VRAM budget

The RTX 2050 has 4 GB VRAM. All three models cannot be co-resident:

SmolVLM  1.04 GB  ┐
CLIP     0.31 GB  ┘  co-loaded in Stage 1+2 (1.35 GB total)
                     → unload both before Stage 3
SD-turbo 1.70 GB     Stage 3 alone

Each module's unload() method calls del model + torch.cuda.empty_cache() to release VRAM before the next stage loads. In prompt mode, SmolVLM is never loaded — CLIP alone runs at 0.31 GB, then hands off to SD-turbo.

Why SD-turbo uses guidance_scale=0.0

Classifier-Free Guidance (CFG) requires two UNet forward passes per denoising step — a conditional pass and an unconditional pass — and then extrapolates between them. SD-turbo was adversarially distilled to work in a single step with guidance baked in. Running CFG on top degrades quality. guidance_scale=0.0 means: use only the conditional prediction, no extrapolation.

How temperature works

Standard SD-turbo (temperature=0.0) is fully deterministic given a seed. Temperature injects Gaussian noise into the clean latent after denoising, before the VAE decoder:

noise_std = temperature × 0.5
latent = latent + randn_like(latent) × noise_std

This is safe because the denoised latent is in a well-conditioned range (~[-3, 3]) and the VAE decoder is robust to small perturbations. Injecting noise at the input of the UNet instead would push it outside its training distribution (trained at σ=14.6 only) and produce NaN/black images in fp16.

Scale mirrors LLM temperature: 0.0 = deterministic, 1.0 = maximum variation.

HNSW index

brain/index.py encodes your image library once with CLIP and stores the vectors in an HNSW (Hierarchical Navigable Small World) graph. At query time, find.py encodes only the query (~10ms) and searches the graph in ~1ms regardless of library size — far faster than re-encoding all images per query.

Parameters: space="cosine", dim=512, M=16, ef_construction=200, ef=50.

Offline mode

All three brain modules set HF_HUB_OFFLINE=1 before importing transformers/diffusers, which prevents any outbound network calls. Models must be present in ~/.cache/huggingface/ (SmolVLM, CLIP) and sd-turbo/ (SD-turbo) before running. The env var is set before the library import — setting it after has no effect as HuggingFace Hub reads it at import time.

---

Dependencies

transformers >= 5.2.0
diffusers >= 0.37.0
torch (CUDA)
hnswlib >= 0.8.0
accelerate >= 1.12.0
pillow >= 12.1.1
huggingface-hub[cli] >= 1.4.1

transformers 5.x API note: AutoModelForVision2Seq → AutoModelForImageTextToText, and from_pretrained uses dtype= instead of torch_dtype=. Both are already accounted for in the codebase.