“vctrstyle” LoRA

Teaching a model a visual direction, and proving it

Role: Data, training, evaluation
Tools: PyTorch (MPS)
Diffusers
PEFT
ComfyUI
Qwen-Image
CLIP
Status: Functional

Same prompt, same seed: base model, then LoRA v2 at steps 500, 1,000 and 1,500.

style gain, 95% CI [+0.053; +0.095]: +0.074
v1, measured: the starting point for v2: +0.003
images kept by automatic quality control: 45/48
of training, 1,500 steps on an M1 Pro: 88 min

The problem: Teaching Stable Diffusion a precise visual direction, locally, and proving the difference instead of claiming it.
For whom: The VECTEUR direction, a personal project: dark basalt rock, cold raking light, desaturated palette.
The outcome: A rank-16 LoRA trained in 2 h 53 on an M1 Pro, three exported checkpoints and a 32-image base-vs-LoRA harness.

Context

The target direction: dark basalt rock, cold raking light, desaturated palette, cinematic framing, trigger “vctrstyle”. Measurement drives the work: on v1 (31 images, 300 steps), the style gain stayed within noise (+0.003); v2 reworks data, training and evaluation, and reaches +0.074.

What I built

Planned dataset: 24 subjects × 2 compositions generated locally by Qwen-Image 2.1 through the ComfyUI API, 8 style references held out, manifest with hashes.
Automatic quality control: sharpness, exposure, dark share, perceptual-hash duplicates; 45 images kept, 3 rejected with their reason.
Rank-32 LoRA training: min-SNR, noise offset, precision benchmark before the run, time guard, exact resume, ComfyUI-compatible export.
Paired evaluation: 12 subjects including 6 out-of-domain × 2 seeds × 7 variants, style gain and text adherence with bootstrap 95% CIs, diversity, copy index.

Architecture 7 blocks

01 · InputVECTEUR prompts8 shots derived from the visual direction brief
02 · ModelDatasetSD 1.5 via the ComfyUI API, 8 shots × 5 seeds
03 · HumanCuration31 images kept by eye, 9 rejected and logged
04 · DataCachePre-computed VAE latents and CLIP embeddings
05 · ModelTrainingRank-16 LoRA, fp32 on MPS, 300 steps
06 · ToolEvaluation4 out-of-dataset prompts × 2 seeds × 4 variants
07 · OutputExport kohyaCheckpoints 75, 150 and 300, ComfyUI-compatible

Pipeline 4 steps

Generate48 Qwen-Image images, 8 references aside.
FilterAutomatic QC: 45 kept.
Train1,500 steps, 88 min, fp32 on MPS.
Prove168 images, paired comparisons.

Decisions the choice, the reason, the alternative ruled out

D1
Train in fp32 on the Apple GPU (MPS).
The bf16 attempt was twice as slow on this machine (69 s per step against about 30): measured, then rejected.
Ruled outbf16 autocast.
D2
Rank 16, alpha 16, on every UNet attention projection.
A trade-off between adapter size and capacity, suited to 31 images.
D3
Export in native kohya format.
Loaded directly by ComfyUI’s LoraLoader, with no third-party converter; loading is tested by the harness.
Ruled outA third-party converter.
D4
Evaluate on out-of-dataset prompts, same seeds, with scores and an eye check.
The CLIP score alone does not separate the checkpoints (+0.0015 at 150 steps): the sheets decide by eye.
Ruled outPicking the checkpoint on score alone.

Results

Evaluation on 24 pairs (same subject, same seed) — difference from the base model

Variant	Style gain [95% CI]	Δ text adherence	Diversity
Base	0	0	0.43
Style written in the prompt	+0.043 [+0.024; +0.063]	+0.004	0.37
LoRA v1 (step 150)	+0.003 [−0.010; +0.015]	+0.005	0.42
LoRA v2, step 500	+0.040 [+0.024; +0.056]	+0.011	0.40
LoRA v2, step 1,000	+0.062 [+0.042; +0.085]	+0.004	0.38
LoRA v2, step 1,500	+0.074 [+0.053; +0.095]	+0.007	0.36

CLIP ViT-B/32 score against the 8 held-out references. The prompt-only style also scores, but the sheets show it loses the subject (tiny owl, abstract cello); the LoRA keeps it.

Deliverables 4

Three verified LoRA exports (75, 150 and 300 steps, 128 modules each).
Curated dataset: 31 kept images and a dated list of rejects.
Base-vs-LoRA comparison sheets for 4 prompts.
Per-variant metrics: CLIP, distance to dataset, luminance, dark share.

Environment

Hardware and measurement setup

Trained on a MacBook Pro M1 Pro, fp32 on MPS: 1,500 steps in 88 min.
Dataset generated locally by Qwen-Image (Qwen Research licence, non-commercial).

Reproduce README commands

Private code: read access under NDA, or a 30-minute live demo. Request access →

PY=~/Documents/ComfyUI/.venv/bin/python
$PY train/train_lora.py --dossier donnees --out train/sortie --steps 300
$PY eval/comparer.py                # via ComfyUI (serveur sur 8188)
$PY eval/metriques.py               # CLIP + traits + résumé