Tag-forced captioning — pairing a frozen visual tagger with a free-form VLM for honest dataset descriptions

The captioning bottleneck in personal dataset prep

If you’re training a LoRA, fine-tuning a diffusion model, or hand-rolling a dataset for an embedding, the bottleneck very quickly stops being model size and starts being captions. A modern diffusion model trains best on flowing natural-language descriptions of what is in each image — not a flat list of booru-style tags, not a one-sentence summary. The model wants prose that names what the subject is wearing, what pose they’re in, what the camera angle is, what the lighting is doing, what the environment looks like.

Producing those captions at scale is hard for two practical reasons:

• Vision-language models drift. A free-form caption from a VLM can be perfectly fluent and miss two of the four most important details. On a single image that’s annoying; on two thousand images that’s a dataset that pulls the trained model in unpredictable directions.
• Many VLMs are deliberately tuned to refuse or to omit on certain content. For dataset preparation where the user is captioning their own images (often portraits, sometimes adult content for an explicit-by-design training run), a model that refuses or sanitises produces unusable output. The user’s only option is to caption by hand — days of repetitive work, which most users abandon halfway through.

The Uncensored Captioner is built around a workflow that addresses both problems at once. The framing is professional and the legal frame is explicit — users are captioning content for which they hold the appropriate rights, on their own machine, with no upload, no cloud, no telemetry.

What the “abliterated” variants are, and why they’re used here

The Qwen-VL family is a series of vision-language models published by Alibaba Cloud’s Qwen Team. The base models are excellent at visual description in general — they’re the upstream work that makes everything in this article possible. They are, however, instruction-tuned to refuse certain content categories.

The open-source community has published “abliterated” variants of these models — community-prepared checkpoints where the refusal direction in the model’s representation space has been surgically projected out, leaving the visual-description capability intact and removing the refusal behavior. The relevant authors and their permissive license terms:

• huihui-ai — the most-recognised abliterated checkpoint preparer in the open-source community. The variants for Qwen2.5-VL and Qwen3-VL used in this app are Apache 2.0.
• prithivMLmods — alternative abliterated variants, also Apache 2.0.

The installer offers five caption-model choices in the Run tab’s model loader (3 from huihui-ai, 1 from prithivMLmods, plus the prithivMLmods 7B Caption-it). All five are downloaded only at the user’s request, through the user’s own HuggingFace credentials — none of the weights are bundled in the installer ZIP. The user accepts each upstream license at download time. ExpSoft is an independent installer and is not affiliated with Alibaba Cloud, huihui-ai, prithivMLmods, or SmilingWolf.

The tag-forced workflow — the signature feature

The most interesting engineering inside the app is not the VLM inference (the upstream models do the heavy lifting). It’s the pipeline that ties two independent models together for a result neither produces alone.

Step 1 — WD14 generates booru-style tags

SmilingWolf’s WD SwinV2 Tagger v3 is a SwinV2-based image classifier trained on a large corpus of booru-style tagged images. Given an image, it emits a list of tags with confidence scores: 1girl, blue eyes, smile, simple background, looking at viewer, ... The tags fall into three categories — general (visual content), character (named characters when recognised), and rating (the overall safety level: general / sensitive / questionable / explicit).

Inference is fast — a single SwinV2 forward on a 448 px image, running through ONNX Runtime with the CUDA execution provider when available. On a consumer GPU it’s a few tens of milliseconds per image. The tagger’s output is filtered to keep only tags above a configurable confidence threshold (default 0.85), then optionally further filtered by category according to the user’s settings (include character tags, include rating tag).

Step 2 — tags are injected into the VLM prompt

The resulting tag list is injected into a prompt template that the VLM will receive along with the image. The default template:

Describe this image in detail for AI training. You MUST naturally
incorporate ALL of the following elements in your description:
{tags}. Write detailed, flowing sentences that include every listed
element. Do not simply list the tags.

The {tags} placeholder is replaced with the comma-separated list from WD14. The wording is deliberate — it instructs the VLM that the tags are the ground truth that must be covered, that the output must be prose rather than a list, and that the VLM has the latitude to add any context it sees that the tagger missed (the tagger is good at the visual-content axis but blind to ambience, mood, and global composition).

Step 3 — the VLM writes the description

The chosen Qwen-VL variant receives the image and the prompt-with-tags and produces a flowing description. The output is constrained by the prompt to cover every listed tag, so the model is much less likely to skip the visual elements the tagger identified. The output is freer than a flat tag list — the model fills in everything the tagger doesn’t know (lighting direction, atmospheric quality, subject expression, the implied context of the shot).

The result is a caption that combines the strengths of both models:

• From WD14: structured, high-recall coverage of the booru-vocabulary visual elements. The tagger almost never misses a clearly-visible element.
• From Qwen-VL: prose fluency, contextual interpretation, the implicit-and-atmospheric content that doesn’t appear in any tag vocabulary.

Step 4 — pre-tags, post-tags, multi-replace

Three post-processing affordances are applied to the VLM’s output before it’s written as the caption:

• Pre-tag — a fixed string prepended to every caption (typical: a trigger word for the LoRA being trained, like "Subject " or "1girl, ").
• Post-tag — a fixed string appended (typical: a quality tag).
• Multi-replace — a list of old::new substitutions applied across the caption, useful for normalising vocabulary across a dataset (e.g. 1girl::1woman).

The combination is dataset-wide consistency by construction: every caption starts with the same trigger, every caption ends with the same quality tag, and any vocabulary normalisation happens once at write-time rather than as a post-process pass later.

The supporting engineering

The workflow above is the “what.” The reason it runs on a 12 GB consumer GPU is the supporting engineering.

4-bit NF4 quantisation by default

Qwen3-VL-8B in fp16 is around 16 GB of VRAM — comfortably above the budget of the typical consumer card (8-12 GB still represents most of the installed base). In 4-bit NF4 via bitsandbytes the model footprint drops to roughly 5-6 GB. The quality difference for captioning, in side-by-side testing on a representative dataset, is in the “a few percent” range — visible if you look hard at fluency on rare vocabulary, indistinguishable on coverage and accuracy. The default in the UI is 4-bit on; a user with 24 GB+ of VRAM can flip it off for a small quality bump.

max_pixels = 512 × 28 × 28 — bounding the visual-token count

Qwen-VL processes images by partitioning them into 28×28-pixel patches. A 4K image generates well over ten thousand visual tokens — generation slows dramatically and the KV cache balloons past what a 12 GB card can hold. The processor exposes a max_pixels parameter that limits the effective resolution of the partitioning.

The app sets max_pixels = 512 * 28 * 28 = 401408 (≈ 633 × 633 effective resolution), with an additional Pillow downscale to 1024 px on the longest side before the image even reaches the processor. The combined budget caps the visual token count at roughly 512 tokens — enough for the VLM to “see” the image well, small enough to keep the KV cache and generation time bounded.

The quality cost is negligible. A VLM looking at an image for description is doing global understanding, not OCR or fine-detail recognition — a 633 × 633 effective resolution captures everything that matters for the captioning task on this generation of models.

Aggressive inter-image VRAM cleanup

Without explicit cleanup between batch iterations, the VRAM consumption climbs from one image to the next — PyTorch’s caching allocator holds onto intermediate tensors, the KV cache from the previous generation lingers, and after thirty or forty images the OOM lands. The captioning loop deletes every intermediate tensor explicitly after writing the caption (inputs, generated_ids, image_inputs, video_inputs, text, the message dict), then calls gc.collect() and torch.cuda.empty_cache().

The verbose memory snapshots are logged to logs/mem_debug.log before and after each step, which makes the very rare OOM-on-image-N case actionable — the user can see exactly which tensor wasn’t released and bring it up against an upstream release.

Skip-existing for batch resumability

A 2 000-image dataset takes several hours to caption even on fast hardware. The PC will go to sleep, the user will reboot for an update, something will crash, the run will be interrupted at image 1 500. The Skip Existing checkbox (default on) makes the batch idempotent: before captioning an image, the loop checks for a .txt file at the target name and skips that image if one is present. Re-launching the batch finishes the remaining 500 images without redoing the 1 500 that completed before the interruption.

WebSocket + REST hybrid backend

The Python backend is a FastAPI server running under uvicorn on a local port. Two transports between the C# UI and the backend:

• REST for the discrete actions — load_model, caption, batch_start, batch_status, batch_pause, batch_stop, system_info. Each is a stateless HTTP call.
• WebSocket at /ws/logs for the real-time log stream. The C# UI keeps a single connection open for the lifetime of the server and appends every message it receives to the fixed log pane.

The batch status is polled rather than pushed — GET /batch/{job_id}/status every two seconds. Pushing the status over WebSocket would have required a second persistent connection per session, with reconnect logic; polling is simpler, and a 2-second cadence is well within UI responsiveness for a batch that processes one image every 5-15 seconds. The trade is the right one for this scale.

Portability — same Pinokio-style isolation as the rest of the catalogue

As elsewhere in the ExpSoft catalogue, the entire install lives under the install folder:

• Miniconda + the venv in runtime/ and backend/venv/.
• Models in models/ (caption + tag models, downloaded with the user’s HF token).
• All caches (HuggingFace, PyTorch, pip) redirected under cache/.
• Logs under logs/.
• Config under config.json.

Nothing in %USERPROFILE%, %APPDATA%, or the registry. PathService.AutoPatchVenvPaths() runs at every Run-tab launch and rewrites any hardcoded paths in the venv if the install location has changed since last run. The user can zip the entire install folder, drop it on a different machine with a different drive letter, double-click, and the captioning workflow runs.

The three build variants and the GPU mismatch warning

The PyTorch wheel sets for cu124, cu128, and cpu are mutually exclusive. The installer ships in three flavours:

• Release-40xx — for RTX 20xx / 30xx / 40xx. torch==2.6.0+cu124.
• Release-50xx — for RTX 50xx Blackwell. torch==2.7.0+cu128.
• Release-CPU — captioning runs on CPU. Slower but functional for systems without a NVIDIA GPU.

The C# project file holds these as three Release configurations with DefineConstants; the Python-facing constants (TorchVersion, TorchCudaIndex, TorchIndexUrl) are #if-gated. At Setup step 3, the GPU detection step compares nvidia-smi’s reported card series against the build variant and shows a friendly warning if there’s a mismatch — “you have a 50xx-series card but downloaded the 40xx package; please use the 50xx package for full performance.”

Credits and license

• Qwen2.5-VL / Qwen3-VL — Alibaba Cloud (Qwen Team), Apache 2.0 / Qwen License. The vision-language model family that does the captioning. All credit for the visual-description capability belongs to the Qwen Team.
• huihui-ai abliterated variants — Apache 2.0. Community-prepared checkpoints used here for the captioning step.
• prithivMLmods abliterated variants — Apache 2.0. Alternative community-prepared checkpoints.
• WD SwinV2 Tagger v3 — SmilingWolf, Apache 2.0. The tagger that produces the booru-style ground truth used by the tag-forced workflow.
• PyTorch — Meta + community, BSD-3-Clause.
• Transformers + Accelerate + huggingface_hub — HuggingFace, Apache 2.0.
• bitsandbytes — MIT. 4-bit NF4 quantisation.
• ONNX Runtime — Microsoft, MIT. WD14 inference.
• FastAPI + uvicorn — Apache 2.0 / BSD. The local backend server.
• Pillow — HPND. Image pre-processing.
• Miniconda + Python — Anaconda Inc / PSF.
• Microsoft.Web.WebView2 — Microsoft, distribution permitted.

The C# / WPF EasyInstaller shell is original ExpSoft work. None of the model weights are bundled in the installer — every caption model and the tag model are downloaded with the user’s HuggingFace credentials at Setup time, which records the user’s acceptance of each upstream license. ExpSoft is an independent installer and is not affiliated with Alibaba Cloud, huihui-ai, prithivMLmods, SmilingWolf, or any other upstream author.

Take-aways

• Pair a frozen classifier with a free-form generator and constrain the generator to cover the classifier’s output. The result has the recall of the classifier and the fluency of the generator — better than either model alone, with one prompt template.
• 4-bit NF4 quantisation costs almost nothing in caption quality and is worth roughly a 3× VRAM saving — default it on, let the user with abundant VRAM flip it off.
• Cap visual tokens at the start of the pipeline. A VLM describing an image globally doesn’t need 10 000 visual tokens; 500 is enough and the speed gain is enormous.
• Make batch jobs idempotent by default. Skip-existing turns a multi-hour run from “restart from scratch on any interruption” into “restart and finish the remaining items.”
• WebSocket for logs, REST for actions, polling for status. The combination is simpler than pushing everything over a single WebSocket, and the polling cadence (2 seconds) is invisible at the per-image cost.