The dataset-prep loop, and the friction it usually has
Anyone who has prepared an image dataset for fine-tuning a diffusion model or training a LoRA recognises the loop: a few hundred to a few thousand source images land in a folder. They have terrible filenames inherited from whatever camera or screenshot tool produced them. The training pipeline expects each image paired with a .txt caption next to it. The captions need to be edited in bulk to remove a tag that’s polluting the dataset, or to normalise a synonym. The images need to be resized to a uniform side length. And on the video side, a long source needs to be split into uniform-duration chunks, or a folder of short clips needs to be merged into one longer reference, or a video needs to be sampled to extract training frames.
Every one of those steps is solvable with a command-line invocation. The friction is not capability — it’s the cumulative cost of context-switching between four terminal windows, remembering whether the -ss flag goes before or after -i this time, and trying to keep three running operations in mental sync.
ArckheensToolzBasic doesn’t try to be a new abstraction over those operations — it bundles them in one app, with each tool in its own sub-tab, and each tool with a workflow-friendly set of defaults. The interesting engineering inside is the per-tool defaults. Eight tools, eight small but specific choices.
Choice 1 — Renamer: randomise order by default
The Renamer tool takes a folder of images (or arbitrary files when “Images only” is unchecked) and renames them all to a clean <Subject>[-<Concept>]-NNNN.ext pattern with configurable digit width and start number. The non-obvious default is the Randomize Order checkbox, which ships on.
The reason is one of those things only training experience reveals. A typical input folder is sorted lexicographically by the camera’s naming convention, which usually correlates with capture time. If the source photos were taken in a single session of indoor portraits followed by a second session of outdoor street photography, the lexicographic order roughly groups the indoor shots before the outdoor ones. When this folder is renamed to Subject-0001, Subject-0002, ..., the alphabetical order of the renamed files inherits the original grouping.
Several popular training configurations on Windows trainers sort the input by filename. If the dataloader doesn’t aggressively shuffle, the trainer sees “lots of indoor portraits, then lots of outdoor street” across the early epochs. This can encourage the model to learn a lexicographic-ish bias on top of the actual visual content.
Randomising at rename time is the fix. The renamed files are still Subject-0001 through Subject-9999, but the mapping from source to target is shuffled. Even with a deterministic dataloader, the first batches see a representative sample of the dataset. The user who knows they want a specific order can flip the checkbox off; the default protects the user who doesn’t.
Choice 2 — Captions: backup-before-overwrite, timestamped
The Captions tool creates a .txt file next to every image in a folder. Optionally, it pre-fills each file with a shared trigger-word string (typical for LoRA training datasets). If a .txt already exists at the target name, the default behaviour is not to overwrite — it’s to rename the existing file to <basename>.YYYYMMDDHHMMSS.txt first, preserving the old content. The new file is then written.
The timestamp is the part that matters. Multiple backups don’t collide — every run writes a different timestamp suffix — so the user can re-run the tool ten times and still see the full history of what was overwritten and when. It’s a one-line addition to the implementation; it costs a constant amount of disk space (much smaller than the images themselves); and it eliminates the “I just lost two hours of caption work” class of bug entirely. The same pattern applies to the Tag Editor, which uses .YYYYMMDDHHMMSS.bak for the same reason.
Choice 3 — Tag Editor: cleanup orphan separators after a delete
Captions for booru-style datasets look like 1girl, blue eyes, watermark, smile, simple background. When the user does a global delete to remove “watermark” from every caption in the dataset, the naive result is 1girl, blue eyes, , smile, simple background — the comma-space delimiters are now orphaned around an empty token.
Some downstream trainers handle this gracefully. Some interpret the empty token as a real (zero-length) tag and treat it as a degenerate concept that all images share, with mysterious effects on training dynamics. Either way, the result is ugly.
The Tag Editor’s Cleanup Separators checkbox (default on) runs a post-process regex after the search-and-replace pass:
- Sequences of consecutive commas (with optional whitespace) collapse to a single comma.
- Leading and trailing commas on each line are trimmed.
- Runs of whitespace collapse to a single space.
The output of the “delete watermark” operation becomes 1girl, blue eyes, smile, simple background — what the user actually wanted. The post-process is invisible (the user doesn’t see the intermediate broken state); the regex is simple; the result is consistent across the whole dataset.
Choice 4 — Prompts: collapse multi-line captions into single lines
The Prompts tool walks a folder (recursively if the user opts in) and concatenates every .txt into a single prompts_made.txt. The non-obvious step is the per-line whitespace normalisation: each source .txt is read fully, then any sequence of whitespace characters (including newlines, tabs, multiple spaces) is collapsed to a single space, producing one line per source file in the output.
Why: many of the consumers of prompts_made.txt (batch-generation scripts for stable diffusion front-ends, model benchmark harnesses) expect one prompt per line. If a source caption was written with line breaks for readability — a habit some captioners pick up — the naive concatenation produces a broken output file with one prompt spread across multiple lines. The single-line normalisation makes the output stable regardless of how the source captions were formatted. An optional prefix and suffix wrap each line for cases where the consumer wants a fixed pre-text like a quality tag.
Choice 5 — Resizer: PNG output only, by deliberate choice
The Resizer tool takes a folder of mixed-format images, lets the user select which ones to include with thumbnail checkboxes, and batch-resizes them to a configurable longest-side or shortest-side length. The output format is PNG only — there is no JPEG option in V1.
The reasoning is specific to dataset prep. Resizing in this context is almost always a downscale (training resolutions are smaller than source camera resolutions). Saving a downscaled image as JPEG introduces a second lossy quantisation on top of whatever the source already had — a phone JPEG re-saved as a JPEG accumulates noticeable artifacts, particularly in low-contrast areas where the trainer is most sensitive. PNG is lossless; the cost is larger files (typically 3-5× JPEG on photos), which is well worth it for a training dataset.
The scaling algorithm uses WPF’s BitmapScalingMode.Fant, which is the highest-quality WPF resampler — equivalent to a Lanczos-like filter on the output. The combination of lossless format and good resampler is what produces clean training inputs from a mixed-source folder. A future release will expose other output formats for users who specifically want JPEG; for V1 the no-choice default protects the dataset from accidental quality loss.
Choice 6 — Video Merge: stream-copy, no re-encode, with a documented constraint
Video Merge uses FFmpeg’s concat demuxer with -c copy — no re-encode, the streams are copied byte-for-byte from each input file into the output. The performance is dramatic: two gigabytes of source video merge in roughly four seconds on a modern SSD, vs several minutes if a re-encode happened.
The constraint is honest and documented in the disclaimer next to the tool: all input clips must have identical codec, resolution, frame rate, and audio sample rate. This is typically the case for outputs of a single generator or recordings from a single camera — and is the dataset-prep scenario the tool targets. If the user feeds mixed sources, FFmpeg either produces visible glitches at the seams or fails outright; the error is shown in the live log pane.
A “re-encode for normalisation” mode using FFmpeg’s concat filter is on the V1.1 backlog. For V1, the stream-copy mode is the right default for the dataset workflow: 95% of the use case is homogeneous sources, and the speed difference matters. The trade-off is explicit, the failure mode is observable, and the user can always re-encode separately upstream.
Choice 7 — Frame Extractor: four explicit modes, not a wizard
Extracting frames from a video has a small number of distinct strategies that map cleanly to four FFmpeg incantations:
- Keyframes only —
-vf "select=eq(pict_type,I)" -vsync vfr. The fastest option, samples the natural key-frame positions in the encoded stream. - All frames — no filter. Dumps every decoded frame.
- Interval —
-vf "fps=1/N". One frame every N seconds. - Custom — the user types their own
-vffilter chain.
The naive UI choice is a wizard: ask the user “what are you trying to do?” and infer the mode. The Frame Extractor doesn’t — it exposes the four modes as a radio-button group, with the Custom mode as an explicit escape hatch.
The reasoning is the audience. Dataset creators who need frame extraction already know the four concepts (key frame, fps interval, all frames). The wizard would slow them down with abstraction they don’t need. For the Custom mode, exposing the raw -vf chain lets a power user run something like select='gt(scene,0.4)' for scene-change detection without the tool having to anticipate every filter they might want. The output naming pattern is fixed (<base>_frame_%06d.<ext>) and the count of produced frames is logged at the end of the run.
Choice 8 — Static FfmpegService.IsInstalled with manual property-change
This one is more about how the app stitches state together than about dataset prep. The three video tools (Video Merge, Video Split, Frame Extractor) all need to know whether FFmpeg has been installed via the Setup tab. Each video ViewModel exposes a property:
public bool IsFfmpegInstalled => FfmpegService.IsInstalled;The right pattern in most contexts is a service event — FfmpegService.InstallationChanged — that the ViewModels subscribe to. With three ViewModels and one transition (off → on), that infrastructure is heavier than the problem requires. The actual implementation is simpler: MainWindow.xaml.cs intercepts the root TabControl.SelectionChanged event, and when the user enters the Tools tab, calls MainViewModel.RefreshFfmpegStatus(), which in turn calls OnPropertyChanged(nameof(IsFfmpegInstalled)) on each video ViewModel.
The result: a user who installs FFmpeg on the Setup tab and switches directly to Video Merge sees the “FFmpeg required” panel disappear immediately — without any cross-VM event plumbing. The cost is one explicit refresh call in one place. For a state transition that happens at most once per session, the explicit refresh is honest and small.
The four PowerShell ancestors
ArckheensToolzBasic isn’t a fresh design — it’s the consolidation of four PowerShell scripts that lived in the author’s personal toolbox for a couple of years before the consolidation:
ArckRenamerV01.ps1— the original rename pattern, with the randomise-by-default behaviour already in place.MakeTXTemptyFilesForImages.ps1— the captions skeleton creator.Prompts_maker.ps1— the prompt collector with whitespace normalisation.VideoToolkitv0.3.ps1— the FFmpeg wrappers for merge, split, frames.
The migration to a single .NET 8 WPF app brought asynchronous UI feedback, drag-and-drop, the timestamped backups, the WebView2 cross-app discovery tab, and the version pill that detects updates. The core logic of each tool is recognisably the same as in the PowerShell originals — when something works for years, the rewrite is about packaging and feedback, not algorithms.
Credits and license
- .NET 8 + WPF — Microsoft, MIT.
- FFmpeg — LGPL-2.1+ / GPL-2.0+, build packaged by the BtbN / FFmpeg-Builds project. Downloaded on first-run rather than bundled, to keep the redistribution profile of the installer clean.
- Microsoft.Web.WebView2 — Microsoft, distribution permitted. Used by the More Tools tab.
No model weights, no Python runtime, no scientific libraries. The toolkit is pure C# for the image / text tools and a thin Process.Start wrapper around FFmpeg for the video tools. Everything operates on files that already live on the user’s machine.
Take-aways
- Defaults matter much more than feature lists for tools that get used daily. A “random shuffle on rename” default that costs one checkbox saves the user from a class of training bias they don’t know about.
- Timestamped backups before every destructive operation are cheap insurance. Multiple backups don’t collide because of the timestamp suffix; the user gets full history for free.
- Post-process the obvious correctness problems (orphan separators after a tag delete) — the user shouldn’t have to clean up after the tool.
- When the four FFmpeg modes for frame extraction are the four right modes for the audience, exposing them as radio buttons beats hiding them behind a wizard.