Tile-based video super-resolution on Tensor Cores — bringing NVIDIA’s VFX SDK to a Windows desktop workflow

The use case — and the gap that this fills

If you’ve worked with image or video material for any length of time, the upscaling problem is familiar. You have:

• A library of old photos at 1024 or 2048 pixels, taken a decade ago, that you’d like to see at 2K or 4K for printing or wallpapers.
• A folder of 720p phone footage that you’d like to remaster at 1440p or 4K for sharing, archiving, or as training data.
• A low-resolution training dataset that needs to be upscaled in bulk before fine-tuning a high-resolution image model.

The open-source super-resolution ecosystem (the model family that started with ESRGAN and has descendants like Real-ESRGAN, SwinIR, and the various Diffusion-based upscalers) produces excellent results on textures and stylised content. The practical problem on a personal Windows workstation is that those models are typically designed to load the full input image into VRAM in one go. For a 12 GB consumer GPU, that caps the practical input resolution around 2K-4K — beyond that, the user has to manually tile the input, run the model on each tile, and stitch the results, which is exactly the work the model should be doing.

NVIDIA shipped a Video Effects SDK in 2025 as an official pip package — nvidia-vfx. The same RTX Video Super-Resolution models the user might be familiar with from the GeForce driver’s Chrome / Edge integration are exposed through this SDK, but with two key differences: they’re callable from Python (not stuck behind a browser-playing-a-video gate), and they use a tile-based inference path internally. The tile-based approach is what makes the SDK well-suited to consumer GPUs.

ExpSoft RTX VSR is the personal-use wrapper that turns the SDK into a desktop workflow: install the wheel and its dependencies, drag a file in, pick a quality, click Run. The technical content of the rest of this article is the engineering around that core idea.

What “tile-based” actually means here

An open-source super-resolution model in its naive form takes a tensor of shape [1, 3, H, W] (single image, three channels, height, width) and produces a tensor of shape [1, 3, H*scale, W*scale]. The intermediate activations needed to compute that output scale with H × W, and on a model with deep convolutional layers the peak VRAM footprint is several times the size of the activation tensors. For a 4K input (3840 × 2160) at 4× upscale to 16K, the intermediate footprint can reach tens of gigabytes — well past any consumer card.

Tile-based inference partitions the input into overlapping rectangular tiles (typically 256 × 256 or 512 × 512 pixels), runs the model on each tile independently, then stitches the outputs together with feathered blending at the overlaps. The VRAM footprint per tile is small and bounded — proportional to the tile size, not the input size — so any input resolution becomes manageable. The Tensor Cores on RTX GPUs are well-suited to the small, dense matrix multiplications that dominate the per-tile inference, which is what produces the high throughput: 30-100 fps on 1080p-to-2K on an RTX 4070+, and the equivalent on Blackwell-class cards.

The trade-off is that the model is closed-source: the weights ship inside the wheel and aren’t available for inspection or fine-tuning. For a tool whose user wants the quality NVIDIA’s research team has shipped, with one pip install and no per-image manual tuning, that’s the right side of the trade.

The build matrix — 40xx and 50xx, two separate installers

PyTorch wheels are CUDA-version-specific. The RTX 50xx (Blackwell) generation requires PyTorch 2.7+ on the cu128 wheel index; PyTorch 2.6 on cu124 covers RTX 20xx / 30xx / 40xx but explicitly doesn’t support Blackwell. The wheels are mutually exclusive — switching after install requires pip uninstall followed by pip install.

The installer ships in two flavours:

• Release-40xx — Turing, Ampere, Ada (RTX 20xx / 30xx / 40xx). PyTorch 2.6.0 + torchvision 0.21.0 with CUDA 12.4 wheels.
• Release-50xx — Blackwell (RTX 50xx). PyTorch 2.7.0 + torchvision 0.22.0 with CUDA 12.8 wheels.

The split lives in the C# project file as two Release configurations with a single #if GPU_50XX guard in SetupService.cs that swaps the pinned PyTorch versions and the wheel index URL. Setup step 2 runs nvidia-smi to detect the installed GPU series and surfaces an explicit error if the user picked the wrong package (“you have a 50xx-series card but downloaded the 40xx package; please use the 50xx package”). Catching the mismatch up-front is dramatically better than an obscure CUDA error two hours into the install.

PyAV streaming — no intermediate frame PNGs

The first cut of the video pipeline was the obvious one: extract every frame to work/<video-name>/frames/<n>.png, upscale each PNG with nvidia-vfx, then re-encode the upscaled PNGs into a video file with FFmpeg. The structure is simple and easy to reason about — you can resume a failed run, you can inspect individual frames, the debugging story is clear.

The performance is terrible. A 1080p 60-second clip at 30 fps is 1 800 frames; each PNG is around 3 MB; that’s 5+ GB of disk traffic per minute of video. On a typical SSD, the disk I/O dominates the total time — the GPU is mostly waiting for the next frame to read in or for the previous frame to flush to disk. Worse, the disk fills up surprisingly fast on multi-minute clips.

The right approach is streaming with PyAV (Python bindings for FFmpeg, BSD-3-Clause). The pipeline becomes:

1. Open the source video with PyAV in decode mode.
2. For each frame: decode → convert to a CUDA tensor → run nvvfx.VideoSuperRes.run() → produce a CUDA tensor of the upscaled output → copy to a numpy array → wrap in av.VideoFrame → write to the encoder.
3. Mux the encoded video stream into the output container.

No frames touch disk. The GPU is occupied continuously. A 60-second 1080p clip that previously took 8 minutes (mostly disk-bound) now takes about 90 seconds (compute-bound). The cost is that mid-video resume isn’t possible — if the user cancels at frame 1500/3600, the partial output isn’t a usable video. For V1, the simplification is worth it; the chunking infrastructure (FfmpegService.SplitVideoAsync) is still in the codebase, ready for a future “very long video” mode that combines the speed of streaming with the resumability of chunking.

Audio extract-and-remux via FFmpeg

The video super-resolution pipeline only sees the visual stream — the audio is invisible to nvidia-vfx. The naive approach would be to have PyAV pass the audio packets through unchanged, copy them into the output. PyAV can technically do this, but its audio handling is fragile on a few specific container/codec combinations (.mov with H.264 video and AAC audio is the canonical case).

The reliable path is two extra FFmpeg calls bracketing the upscale:

Step 1/3 — extract audio:
  ffmpeg -i input.mp4 -vn -c:a copy audio.m4a

Step 2/3 — upscale video (no audio):
  python rtx_vsr.py --input input.mp4 --output upscaled_noaudio.mp4 --quality HIGH --scale 2

Step 3/3 — mux:
  ffmpeg -i upscaled_noaudio.mp4 -i audio.m4a -c:v copy -c:a aac -shortest -y final.mp4

Two FFmpeg invocations, both cheap (the extract is stream-copy, the mux is stream-copy on the video side and re-encodes audio only to ensure compatibility), with the heavy work in the middle. Sync is perfect because the audio file is the original sample-accurate version. Zero audio quality is lost. If the source has no audio track, the extract step is skipped and the upscaled-no-audio file is renamed directly.

NVENC priority, libx264 fallback

The output encoder pick is in two tiers:

• NVENC (h264_nvenc) with preset p4 (balanced quality/speed) and -rc:v constqp -qp 18 (very high quality, close to visually lossless). This is what the RTX user expects — the same hardware encoder used by streaming software and game-capture tools, which produces excellent quality at minimal extra time after the upscale.
• libx264 with -crf 10 -preset fast (very high quality, visually lossless, CPU-encoded) — the fallback for the rare case where NVENC isn’t available on the system (older driver, NVENC quota exhausted, non-NVIDIA GPU through Optimus, etc.).

Both paths use -pix_fmt yuv420p for output, which is what every consumer player expects — smartphones, browsers, sharing platforms. The encoder selection happens at the start of the upscale: if NVENC initialisation succeeds, NVENC; otherwise libx264. The user never sees the choice; the log records which path was taken.

Output dimensions rounded to multiples of 8

NVIDIA’s VFX SDK requires output width and height to be multiples of 8 — a Tensor-Core alignment constraint. A user dragging a crop region or asking for a 2× upscale of an odd-resolution source can easily produce output dimensions that don’t respect that constraint.

The fix is a defensive round at the moment the dimensions are computed:

output_w = (input_w * scale // 8) * 8
output_h = (input_h * scale // 8) * 8

The rounding is always down — at most 7 pixels lost on each axis. On a 4K output, 7 pixels is invisible (less than 0.2% of the dimension). The alternative — rounding up — would produce dimensions slightly larger than the user asked for, which is less ergonomic for downstream uses (a request for “1920” returning “1928” is confusing). The down-rounding produces 1920 from any input that intends 1920 ± 7, which is the right behaviour.

The Before/After slider — 150 lines of WPF

The result-comparison surface is a custom WPF control: two Image overlays stacked, with a vertical drag handle in the middle, and a RectangleGeometry clip on the “After” layer that tracks the handle position. The user grabs the handle, drags left-right, and watches the upscaled output reveal itself over the original. It’s about 150 lines of C# total (the drag handler, the clip-rect update, the layout) plus a small XAML UserControl.

The reason it exists as a custom control rather than a NuGet dependency is that none of the available WPF before/after sliders quite hit the right defaults — the handle styling, the drag responsiveness, the way the layered images snap to the same dimensions even when the upscaled output is bigger than the original. Writing it once is faster than configuring around someone else’s defaults, and it’s a control the rest of the ExpSoft catalogue can re-use (it shows up in DiffusionForce and others too).

The visual delta on a 2× ULTRA upscale of a decade-old photograph is striking — the slider is the moment most users decide whether the tool is worth keeping in their workflow.

The RAM watchdog — the long-video scenario

A 4K-output upscale of a multi-minute clip can transiently consume 8-16 GB of system RAM (PyAV decode buffers, the upscaled-frame intermediate, the encoder input queue). On a 16 GB-of-RAM machine with a browser and an IDE already open, that pushes Windows into the swap-thrash zone — mouse lag, UI freeze, eventual OOM.

The watchdog is the same pattern used elsewhere in the ExpSoft catalogue: a background task that polls Win32 GlobalMemoryStatusEx every three seconds. If available physical RAM drops below 2 GB, the watchdog signals the cancellation token, which gracefully terminates the Python sub-process. The user sees a clear warning in the log — “Available RAM critically low (XXX MB). Cancelling…” — rather than a silent crash or a system-wide freeze. Closing other applications and re-running typically completes successfully.

Source quality matters — the explicit user tip

A small piece of UX worth highlighting: the Run tab includes a fixed card titled “Source Quality Matters” explaining that super-resolution enhances detail but doesn’t restore artefacts of compression. A JPEG that’s been through three rounds of social-media re-encoding doesn’t become a clean 4K just because it’s been upscaled — the upscaler faithfully enlarges the JPEG’s block artefacts along with the actual image content.

The right input for the best output is the highest-quality source available: PNG or TIFF for images; CRF<18 or ProRes for video. The card surfaces this expectation upfront so users don’t conclude the tool is broken when they feed it a heavily compressed source and get heavily upscaled compression back. One paragraph of text, a lot of avoided frustration.

Portable install — drive-letter resilience

The install footprint is fully self-contained next to the exe: Miniconda under runtime/conda/, the venv under runtime/venv/, portable FFmpeg under runtime/ffmpeg/, the Python inference script under project/, output and work folders, logs and config — nothing in %USERPROFILE%, %APPDATA%, or the registry. The .condarc redirects every conda location to the local install folder; the venv’s caches (HuggingFace, PyTorch, pip) are redirected via environment variables at sub-process launch.

At every Run, PathService.AutoPatchVenvPaths() checks whether the install folder’s drive letter has changed since the venv was created. If yes, the helper rewrites every absolute path inside the venv’s text files (pyvenv.cfg, activate.bat, .pth entries, conda meta JSONs) to match the new location. The patch is idempotent — running it on an already-correct venv is a no-op. End result: zip the install folder, drop it on another Windows machine with a different drive letter, double-click, and the upscaling workflow runs.

Credits and license

• NVIDIA Video Effects SDK (the nvidia-vfx pip package) — NVIDIA Corporation, proprietary SDK. Downloaded directly from NVIDIA’s official pip servers at Setup time. The user accepts NVIDIA’s SDK terms when the wheel installs. ExpSoft never bundles the SDK in the installer ZIP.
• Comfy-Org/Nvidia_RTX_Nodes_ComfyUI — Apache 2.0. Referenced for the SDK API call patterns; the implementation in this app is original ExpSoft work.
• PyTorch — Meta + community, BSD-3-Clause.
• PyAV — BSD-3-Clause. FFmpeg bindings for Python, used for the streaming video pipeline.
• Pillow — HPND. Image I/O for the single-image and batch-image modes.
• FFmpeg — LGPL-2.1+ / GPL-2.0+ (LGPL-shared Windows build from gyan.dev). Audio extract/remux only — encoding goes through PyAV.
• Miniconda — Anaconda Inc, BSD-3-Clause.
• Microsoft.Web.WebView2 — Microsoft, distribution permitted.

The C# / WPF / Python code of the wrapper is original ExpSoft work. ExpSoft is an independent installer and is not affiliated with NVIDIA Corporation. No model weights are bundled in the installer — the SDK’s models live inside the nvidia-vfx pip wheel and are downloaded from NVIDIA’s servers when the user runs Setup.

Take-aways

• Tile-based super-resolution beats whole-image super-resolution on consumer GPUs. The VRAM footprint is bounded by tile size, not input size, which makes 4K-to-8K practical on 12 GB cards.
• For video pipelines on personal hardware, prefer streaming over disk-staged intermediates. The disk I/O cost of staging frames typically dwarfs the compute cost of the upscale itself.
• Extract-and-remux audio via FFmpeg is more robust than passing audio packets through a Python video pipeline. Two cheap FFmpeg calls bracket the heavy compute, sync is perfect.
• NVENC for output encoding is what RTX users expect on the encoded output, with libx264 as a clean fallback for the rare case it’s unavailable. Yuv420p output plays everywhere.
• Round output dimensions to the alignment your SDK requires. Mis-aligned dimensions produce silent CUDA errors that are hard to debug; a defensive round is two lines and prevents the entire class.