Direct Store

Direct store is Weaver’s way of skipping a step that every Usenet downloader has always performed: writing the archive volumes to disk. For eligible jobs, the bytes inside the archive are written directly to the final extracted files as they download — the .rar volumes themselves never exist as files on your disks.

This page explains what direct store actually does, why it’s possible at all, what it saves, and how it relates to incremental extraction — a different mechanism it’s easy to confuse it with.

Why this is even possible

Most large Usenet releases are packed into RAR volumes for one historical reason: splitting content into evenly-sized, recoverable pieces that PAR2 can protect. Compression is usually beside the point — video and other already-compressed content doesn’t shrink — so the dominant posting convention stores the content inside the archive rather than compressing it.

A stored archive member is the original file’s bytes, laid out inside the volumes with archive framing around them. Which means: if a downloader understands the archive’s layout up front, the “extraction” step is revealed to be a copy — reading bytes out of volume files and writing the same bytes into the output file.

Direct store removes the copy. Weaver reads the archive framing as volumes download, learns exactly where each member’s bytes live, and routes those bytes as they arrive to their final position in the output file. The framing is understood and retained; the payload goes where it was always going to end up.

What it saves

Follow one large job’s payload through a traditional pipeline:

1. Write every archive volume to the temporary download area.
2. Read every volume back during extraction.
3. Write the extracted output.

That’s two full writes and one full read of the entire payload — before the completed files move anywhere. With direct store, the payload crosses your disk once: the single write of the output files.

In practice that buys:

• Faster job turnaround. The read-back-and-rewrite phase doesn’t exist, so there’s nothing to wait for after the download completes.
• Less temporary space. A job never occupies archive-copy and extracted-copy space at the same time.
• Less disk wear. For SSD-backed download areas, total bytes written per job drop by roughly half — meaningful at scale.
• Less I/O contention. Concurrent jobs aren’t competing with a full-speed re-read of a job that already finished downloading.

Fast server, slow storage

The savings above assume one disk. Direct store gets genuinely powerful in the setup many homelabs actually run: Weaver on a capable server, with completed downloads living on a NAS behind an NFS or SMB mount — a link that is often the slowest component in the whole chain.

A traditional downloader forces a choice between two staging layouts in that topology, and both have a real cost:

• Stage locally, then move. Volumes are written, verified, and extracted on the server’s fast local disks, and the finished output is copied across the link at the end. The payload crosses the slow link once — but as a bulk copy after the job is otherwise done, a dead-time tail that runs at NAS speed while the job sits in a “moving” state. And the server needs local capacity for the largest job you’ll ever run, twice over: archive copy plus extracted copy.
• Stage on the NAS directly. No local space needed — but now the volumes are written over the link, read back over the link for extraction, and the output is written over the link again. The payload crosses the slowest hop in the system three times, and the post-download phases all run at link speed.

Direct store dissolves the choice. The output files are assembled directly at their final destination, so the payload crosses the slow link exactly once — and that one crossing happens during the download, where the link only has to keep pace with your connection speed rather than sustain a separate bulk-copy phase afterward. Verification needs no read-back on a clean job, there’s no extraction pass, and there’s no move step, because the files were built where they belong. The server needs essentially no staging capacity, and the job is done when the download is done.

One honest caveat for this topology: when a job is damaged, repair reads the affected data back through the virtual-volume view — which, with output on the NAS, means reads over the link proportional to what repair needs. That’s the same class of cost a traditional client pays on damaged jobs in any layout; the difference is that direct store stops charging it on the healthy ones.

Verification without volumes

The natural objection: if the volumes never exist, what exactly gets verified?

Weaver’s block-level verification is the answer, and direct store is the reason that machinery earns its keep. Every PAR2 block gets its own verdict during download, derived from the same checksum pass that validates each article — so a direct-store job finishes with per-block proof of integrity covering data that was never written as archive files.

When deeper verification or repair is needed, Weaver reads the virtual volumes: it reconstructs any archive volume’s bytes on demand from the output files plus the retained framing, exactly as they would have appeared on disk. Damaged blocks are repaired in place through the same virtual view, and the repaired state is re-verified the same way. The job gets the full verify–repair safety net of the traditional pipeline — against files that were never materialized.

Direct store vs. incremental extraction

Weaver also performs incremental extraction: for conventionally-downloaded archives, extraction runs while the download is still in progress, member by member, as the needed volumes complete. SABnzbd and NZBGet users know the equivalent feature as Direct Unpack.

These two mechanisms solve different problems, and it’s worth being precise:

• Incremental extraction is about time. The volumes are still written to disk, still read back, still extracted — but the work overlaps the download instead of following it. Total disk traffic is unchanged; the wall-clock serialization is what disappears.
• Direct store is about work. The intermediate copy is eliminated entirely. There is no extraction step to overlap, because there’s nothing to extract — the output files are being assembled directly.

Both are active in Weaver, chosen per archive set automatically:

	Incremental extraction	Direct store
Archive volumes on disk	Written, then cleaned up	Never written
Payload disk crossings	Two writes + one read	One write
What it overlaps	Extraction runs during download	Output assembly is the download
Applies to	Any extractable archive set	Archive sets with stored (uncompressed) content
Verification and repair	Standard pipeline over volume files	Block verdicts + virtual volume reads

Eligibility and fallback

Direct store is on by default and decided automatically, per archive set. The common case qualifies: multi-volume RAR sets whose members are stored — the standard shape for large releases. A set with a few small compressed members alongside stored ones can still route direct, with those members extracted conventionally within a bounded allowance, and password-protected archives in the common stored shape can qualify as well.

When a set doesn’t fit — solid archives, compressed content, layouts that can’t be mapped confidently, or a filesystem that can’t support the sparse output files direct store relies on — that set simply falls back to the conventional path: volumes are written normally and incremental extraction takes over. The fallback is automatic, per set, and can happen mid-job without losing the download’s progress. From the outside, the job looks the same either way; the difference is how many times its bytes touched your disks.

The honest framing, consistent with the comparison page: direct store doesn’t make downloads safer than the traditional write-verify-extract pipeline — it makes the same safety guarantees while doing dramatically less I/O, and Weaver’s per-block verification is precisely what makes that trade sound.