F34.us Content Aggregator Prototype

Quarantine Before Trust

A deep dive into manifest evidence, total-snapshot deletion semantics, change-span gates, transactional quarantine, and crash-resilient audit trails.

A local file is close enough to feel trustworthy. It was exported from an application I chose, moved through storage I control, and never handed to a third-party warehouse. But locality says who owns the path, not whether the file is complete. An interrupted export, a malformed timestamp, or an accidental bulk edit can still turn a familiar CSV into a destructive instruction. I built TimeTrackDB's ingestion boundary around that distinction: source data becomes trusted current state only after the proposed transition has earned it.

Every arrival first receives a manifest record. TimeTrackDB records its source identity, export type, SHA-256 digest, byte size, row count, arrival time, and processing status before treating its contents as publishable. A previously seen file identifier can be skipped, while the digest and dimensions leave useful evidence about what actually landed. This is not the same as accepting the rows. The manifest is a receipt for an input, not a certificate that the input was good.

The parser then attempts to construct the complete candidate event set. Required fields, timestamp conversion, adapter mappings, and event transformations run before Bronze fingerprints or Silver rows are allowed to survive. A malformed required row quarantines the file at file scope; the system does not quietly keep the rows that happened to precede it. The physical source is moved away from the normal processed path. Its manifest retains the arrival and quarantined status, while the audit trail records the reason. That gives me something concrete to inspect and retry without letting a partial interpretation leak into current truth.

Complete exports introduce a subtler risk. Because they are snapshots, absence carries meaning: if a previously known day disappears from a later total export, that may be a real upstream deletion. TimeTrackDB detects that absence, assigns the day the digest of an empty set, and sends an empty replacement through the same publication path as any other change. Deletion is therefore explicit and historical rather than a stale row that survives forever. At the same time, a truncated export could resemble a huge set of legitimate deletions, so snapshot semantics need a second line of defense.

That defense measures the proposed change, not the age or raw size of the file. TimeTrackDB compares per-day digests and counts only the days whose truth would actually change, including days removed from a total snapshot. The default policy warns above seven changed days and quarantines above twelve, with both thresholds configurable. A large historical export containing mostly known data can pass normally; a small-looking file that would erase or rewrite a broad span cannot. This was an important design choice for local-first data: routine re-exporting stays cheap, but surprising scope requires review.

The quarantine decision is backed by transaction structure. The daemon writes the landing manifest, establishes a database savepoint, and lets Bronze change detection and Silver publication share the same cursor. If validation or the change-span gate refuses the candidate, it rolls back to that savepoint. Candidate fingerprints, digests, current events, history rows, and publish records disappear together, while the earlier manifest remains and changes status to quarantined. If Silver fails after Bronze has started, the surrounding transaction likewise prevents a half-published file. The source can be unsafe without making the warehouse ambiguous.

Operational evidence intentionally follows a different durability rule. The audit logger uses its own autocommit connection and records stage transitions, durations, changed and published day counts, anomalies, and failures as processing advances. Its failures are contained so observability cannot break ingestion, but its successful milestones survive a rollback in the data transaction. On startup, the daemon also identifies runs left active beyond the crash window and marks them as crashed. This separation lets an operator answer both sides of an incident: what data was committed, and how far did the failed attempt get?

The result is more than a folder named quarantine. It is a controlled promotion protocol for personal event history. TimeTrackDB preserves the original arrival, evaluates the semantic size of its effect, commits the accepted transition atomically, and keeps independent evidence when it declines. Local-first ownership matters because the data remains under my control. The larger lift comes from making that controlled data worthy of downstream questions.

F34.us Background Image