The dead-letter pattern: never corrupting the client's view

Every pipeline that ingests outside data eventually meets a record it can't handle. A field that should be a date arrives as the string N/A. An identifier points to something not in the reference set. The interesting question is not whether this happens, because it always happens. It is what the pipeline does when it does.

The two bad options

Drop it. Skip the row, keep going. The job finishes green and the table looks clean. But every dropped record silently lowers a total somewhere, and because nothing was logged, there is no way to know how much is missing or why.

Force it in. Coerce the bad value into something that fits. Now the corruption is inside the table people trust, wearing the same format as the good data. It does not look like an error, so nobody investigates until a decision has already been made on it.

The pattern: set it aside

Messaging systems solved a version of this with the dead-letter queue. A message that cannot be processed is not discarded and is not left to jam the pipe. It is moved to a separate place for failed messages, where it can be inspected and retried. The same idea applies cleanly to data pipelines: a record that fails validation goes to a quarantine table that holds the raw row, the rule it failed, and a timestamp. The main pipeline continues with the records that passed.

The rule we hold to: a record we can't validate never silently changes the client's numbers. It waits where we can see it.

None of this is clever. It is the difference between a pipeline that fails loudly and recoverably and one that fails quietly and permanently. When the data feeds a real decision, that difference is the entire job.