Retry, Backoff and Idempotency Without Double Spending

On Solana a retry is safe when it resends byte-identical signed bytes, because the network deduplicates by signature and the transaction can execute at most once. A retry is dangerous when it rebuilds, because a rebuild produces a new signature that can land alongside the original. Every correct retry design on this chain follows from that one distinction.

The corollary is that the retry policy is not really a policy about waiting. It is a policy about evidence: what the engine knows, what it merely believes, and what it is allowed to do while a result is still ambiguous. An engine that retries on a timer is making decisions without evidence, and it will eventually make an expensive one.

Resend is safe, rebuild is not

Resending means broadcasting the exact bytes you already signed, with the same blockhash, the same instruction list and the same fee settings. The signature is identical, so if the transaction already landed, the second broadcast changes nothing. This is the retry path an engine should take by default, and taking it requires that the signed bytes were persisted before the first submission.

Rebuilding means constructing a new transaction for the same intent, usually because the original's blockhash has expired or because you want a higher fee. It produces a different signature. If the original is still inside its window, the chain has no way to know the two are meant to be the same piece of work, and both can execute.

The rule that follows is short enough to put on a wall. Rebuild only when the original is provably dead, and the only proof that costs nothing is blockhash expiry. Any other justification for rebuilding, including that it has been a while and the operator is anxious, is a decision to accept duplicate risk.

What the blockhash window actually guarantees

Every transaction references a recent blockhash, and validators reject transactions whose blockhash is older than a recent window of roughly 150 blocks. This is a replay protection mechanism, and for an engine it is the single most useful guarantee available, because it converts an open question into a closed one.

Before expiry, the answer to whether a transaction can still land is yes, regardless of what your endpoint told you. After expiry, the answer is no, permanently, and it cannot change later. There is no ambiguity window after expiry and no scenario where a transaction resurfaces. That is what makes expiry a safe rebuild trigger and everything else a guess.

Two consequences follow for the sender. The deadline must travel with the artefact, so that any component deciding whether to keep trying reads the same value. And the deadline starts at build time, so a transaction that waited in a queue before its first submission has less of its life remaining than the nominal window suggests. Building late is a reliability decision, not a performance one.

Three outcomes, not two

The most consequential modelling error in these systems is a boolean success flag. A submitted transaction has three terminal states and they require different handling.

Unknown is rarer than the other two and it accounts for most of the real incidents. It occurs when an endpoint stops answering, when a confirmation subscription drops without the engine noticing, or when the process restarts holding a signature it never got an answer for. In every case the transaction may have landed, and any action taken on the assumption that it did not is a bet.

Resolving unknown is not hard, it is just work that has to exist. Query the chain for the signature, with a stricter commitment than the one used for progress display, and accept whatever it says. If the answer is still not available after a bounded number of attempts, the intent stays unknown and a human resolves it. What must never happen is an automatic reclassification of unknown into failed.

Classifying errors before retrying them

Not every error deserves a retry, and retrying the wrong class is how an engine turns a configuration bug into a sustained load problem. Classify at the point of failure into four buckets and attach the policy to the bucket rather than to the call site.

• Transport. Timeouts, connection resets, gateway errors. Retry the same bytes with backoff. These are the errors retries were invented for.
• Throttle. Rate limit refusals. Retry the same bytes, but the backoff must be coordinated across the whole sender rather than per call, and the scheduler needs to hear about it.
• Construction. Malformed transaction, oversized message, invalid account, unresolved lookup index. Never retry. The same bytes will fail identically and the fix is upstream in the builder.
• Execution. The transaction was processed and failed: slippage exceeded, insufficient funds, compute overrun. Never resend, because it already executed and charged you. A rebuild may be appropriate after the underlying condition is addressed.

The distinction between construction and execution errors matters for accounting as much as for retry logic. A construction error costs nothing and never touched the chain. An execution error cost the base fee and produced a real, failed transaction that belongs in the ledger with its signature. Collapsing them makes your failure metrics meaningless and hides the fee spend.

Backoff shapes compared

Full jitter is the right default for most engines: it is simple to implement, it never produces a synchronised wave, and its expected delay is half the exponential value so recovery is not needlessly slow. Decorrelated jitter is worth the extra complexity only when a single engine has many attempts failing simultaneously, which is a symptom worth fixing at the concurrency level anyway.

Whatever the shape, cap it against the blockhash deadline rather than against a number of attempts. A backoff sequence that would place the next attempt after expiry should not sleep at all; it should terminate the resend path and hand the intent to the rebuild decision. Sleeping past your own deadline is a common and entirely avoidable waste of a lease.

The idempotency key is the signature

In most distributed systems you invent an idempotency key and hope the downstream service honours it. Solana hands you one for free: the transaction signature is a deterministic function of the signed message, and the network enforces at-most-once execution per signature. That is a stronger guarantee than most payment APIs offer.

Using it properly means treating the signature as the primary key of the attempt record. Every log line, every metric label, every operator-facing row references the signature. When a duplicate is suspected, the question becomes concrete: are there two distinct signatures for one intent, and did both land? That is answerable from the ledger in seconds, whereas a system keyed by internal ids requires a reconstruction.

Intent  1 --- N  Transaction (signature, blockhash, valid_until, bytes)
Transaction 1 --- N  Attempt (at, endpoint, result, error_class)
Transaction 0 --- 1  Outcome (landed_slot, fee, amounts)

invariant: for any intent, at most one transaction has an Outcome
           where landed_slot is not null

violation  = duplicate execution
detection  = a query, not an investigation

Writing that invariant down as an actual check, run at the end of every run, is the difference between an engine that could have duplicated and an engine that can prove it did not. It costs one query. Its absence is why some operators genuinely do not know whether their engine has ever double-sent.

Crash recovery without a scan

A process that dies mid-run leaves transactions in flight. Recovery is only tractable if the ledger already contains, for every one of them, the signature and the deadline. With those two fields the recovery procedure is mechanical.

1. Load every transaction with no outcome recorded. This is the in-flight set, and it is bounded by the concurrency ceiling rather than by the size of the run.
2. For each, check whether its deadline has already passed. If it has and the chain has no record, it expired and the intent can be safely rebuilt.
3. For each still inside its window, query the signature. If it landed, record the outcome. If not, resend the stored bytes; this is safe precisely because they are the same bytes.
4. Anything that cannot be resolved after the deadline passes becomes unknown, and the account it came from stays quarantined until a human clears it.
5. Only after the in-flight set is empty does the scheduler resume releasing new intents.

One detail in that procedure is easy to get wrong. Step five says the scheduler waits until the in-flight set is empty, and the temptation during a restart is to resume releasing immediately so the run keeps its pace. Resist it. A restart is exactly the moment when the engine's picture of the world is least reliable, and adding new work while the old work is still being resolved means any duplicate that does surface will be tangled up with fresh transactions from the same accounts.

The alternative, for an engine that did not persist signatures before submitting, is scanning each account's transaction history and trying to match transactions to intents by amount and timestamp. That is slow, ambiguous when swap sizes repeat, and it is the reason the persist-before-send ordering in the pipeline is described as non-negotiable rather than as a good practice.

Auditing for duplicates you did not notice

Duplicates are quiet. Nothing errors, the run completes, and the totals are higher than planned in a way that is easy to attribute to slippage or to a miscount. The audit that catches them is a single grouping over the ledger: intents with more than one landed transaction.

Run it at the end of every run, not only when something feels wrong. If the count is zero, record that it was zero, because an audit that only runs during investigations gives you no baseline. If the count is not zero, the interesting question is which rebuild triggered it, and that is answerable because each transaction carries the reason it was built.

This is also the property to interrogate when comparing engines rather than building one. A tool that reports only successful swaps cannot tell you whether an intent produced one landed transaction or two, and the difference is real money. Asking a vendor how they detect duplicate execution is a sharper question than asking about throughput, and a professional Solana volume bot should be able to answer it with a description of evidence rather than a reassurance.

Retry policy checklist

• Does the retry path resend stored bytes, and is rebuilding a separate, explicitly triggered path?
• Is the only automatic rebuild trigger blockhash expiry, rather than elapsed time or operator impatience?
• Are outcomes modelled as landed, expired and unknown rather than as a boolean?
• Does an unknown outcome quarantine its account instead of feeding back into the retry loop?
• Are errors classified into transport, throttle, construction and execution before any retry decision?
• Is backoff exponential with jitter, and capped against the deadline rather than an attempt count?
• Is the transaction signature the primary key of every attempt record and log line?
• Does a duplicate audit run at the end of every run, and is a zero result recorded?

The last two are what turn this from a design into something you can demonstrate. An engine that keys everything on signatures and runs the duplicate audit can answer the only question that matters after a strange run, which is whether the engine did what the ledger says it did. Read the rate limit note next for the throttle half of the classification, and the observability note for how these outcomes become metrics.

Questions this note gets asked