Scheduling and Pacing: Turning a Budget Into Releases

A scheduler converts a plan into timed releases. It receives an ordered list of intents and a pacing policy, and it decides for each one whether it may go now, must wait, or should be dropped because the window has closed. It never builds, never signs and never talks to an endpoint, and that restriction is what makes it testable without touching a network.

The reason this component deserves its own note is that pacing is where an engine's behaviour is most visible and most often wrong. Every other component can be correct while the scheduler quietly emits a burst at the top of each minute, and the symptoms will show up as rate limit errors in the sender, lease starvation in the fleet, and a landing rate that nobody can explain.

The scheduler has exactly one job

The job is to answer, repeatedly, whether the engine may start one more unit of work. Everything a scheduler contains exists to make that answer correct: a clock, a policy, a count of what is in flight, and a signal from downstream telling it whether the system is coping. Adding anything else to this component, particularly route selection or wallet choice, produces a class of bug where timing and execution are entangled and neither can be reasoned about alone.

Stated as a contract, the scheduler receives Intent[] and returns a stream of release events. It is allowed to reorder within the constraints each intent carries, to hold an intent past its earliest time, and to expire an intent that has passed its latest time. It is not allowed to change a size, a venue or a side, because those are the planner's decisions and quietly editing them makes the run untraceable.

Why a fixed interval is the wrong default

The first scheduler anyone writes sleeps for a fixed number of seconds between releases. It is the wrong default for three separate reasons, and they compound.

The first is queueing. A fixed interval assumes work completes in a fixed time, and it does not. When confirmations slow, a metronome keeps emitting at the same rate into a system that is draining more slowly, so the in-flight count climbs until something breaks. A scheduler that paces on completions rather than on a clock naturally slows down when the network does, which is the behaviour you want and the one a fixed interval cannot express.

The second is collision. Every process pacing on a round number tends to fire on the same boundaries, and shared infrastructure sees those boundaries as a spike. Your own components do the same thing to each other: if the confirmation poller also runs on a fixed tick, the two align and produce a request pattern with sharp peaks and idle troughs, which is the worst possible shape to present to a rate limiter that measures in windows.

The third is that a fixed interval carries no information. If the interval was chosen because it produced the right total, then any change to the plan silently changes the pace. Deriving the interval from the budget, the window and the observed completion rate keeps the relationship explicit and lets the engine adapt without a human editing a constant.

Three pacing algorithms compared

Three shapes cover almost every sensible scheduler, and the choice between them is a real trade-off rather than a matter of taste.

For a volume engine, a token bucket with a small cap is usually the right starting point. The accrual rate encodes the intended pace, the cap encodes how much burst you are willing to tolerate after a pause, and both are legible to an operator. A cap of one collapses it into a leaky bucket, which is a useful thing to know when someone asks for maximally smooth behaviour.

Jitter: what randomness actually buys

Jitter means adding a random offset to each release time rather than firing on an exact schedule. Its value is mechanical. Independent random offsets decorrelate your releases from each other and from any other periodic process sharing the same endpoint, which flattens the peaks a rate limiter reacts to. It also decorrelates retries, which is the same reason exponential backoff is almost always specified with jitter.

The distribution matters less than people assume, but it is not irrelevant. A uniform offset over a full interval is simple and effective. An exponential inter-arrival time produces a Poisson process, which is the mathematically natural shape for independent events and has the pleasant property that the average rate is exactly the parameter you set. The practical difference between them is small; the difference between either of them and no jitter at all is large.

What jitter does not buy is a change in your totals. A run with jitter moves the same value over the same window as a run without it, arriving in a less regular pattern. Any claim that randomised timing produces a different outcome on chain is a claim about somebody else's classification logic, and this desk does not make claims about systems it cannot inspect.

Session curves and shaping a window

Real activity is not flat, and a scheduler that supports only one rate for the whole window will produce a shape that no plan asked for. A session curve is a multiplier over time: the pacing rate at minute five differs from the rate at minute two hundred, according to a curve the planner supplies.

Three curves are worth having. A flat curve is the honest default and the easiest to reason about. A ramp curve starts slower and increases, which is useful when you want early releases to confirm before committing the fleet at full rate. A tapering curve does the opposite and is useful when a window has a hard end, because it leaves headroom to finish outstanding work instead of ending with a queue full of intents nobody will get to.

Implement the curve as data rather than code. A list of breakpoints with a multiplier at each one is enough, it can be logged with the run, and it can be replayed later when somebody asks why the middle hour looks different from the first. A curve expressed as a function inside the scheduler is a curve nobody can audit.

Sizing the programme before pacing it

Pacing is downstream of sizing, and the two get confused constantly. Sizing asks how much total activity the campaign should contain and over what window; pacing asks how to distribute it. Getting the pace right on a programme that was never sized properly produces a technically well-behaved engine executing an arbitrary plan.

The inputs to sizing are the total notional, the swap size distribution, the window length and the venues involved. From those you get the intent count, and from the intent count plus the observed confirmation time you get the concurrency the scheduler needs to sustain. If you are still deciding what total to aim at rather than how to pace it, the question of how much volume a token needs belongs before this note rather than after it, because a scheduler cannot compensate for a plan that was guessed.

One arithmetic check is worth running every time, and it is illustrative rather than prescriptive: divide the intent count by the window in seconds to get the mean release interval, then compare it against the observed confirmation time. If the interval is shorter than the confirmation time divided by the number of accounts that can sign independently, the plan is not schedulable with that fleet and no pacing algorithm will fix it.

Concurrency ceilings and the pause switch

The concurrency ceiling is the maximum number of intents the scheduler will allow to be in flight simultaneously. It is not a performance tuning parameter; it is a safety limit, and it should be derived from the fleet rather than chosen for throughput. A ceiling higher than the number of accounts available to sign converts into lease starvation, which surfaces as a confusing stall rather than as a clear limit.

The pause switch is the other half. It must stop new releases immediately, must not attempt to cancel work already on the network, and must leave the confirmation path running so outstanding items resolve. Pausing is the operator's only real lever during an incident, so its semantics need to be stated in one sentence that an operator can recall under pressure: no new work starts, existing work finishes, and the run reports what is still outstanding.

A pause that also stops the confirmation tracker is worse than no pause at all, because it converts a set of known-pending transactions into a set of unknowns. That is the exact state the fleet's quarantine logic is designed to avoid, and it means the pause has created the problem the operator pressed it to prevent.

The backpressure handshake

The sender knows things the scheduler does not: how many endpoint credits remain, how many submissions are being refused, and how long confirmations are taking. The scheduler owns the only lever that can act on that information. Connecting them is the handshake, and it should carry a level rather than an event.

sender -> scheduler:  pressure = ok | elevated | critical

ok        -> pace at policy rate
elevated  -> multiply rate by 0.5, log the reason
critical  -> hold all new releases, keep confirming

pressure derives from:
  queue_depth / ceiling
  refusal_rate over last window
  p90 confirmation time vs baseline

Sending a level rather than an error count matters because it puts the interpretation in the component that has the context. If the sender emits raw 429 counts, every consumer has to decide what a given count means, and those decisions drift apart. If it emits a level, the policy lives in one place and can be changed without touching the scheduler at all.

Note that queue depth appears in the derivation, not error count alone. Errors tell you something already went wrong; depth tells you the system is not keeping up while everything is still nominally succeeding. An engine that only reacts to errors is always reacting late, and the note on rate limits works through why depth is the better control signal in more detail.

Determinism and replay

A scheduler that cannot be replayed cannot be debugged. The requirement is not that the same run repeats identically, since jitter is deliberately random, but that a recorded run can be re-examined: the plan, the policy, the curve, the seed for the randomness, and the actual release timestamps should all be in the ledger.

Persisting the random seed is the detail most often skipped and the one that makes the difference. With the seed and the policy you can reconstruct exactly what the scheduler intended, which lets you separate a pacing problem from a network problem when the two look identical after the fact. Without it, every post-mortem includes an unresolvable argument about whether the engine sent when it meant to.

The same recording discipline is what makes an engine comparable to anything else. If you are evaluating a hosted volume bot for Solana against a build of your own, the question to ask is not how fast either can send, but whether either can hand you a per-release record you could reconstruct a run from. That is a much harder property than throughput and a much more useful one.

Scheduler review checklist

• Is the release rate derived from the plan and the window, or is it a constant somebody typed?
• Does the scheduler pace on completions as well as on a clock, so it slows when the network does?
• Is there jitter on every release, and is its seed recorded with the run?
• Is the concurrency ceiling derived from fleet capacity rather than chosen for throughput?
• Does pause stop releases without touching in-flight work or the confirmation path?
• Does the scheduler receive a pressure level from the sender rather than raw error counts?
• Is the session curve stored as data alongside the run, so it can be replayed?
• Can the scheduler resume after a crash from persisted state, at the right position?

Most of these are cheap to add at design time and expensive to retrofit, because they change what has to be persisted before an action rather than after it. Pair this note with the fleet design, since concurrency ceilings and fleet size are two views of the same constraint, and with the rate limit note for the downstream half of the handshake.

Questions this note gets asked