Same order, different state: what broke between consensus and execution in a trading-chain prototype
Earlier this year I built 0x26 Chain, a prototype layer-one ledger for trading. It ordered transactions with a DAG protocol derived from Mahi-Mahi and executed them in a native order-book engine. It ran as a controlled testnet with three validators. It was an engineering spike, and it is no longer running. This post is about the most useful thing the prototype taught me. Four validators agreed on exactly the same sequence of transactions and still ended up with different state. Consensus was fine. The bug was in the gap between consensus and execution, and I think anyone putting a stateful engine behind DAG ordering can fall into the same gap. The incident DAG blocks in this system are small, so executing them one at a time was slow. Every execution step pays for a checkpoint, a state root, a header and an install. The fix was to merge eight DAG blocks into one execution group and pay those costs once. That left a tail. If five decided blocks were sitting in the queue and no more...