M6: BFT consensus in onxd (ADR-0049 locking) — NO-GO: 1 critical + 112 pts, fix design ADR-0052 proposed #3
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "m6/bft-consensus"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Carries the old
pr59branch (M6: wireonx-consensusintoonxd, BFT locking per ADR-0049) onto the Forgejomain, and adds tests for the last review round.Consensus impact
This PR introduces BFT consensus (proposal → PreVote → PreCommit → Commit, with Tendermint-style locking) into the node. Nodes with this PR and nodes without it do not interoperate.
Could this behave differently on two honest nodes given the same inputs? Yes. Gossip order and timing decide which votes a node has seen before its round timeout, and so which block (if any) it locks on. That's inherent to BFT, but finding 1 below shows that today it can leave honest nodes locked on different blocks permanently.
Specification and decision record
docs/specification/section:docs/specification/consensus.md(updated in this PR).3f2b7e5, before the engine change.)set_head).Checks
Run locally (rustc 1.98.1, pinned toolchain), since the server has no CI. Scoped to the two crates this PR changes (
-p onx-consensus -p onxd), not--workspace.cargo fmt --all -- --check: cleancargo clippy -p onx-consensus -p onxd --all-targets -- -D warnings: cleancargo build -p onx-consensus -p onxd --all-targets: okcargo test -p onx-consensus -p onxd: all passcargo test ... -- --ignored: all 3 finding tests FAIL, as expected (see below)python3 scripts/site.py check,check-doc-links.py,check-spec-citations.py,version.py check: pass04f0558(ADR-0052, docs only): the four doc checks pass. The Rust checks were not re-run because04f0558changes no code; the results above are fromc1867bd.Test-environment notes:
producer_loop::daemon_binary_refuses_to_start_without_keyhardcodes<workspace>/target/debug/onxd. With a sharedCARGO_TARGET_DIRit fails to spawn (NotFound). With the target dir in place it passes. This is a latent fragility in the test, not a product bug.Notes
Outstanding findings (not fixed, owner decides)
Scored with CONTRIBUTING.md's point system. Severity follows
docs/claude-review-prompt.md("critical: consensus break, chain halt, …"). Findings 1–3 are each proven by an#[ignore]d test that fails today. Run them withcargo test -p onx-consensus -p onxd -- --ignored. Findings 4–7 were found while designing the fix (ADR-0052 Context). Each was checked against the code at04f0558, but none has a failing test yet. Under the proof gate the two high ones should be proven before they block, so the score is given both with and without them.observed_pol_for_other_block_unlocksforged_future_round_votes_are_not_bufferedstale_buffered_vote_does_not_poison_valid_vote04f0558, no test yetblocksis never pruned04f0558, no test yetEQUIVOCATION evidence04f0558, no test yetconsensus.md§3.7 disagrees with the engine: it locks on a PreVote quorum (engine: PreCommit) and unlocks on a proposal-carried QC (engine: local POL)engine.rsScore (
04f0558):1. POL unlock path is unreachable (critical, chain halt).
2b87615replaced the leader-claimedqc_roundunlock with a local Proof-of-Lock: a locked validator unlocks only if it has itself seen 2/3+ PreVotes for the new block at a higher round (pol[B] > locked_round). Butpolis only written inadvance_after_quorum(PreVote), for the current proposal. A validator locked on A rejects every proposal for B (ConflictingProposal), so it has no current proposal for B.receive_votethen rejects every PreVote for B (ConflictingProposal, since votes must match the current proposal).pol[B]can never be written, and the unlock branch is dead code. If validators end up locked on different blocks (for example, a Byzantine node delivers PreCommits selectively, so some honest nodes reach the lock quorum for A in round r and others for B in round r'), neither side can ever unlock. The height stalls forever.Test result:
panicked at engine_integration.rs:245: locked validator never unlocks despite observing a POL for B at a higher round.Possible direction (not implemented): count PreVotes for any block per round (as Tendermint does), independent of whether the local node accepted that round's proposal, and record
polfrom those counts.2. Unbounded buffering of unauthenticated future-round votes (high, memory DoS). The engine returns
InvalidRoundbefore it verifies the signature. The driver (69f1ae1, "buffer future-round votes") pushes every such vote intopendingwith no bound and no authentication. Any peer can grow node memory without limit.Test result: 1000 forged votes (claimed validator 0, signed with validator 3's key), all buffered:
1000 forged votes buffered.Possible direction: verify the signature (and validator membership) before buffering, and cap the buffer per validator and per round.
3. A stale buffered vote poisons a valid vote (medium, liveness).
2b87615keepspendingacross view changes. A round-0 vote buffered before the view change is stale in round 1. The retry loop inreceive_votepropagates its error with?. So a valid round-1 vote that the engine already counted returnsErr, the PreCommit it triggered is never broadcast (the driver has already marked the phase as voted), and the rest of the drained buffer is dropped.Test result:
a valid vote must not fail because of a stale buffered vote: "consensus: vote rejected: InvalidRound".Possible direction: drop stale-round entries on view change, and handle retry errors per vote instead of with
?.4. More unauthenticated, unbounded buffer paths (high, memory DoS). Fixing finding 2 alone would leave two more ways in. (a) The engine checks that a vote matches the current proposal before it checks the signature, and the driver buffers every
ConflictingProposalvote while no proposal is held (consensus_driver.rs, "Vote arrived before the proposal"). (b) A Commit vote whose block has not arrived is buffered on theblocks.getmiss, before its header signature or vote signature is checked. Neither path has a bound. Also, the engine returnsInvalidRoundfor a wrong height as well as a wrong round, and the driver buffers whenevervote.round > round, so votes for other heights are buffered too.Possible direction: one admission pipeline (height, round window, signature and membership, then buffer) for every path, with a keyed and capped buffer.
5. Blocks stored before the proposal is authenticated (high, memory DoS).
receive_proposalinserts the block intoblocksbeforeengine.receive_proposalchecks the leader and signature. Nothing ever removes entries fromblocks. The only checks before the insert are the hash match andseqno(theprev_hashcheck is dead becauseset_headis never called), so any peer can make the node store any number of blocks of up to 8 MiB each.Possible direction: verify the proposal (leader and signature) before storing, and prune
blockson view change and height change.6. Locked rejection mislabeled as equivocation (medium, false evidence). The engine returns
ConflictingProposalboth for a real double proposal and for "locked on a different block". The driver logs everyConflictingProposalasEQUIVOCATION evidence: leader … double-proposed. An honest leader proposing a different block to a locked validator is recorded as slashable misconduct, which ADR-0050 (accountability) treats as evidence.Possible direction: a separate
LockedOnOtherBlockerror, logged as a normal rejection.7. Spec and engine disagree on the lock rule (low, spec drift).
consensus.md§3.7 rule 2 locks on a >2/3 PreVote quorum; ADR-0049 §1 andengine.rslock on a >2/3 PreCommit quorum. Rule 1 unlocks on a "proposal [that] carries a QC from round >locked_round"; since2b87615the engine ignores the proposal's QC and uses a locally observed POL. CONTRIBUTING asks for spec deviations to be stated in the PR; this records it.Possible direction: keep the engine's rule and rewrite §3.7 (ADR-0052 open decision 4).
Fix design: ADR-0052 (proposed, not implemented)
04f0558addsdocs/adr/0052-pol-certificates-and-bounded-vote-admission.md, amending ADR-0049 §1 and §5. Design only: no implementation code, and the three#[ignore]tests are unchanged.PolCertificatemessage,KIND_POLenvelope) because a Byzantine validator can deliver its PreVote selectively. A locked leader re-proposes its best-certified block.blocksis pruned to at most three.Err.LockedOnOtherBlockerror; only a real double proposal is logged as equivocation.qc_*field names.Review history (from the original PR #59 on GitHub)
052d9a6: regression tests for review probe 2 (votes_before_proposal_are_buffered) and probe 4 (proposal_with_bad_seqno_is_rejected). Both pass.0ddbe44: sign the QC fields.qc_round/qc_blockare included inproposal_signing_bytes, so a proposer can't alter them after signing. fmt and site rebuild. Recorded the known gap:set_headis never called.69f1ae1(re-audit): store the commit-vote header signature only after the engine accepts the vote. Buffer Commit votes whose block hasn't arrived yet. Buffer future-round votes instead of dropping them (see finding 2). A locked leader refuses to propose a different block.2b87615(critical safety fix): unlock only on a locally observed POL instead of the leader-claimedqc_round, which a Byzantine leader could forge to unlock honest validators and fork the chain. Reject Commit votes without a header signature (they counted toward finality but added noSigEntry). Keep the pending buffer across round timeouts (see finding 3).c1867bd(this PR): regression tests for2b87615(forged_qc_round_does_not_unlock,commit_vote_without_header_sig_is_rejected, both pass) and the three finding tests above (all fail).04f0558(this PR): ADR-0052, the fix design for findings 1–6 (docs only; ADR-0049 amendment note; site ADR list regenerated). Drafted in a cloud Claude session on a copy of the repo and applied here byte for byte (same file hashes as that session's patcha4c7212).Known gaps
set_headis never called (recorded in0ddbe44). The producer doesn't setblock.header.prev_hashyet, soproducer.rsleavesdriver.set_head()as aTODO(ADR-0049). The driver's "proposal builds on local head" check is dead code until the producer is fixed.consensus_driver.rs, consensus.md §3.4). Validators check seqno andprev_hash(dead, see above) but don't re-execute the block before voting. The driver has no access to the state at the head yet.History note
The branch history contains the original M6 series twice, a duplicate from a rebase. 15
M6:commits appear twice (the two series 4321eb5…b5a7115 and 2040d6e…c2a70f1). 10 of the pairs have identical patches, and 5 differ only in context from the rebase. The net diff is correct (19 files, +2731/−152 againstmain). Squash-merging would give a clean history. A plain merge keeps the duplicate commits.Merge
091c61amergesmain(the Forgejo migration, PR #1) into the branch.site/index.htmlwas resolved by regenerating it (scripts/site.py build). ADR numbering: this branch keeps ADR-0049, because it was written first (2026-10-09). The council ADR from pr60 became ADR-0051 (PR #2).Prompts
Operator prompt, verbatim, for the session that produced this PR and its sibling. Redactions are marked inline.
Prompts for the
04f0558update (ADR-0052), verbatim:Claude-Session: https://claude.ai/code/session_01E3fK4bHkZ2YbqXZbGZsSh6.Review relay
N/A. The GitHub review relay was removed in the Forgejo migration. Review follows
CONTRIBUTING.md#review-process. Current self-assessment at04f0558: NO-GO. Critical finding 1 is outstanding, and the non-critical findings add 40 points (proven) or 112 (all outstanding). ADR-0052 awaits the owner's decisions before any fix is implemented.🤖 Generated with Claude Code
2b87615+ three failing finding tests c1867bdba9M6: BFT consensus in onxd (ADR-0049 locking) — NO-GO: 3 open findingsto M6: BFT consensus in onxd (ADR-0049 locking) — NO-GO: 1 critical + 112 pts, fix design ADR-0052 proposedView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.