Skip to content

Comments

fix(epoxy): make quorum sizes match paper#4234

Open
MasterPtato wants to merge 1 commit intomainfrom
02-19-fix_epoxy_make_quorum_sizes_match_paper
Open

fix(epoxy): make quorum sizes match paper#4234
MasterPtato wants to merge 1 commit intomainfrom
02-19-fix_epoxy_make_quorum_sizes_match_paper

Conversation

@MasterPtato
Copy link
Contributor

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@railway-app
Copy link

railway-app bot commented Feb 19, 2026

🚅 Deployed to the rivet-pr-4234 environment in rivet-frontend

Service Status Web Updated (UTC)
ladle ❌ Build Failed (View Logs) Web Feb 21, 2026 at 1:41 am
website 😴 Sleeping (View Logs) Web Feb 19, 2026 at 8:56 pm
frontend-inspector ❌ Build Failed (View Logs) Web Feb 19, 2026 at 8:48 pm
mcp-hub ✅ Success (View Logs) Web Feb 19, 2026 at 8:48 pm
frontend-cloud ❌ Build Failed (View Logs) Web Feb 19, 2026 at 8:48 pm

Copy link
Contributor Author

MasterPtato commented Feb 19, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more


How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@claude
Copy link

claude bot commented Feb 19, 2026

PR Review: fix(epoxy): make quorum sizes match paper

Summary

This PR makes good improvements to edge case handling and code clarity, but contains a correctness issue in the fast quorum formula that could violate EPaxos safety invariants.


Critical: Fast Quorum Formula Produces Unsafe Values

The new formula in calculate_quorum for QuorumType::Fast:

QuorumType::Fast => f + (f + 1) / 2,

With integer (floor) division, this gives values that are smaller than EPaxos requires. The EPaxos safety requirement for fast quorums is that any two fast quorums must intersect in at least f+1 replicas. By the pigeonhole principle, this requires:

Q_F ≥ ⌈(N + f + 1) / 2⌉

Comparing the three formulas:

N f Old (n*3)/4+1 New f+(f+1)/2 Required ⌈(N+f+1)/2⌉
3 1 3 ✓ 2 ✗ 3
4 1 4 (over-conservative) 2 ✗ 3
5 2 4 ✓ 3 ✗ 4
6 2 5 ✓ 3 ✗ 5
7 3 6 ✓ 5 ✗ 6

The old formula (n * 3) / 4 + 1 was correct (or slightly over-conservative for even N), while the new formula is consistently under the minimum safe value for all N ≥ 3. This also propagates to calculate_fanout_quorum since it derives from the same formula.

The corrected formula in Rust integer arithmetic (equivalent to ⌈(N+f+1)/2⌉) would be:

QuorumType::Fast => (n + f + 2) / 2,

Verification:

  • N=3, f=1: (3+1+2)/2 = 3 ✓
  • N=5, f=2: (5+2+2)/2 = 4 ✓
  • N=7, f=3: (7+3+2)/2 = 6 ✓
  • N=4, f=1: (4+1+2)/2 = 3 ✓ (vs old formula's conservative 4)

Since calculate_quorum feeds into update_config.rs to set the cluster-wide quorum sizes, this bug would affect the entire cluster's safety.


Good Changes

Slow quorum formula fix (engine/packages/epoxy/src/utils.rs:89): Changing from n/2 + 1 to f + 1 correctly handles even N:

  • N=4: old formula gives 3, new gives f+1 = 2 — which is correct per EPaxos (f=1 for N=4)
  • N=6: old formula gives 4, new gives f+1 = 3 — correct (f=2 for N=6)

This is a genuine improvement.

Edge case handling: The explicit n=0, n=1, and n=2 cases in both functions are correct and the comments explaining why EPaxos doesn't apply below N=3 are helpful.

calculate_fanout_quorum abstraction: Moving the fanout quorum calculation into its own function and calling it before spawning futures is cleaner than computing it after the fact with the inline match block.

anyhow import cleanup (engine/packages/epoxy/src/http_client.rs:1): The glob import removal follows CLAUDE.md conventions.

Ok/Err simplification: Replacing std::result::Result::Ok with Ok throughout is correct cleanup.


Minor Issues

Debug log lost quorum_type (engine/packages/epoxy/src/http_client.rs:64): The updated log drops the quorum_type field, reducing observability:

// before
tracing::debug\!(?quorum_size, len = ?responses.len(), ?quorum_type, "fanout quorum size");

// after  
tracing::debug\!(?target_responses, len=?responses.len(), "fanout target");

Consider keeping ?quorum_type in the log. Also the spacing len=? diverges from len = ? style used elsewhere.

Missing unit tests: The quorum math is subtle enough that unit tests would catch regressions like this. Consider adding a #[cfg(test)] module to utils.rs with assertions for each (N, quorum_type) combination against known-correct values from the EPaxos paper.


Suggested Fix for utils.rs

// See EPaxos 4.3
pub fn calculate_quorum(n: usize, q: QuorumType) -> usize {
    match n {
        0 => 0,
        1 => 1,
        2 => match q {
            QuorumType::Fast | QuorumType::Slow | QuorumType::All => 2,
            QuorumType::Any => 1,
        },
        n => {
            let f = (n - 1) / 2;
            match q {
                QuorumType::Fast => (n + f + 2) / 2,  // ⌈(N+f+1)/2⌉
                QuorumType::Slow => f + 1,
                QuorumType::All => n,
                QuorumType::Any => 1,
            }
        }
    }
}

pub fn calculate_fanout_quorum(n: usize, q: QuorumType) -> usize {
    // Quorum excluding the sender: calculate_quorum(n, q) - 1 for the sender's contribution
    match n {
        0 | 1 => 0,
        2 => match q {
            QuorumType::Any => 1,
            _ => 1,  // only 1 other node exists
        },
        n => {
            let f = (n - 1) / 2;
            match q {
                QuorumType::Fast => (n + f + 2) / 2 - 1,  // ⌈(N+f+1)/2⌉ - 1
                QuorumType::Slow => f,
                QuorumType::All => n - 1,
                QuorumType::Any => 1,
            }
        }
    }
}

@pkg-pr-new
Copy link

pkg-pr-new bot commented Feb 19, 2026

More templates

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@4234

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@4234

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@4234

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@4234

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@4234

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@4234

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@4234

@rivetkit/sqlite-vfs

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sqlite-vfs@4234

@rivetkit/traces

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/traces@4234

@rivetkit/workflow-engine

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/workflow-engine@4234

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@4234

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@4234

commit: 5973cf6

@MasterPtato MasterPtato force-pushed the 02-19-fix_epoxy_make_quorum_sizes_match_paper branch from 788e9db to e7553c7 Compare February 19, 2026 21:14
@MasterPtato MasterPtato marked this pull request as ready for review February 19, 2026 21:14
@MasterPtato MasterPtato force-pushed the 02-17-fix_gas_gracefully_handle_corrupt_wf branch from c7242fc to e418d91 Compare February 19, 2026 22:41
@MasterPtato MasterPtato force-pushed the 02-19-fix_epoxy_make_quorum_sizes_match_paper branch from e7553c7 to 710de07 Compare February 19, 2026 22:41
@MasterPtato MasterPtato force-pushed the 02-17-fix_gas_gracefully_handle_corrupt_wf branch from e418d91 to 0e4f8dc Compare February 19, 2026 22:51
@MasterPtato MasterPtato force-pushed the 02-19-fix_epoxy_make_quorum_sizes_match_paper branch from 710de07 to fbd667c Compare February 19, 2026 22:53
@MasterPtato MasterPtato force-pushed the 02-17-fix_gas_gracefully_handle_corrupt_wf branch from 0e4f8dc to 233582b Compare February 19, 2026 22:53
@graphite-app graphite-app bot changed the base branch from 02-17-fix_gas_gracefully_handle_corrupt_wf to graphite-base/4234 February 19, 2026 22:55
@graphite-app graphite-app bot force-pushed the graphite-base/4234 branch from 233582b to 682f215 Compare February 19, 2026 22:56
@graphite-app graphite-app bot force-pushed the 02-19-fix_epoxy_make_quorum_sizes_match_paper branch from fbd667c to 6b4b34d Compare February 19, 2026 22:56
@graphite-app graphite-app bot changed the base branch from graphite-base/4234 to main February 19, 2026 22:57
@graphite-app graphite-app bot force-pushed the 02-19-fix_epoxy_make_quorum_sizes_match_paper branch from 6b4b34d to 5973cf6 Compare February 19, 2026 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant