ReorderBuffer crash loop: no process error when calling SlotProcessorServer during backfill-to-real-time transition

[hiasr]

Hi!
Once I start to use backfilling, it consistently breaks and requires me to delete the DB connection and replication slot to get it working again.
Backfilling a table works, but once I want to backfill a schema it breaks down.

Context:

• Sequin version: v0.14.1
• Deployment: I tried both on Kubernetes and raw docker compose in EC2
• Postgres: 16 on RDS (no replica)
• Sequin config: RabbitMQ sink with 1 PG schema configured
• Logs: hyperdx_search_results_2026-01-11T10-04-20.csv

Symptoms observed:

• Once starting backfill I can't open the backfill page of the sink anymore
• After a while, or making some change (like adding a schema to the sink), it starts to break down and the SlotProcessor is in an error loop.

LLM log analysis: Gemini thought it might be due to a race condition of many backfills ending at the same time.

What Happened?
The Sequin.Runtime.SlotProducer.ReorderBuffer process crashed repeatedly because it tried to make a synchronous call (GenServer.call) to the SlotProcessorServer, but that process was not alive or registered.

The specific error message is: ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name

This occurred specifically when the ReorderBuffer received a :flush_batch message and tried to send data to the processor.

The Sequence of Events
Backfill Completion: At approximately 09:58:35.059Z, several backfill operations (e.g., IDs 010253bc, d4a5bc88, 22bd7330) completed in rapid succession.

Pipeline Teardown: As each backfill completed, the system sent a Stopping TableReaderServer signal to clean up the backfill readers.

Race Condition: It appears that stopping these readers or the completion of the backfills triggered a restart or reconfiguration of the replication pipeline.

The Crash: During this transition, the ReorderBuffer was still running and attempted to "flush" a batch of data it had collected. However, the SlotProcessorServer it was trying to reach had either already been shut down as part of the transition or had not yet finished starting/registering its name.

Flapping: Because several backfills finished nearly at the same time, the system "flapped"—continually trying to restart the pipeline while more backfills were finishing and triggering more restarts. Each time it restarted, the ReorderBuffer tried to flush and crashed again because the SlotProcessorServer wasn't ready.

The "Hunch" (Root Cause)
The root cause is a race condition during backfill-to-real-time transition.

When multiple backfills finish simultaneously, the frequent starting and stopping of the replication workers (SlotProcessor, SlotProducer, ReorderBuffer) creates a window where the ReorderBuffer starts up and receives data before the SlotProcessorServer is fully registered in the process registry (:syn). Since the ReorderBuffer uses a synchronous call to flush data, it crashes immediately if the target process is missing, rather than retrying or waiting.

Summary of the Log Evidence
Error Level Logs: Repeated GenServer ... terminating errors for the ReorderBuffer.

Stack Trace: Points to reorder_buffer.ex:122 in the handle_info function for :flush_batch.

Correlation: Every crash is preceded by a Stopping TableReaderServer ... backfill_status=completed info log.

Process PIDs: You can see new PIDs for the SlotProcessorServer (e.g., <0.8759.0>, <0.8775.0>) being initialized every second, confirming the system was trying to recover but failing.

Possible Fixes/Workarounds
Sequential Backfills: If possible, avoid having many backfills finish at the exact same millisecond (though this is often hard to control).

Resilience in ReorderBuffer: This may require a code change in Sequin to make the ReorderBuffer more resilient to a missing SlotProcessorServer (e.g., by using a retry mechanism or a longer timeout during name lookup).

[dosubot]

Hi @hiasr! I'm Dosu and I’m helping the sequin team.

This is a known race condition in Sequin: when multiple backfills finish at nearly the same time, the ReorderBuffer tries to flush batches to SlotProcessorServer before it's fully registered, causing a crash loop with the (EXIT) no process error. The codebase (including v0.14.1) does not have resilience or retry logic for this scenario—if SlotProcessorServer isn't alive, the synchronous GenServer.call in ReorderBuffer fails immediately and the process crashes, leading to repeated pipeline restarts and UI flapping source.

There are no config options or operational workarounds in Sequin to mitigate this, except trying to avoid simultaneous backfill completions, which is hard to control in practice. The only way to truly fix this would be a code change to make ReorderBuffer more resilient (e.g., adding retries or longer timeouts when looking up SlotProcessorServer), but Sequin is no longer actively maintained after October 2025 source.

If you need to minimize disruption, try running backfills sequentially and avoid schema-wide backfills when possible. If you want to dig deeper or look for community patches, let me know.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}