Worker event stream

[mitchhs12]

Summary

Adds opt-in Kafka event streaming for worker sync progress, enabling real-time monitoring of dataset sync jobs.

• Emit lifecycle events (sync.started, sync.progress, sync.completed, sync.failed) per-table
• Protobuf encoding via prost; Kafka delivery via rskafka (no librdkafka dependency)
• 1% throttling for progress events to emit every 1% increment; avoids noise on high-throughput chains
• Configurable partition count (default: 16)
• Best-effort delivery with retry backoff (1s, 5s, 60s) - Kafka failures don't fail jobs

See app-ampd-worker-events.md for full documentation.

• Events are per-table (correct for Kafka partitioning), but raw_dataset couples the lifecycle timing of all tables because they share a single block stream.
• Retry policy uses fixed delays (1s, 5s, 60s), max 66s wait. Silent failure after exhaustion is intentional: events are best-effort.
• Per-table partitioning enables consumer parallelism; aggregation by job_id provides job-level view.
• Sync ProgressCallback trait avoids async overhead; WorkerProgressCallback bridges to async via tokio::spawn.

Test plan

• Unit tests for retry backoff and error handling
• Integration tests for all event types
• E2E test with Anvil verifying full lifecycle (3 tables × lifecycle events)

[mitchhs12]

Questions:

1. Should we add a sync.caught_up event for continuous datasets when they first reach chain head? I think this could be useful for consumers.
2. When a chain reorg causes progress to go backwards, should we:
   • Emit a sync.reorg event?
   • Suppress backwards progress events?
   • Let consumers handle it (current behavior)?

[cmwhited]

I am going to keep reviewing. But I have concerns about the KafkaConfig
like I commented:

• it is currently duplicated between the kafka client crate and the config crate
• it doesn't expose all available connection params, including things like auth, which is concerning

[LNSD]

Please, check my comments 🙂

[LNSD]

Please, check my comments 🙂

[LNSD]

As a piece of advice, this is a considerable PR (many logical changes and many files changed). Although it is ok from the POV of having an overall vision of the feature, I would recommend that you work on small incremental PR instead, as big PRs are prone to reworks due to merge conflicts.

Also, I recommend that you run the /code-review agent skill locally to catch code style issues in advance and accelerate the feedback loop.

[LNSD]

Please, check my comments 🙂

[LNSD]

Please, check my comments 🙂

P.S., you’re doing great with this PR, and I really appreciate all the effort you’re putting into it.

[LNSD]

LGTM ✅

All feedback addressed in subsequent commits. LNSD approved.