BUG: watch service can go stale while process stays alive (self-heal + heartbeat)

[phernandez]

Problem

bm watch can remain alive but stop indexing filesystem changes after long uptime (seen overnight on macOS).

This creates a silent failure mode for downstream tools: files exist on disk, but search/read by permalink and graph traversal lag indefinitely until manual watcher restart.

Related downstream issue: basicmachines-co/openclaw-basic-memory#15

Why this looks upstream

openclaw-basic-memory only spawns bm watch; it does not own file watching internals. The watch loop and restart behavior live in:

• src/basic_memory/sync/watch_service.py
• src/basic_memory/cli/commands/watch.py

Reproduction (observed)

1. Run bm watch --project <name>.
2. Leave running for several hours.
3. Create/modify markdown files under project path.
4. Process is still alive, but changes are not indexed until watcher is restarted.

Requested fix

Add explicit stale-watch self-healing and diagnostics in the watch service:

1. Add a heartbeat timestamp in watch status (updated even when no file events occur).
2. Detect stale cycles (watch loop alive but no event processing/heartbeat progress for configurable window).
3. Force a clean watch cycle restart and emit a structured warning when staleness is detected.
4. Expose staleness counters/last-restart reason in watch status JSON for monitoring.

Acceptance criteria

• Long-running watcher recovers automatically from stale state without manual restart.
• ~/.basic-memory/watch_status.json includes heartbeat + last restart reason.
• A synthetic integration test reproduces stale cycle handling (or equivalent deterministic test harness around watch loop restart logic).

[claude]

Claude finished @phernandez's task in 1m 26s —— View job

---

Triage Analysis

• Read issue and gather context
• Examine watch service implementation
• Check for duplicate issues
• Apply labels and provide triage summary

---

Issue Classification

Field	Assessment
Type	Bug (silent failure mode) + Enhancement (self-healing features)
Priority	High — long-running watcher silently stops indexing; no automatic recovery
Component	sync/ (watch service), cli/ (watch command)
Complexity	Complex — requires heartbeat mechanism, staleness detection, auto-restart logic, and status JSON schema changes

No duplicate issues found.

Labels applied: bug, enhancement

---

Root Cause Analysis

After reading src/basic_memory/sync/watch_service.py, the current design relies entirely on watchfiles.awatch() to deliver filesystem events. If awatch() internally goes stale (e.g., due to macOS FSEvents kernel buffer overflow, inotify limit exhaustion on Linux, or a hung OS-level watch descriptor), the async for loop simply blocks forever — there are no events delivered and no timeout or health check to detect this.

The existing _schedule_restart() mechanism does restart the watch cycle periodically (via watch_project_reload_interval), but:

• It only resets after the previous cycle completes, meaning a hung awatch() blocks the restart timer
• There is no heartbeat to distinguish "no files changed" from "watcher is stuck"
• watch_status.json only records last_scan (updated when files actually change), not a liveness heartbeat

Key gap in _watch_projects_cycle (line 120–157):

async for changes in awatch(*project_paths, stop_event=stop_event):
    # This blocks indefinitely if awatch goes silent

If awatch stalls, neither the timer task nor the outer loop detect it — the process remains alive and healthy-looking but processes zero events.

---

Requested Changes Summary

The issue requests four additions:

1. Heartbeat timestamp — updated periodically even when no file events occur (needs a concurrent heartbeat coroutine alongside awatch)
2. Staleness detection — compare last_heartbeat to now; if gap exceeds configurable threshold, trigger restart
3. Forced cycle restart — cancel the hung awatch task, log a structured warning with restart reason
4. Status JSON exposure — add last_heartbeat, stale_cycle_count, last_restart_reason to WatchServiceState / watch_status.json

---

Implementation Notes

• A heartbeat coroutine should run concurrently with awatch using asyncio.gather or asyncio.create_task, updating last_heartbeat every N seconds and checking for staleness
• The staleness threshold should be configurable via BasicMemoryConfig (e.g., watch_stale_threshold_seconds, defaulting to something like 300s)
• When staleness is detected, signal stop_event to break the awatch loop and let the outer while self.state.running loop restart the cycle
• WatchServiceState needs new fields: last_heartbeat, stale_cycle_count, last_restart_reason
• An integration/deterministic unit test should inject a mock awatch that stalls and verify the staleness detection fires and restarts the cycle

[phernandez]

Downstream tracking

• Downstream: BUG: bm watch goes stale — stops indexing filesystem changes openclaw-basic-memory#15