BUG: New entities silently skip embedding generation after sqlite-vec load failure

[bm-clawd]

Problem

If sqlite-vec fails to load during a bm watch session (e.g. SemanticDependenciesMissingError), subsequent new entities get indexed in FTS but not in the vector store. There is no recovery mechanism — the watcher continues running but silently skips embeddings for all new entities.

Reproduction

1. Start bm watch --project claw
2. At some point, sqlite-vec fails to load (observed in logs: SemanticDependenciesMissingError: sqlite-vec package is missing)
3. New files are created and synced — they appear in FTS search
4. But search_vector_chunks has zero rows for these entities
5. Vector and hybrid search can't find them

Evidence

-- 5 of 66 entities had zero vector chunks
SELECT id, title FROM entity 
WHERE project_id = 2 
AND id NOT IN (SELECT DISTINCT entity_id FROM search_vector_chunks);
-- Returns entities created after the sqlite-vec error

Workaround

bm reindex --embeddings -p claw regenerates all embeddings. But users won't know to run this because the failure is silent.

Suggested Fix

1. Log a warning on every sync when embeddings fail (not just the first time)
2. Retry sqlite-vec loading periodically
3. Track embedding failures and expose them in bm status or bm doctor
4. Consider: if semantic search is enabled but vec can't load, should the watcher error out loudly rather than silently degrade?

Environment

• BM v0.18.3, sqlite-vec 0.1.6 (installed but intermittently fails to load)
• macOS, Intel Mac

[claude]

Claude finished @bm-clawd's task in 3m 22s —— View job

---

Triage Analysis

• Analyze issue and classify type/priority/component
• Explore relevant code (sqlite-vec loading, embedding pipeline, watcher)
• Check for duplicate issues
• Apply labels
• Provide triage summary

---

Classification

Dimension	Assessment
Type	Bug — real code defect (silent failure, no recovery)
Priority	Medium — functionality silently degrades but workaround exists (bm reindex --embeddings)
Component	Database / Sync (embedding pipeline)
Complexity	Medium — investigation + error handling changes needed across several files

Labels Applied

• bug ✅

---

Code Investigation

I traced the bug through the codebase. Here's what's happening:

1. Background task failure path

When a file is synced, sync_entity_vectors() is scheduled as a background task in src/basic_memory/deps/services.py:513:

if app_config.semantic_search_enabled and scheduler is not None:
    scheduler.schedule("sync_entity_vectors", entity_id=entity_id)

The task completion callback at deps/services.py:442-446 is:

def _log_task_failure(completed: asyncio.Task) -> None:
    try:
        completed.result()
    except Exception as exc:  # pragma: no cover
        logger.exception("Background task failed", error=str(exc))

When SemanticDependenciesMissingError is raised inside sync_entity_vectors(), it's caught here and only logged — no retry, no user notification.

2. sqlite-vec loading is session-scoped

_ensure_sqlite_vec_loaded() in src/basic_memory/repository/sqlite_search_repository.py:338-366 loads the extension per async session. If loading fails mid-run (e.g., intermittent import failure), the session's vector operations fail, but the watcher continues.

3. Startup protection doesn't help here

There's a fail-fast check at sqlite_search_repository.py:89-92 for startup, but the intermittent failure happens later during background task execution.

---

Root Causes

1. Background task errors are fire-and-forget — only logger.exception() is called, nothing surfaced to the user
2. No retry mechanism — failed embedding tasks are dropped permanently
3. No state tracking — embedding failure count isn't tracked anywhere visible (e.g., bm status)
4. Watch service's error recording (watch_service.py:207-212) only catches errors in the main cycle loop, not in detached background tasks

---

Suggested Fix Approach

The issue suggests 4 options. Based on the code:

1. Highest value / lowest risk: Track embedding failure counts in the background scheduler and expose them via bm status. The watcher already writes a status JSON file (write_status()).

2. Medium effort: After SemanticDependenciesMissingError in a background task, add it to a recoverable error queue and emit a log warning on each new file sync (not just the first).

3. Higher effort but correct: Add an explicit retry mechanism in LocalTaskScheduler with exponential backoff for SemanticDependenciesMissingError specifically.

4. Nuclear option: If sqlite-vec fails during an active watch session and embeddings can't be generated, fail loudly and require the user to restart. This matches the principle of not silently degrading.

The immediate quick fix would be: in _log_task_failure, detect SemanticDependenciesMissingError specifically and set a flag that causes each subsequent sync cycle to emit a logger.warning() with "embeddings skipped — run bm reindex --embeddings to recover."

---

No Duplicates Found

Searched for related issues — no duplicates exist.