Zoekt failed indexing shards #805
Conversation
When zoekt-git-index fails during repository indexing, it can leave behind .tmp shard files that accumulate over time and fill up disk space. This is especially problematic for large repos that repeatedly fail to index. Changes: - Add cleanupTempShards() function to zoekt.ts that removes temporary shard files (files with .tmp in their name) for a specific repository - Call cleanupTempShards() in repoIndexManager.ts when indexGitRepository fails, before re-throwing the error This ensures that even if a repository consistently fails to index, the temporary files created during each attempt are cleaned up. Co-authored-by: michael <michael@sourcebot.dev>
|
Cursor Agent can help with this pull request. Just |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| try { | ||
| const { durationMs } = await measure(() => indexGitRepository(repo, this.settings, revisions, signal)); | ||
| const indexDuration_s = durationMs / 1000; | ||
| logger.info(`Indexed ${repo.name} (id: ${repo.id}) in ${indexDuration_s}s`); |
There was a problem hiding this comment.
I think this is a fair workaround and we should probably have a mechanism for cleaning up these files, but it doesn't really address the root cause for why these files are being created in the first place.
Clean up temporary Zoekt shard files on indexing failure to prevent disk space exhaustion.
When
zoekt-git-indexfails during repository indexing, it leaves behind.tmpshard files. These accumulate over time, especially for repos that repeatedly fail to index, leading to disk space issues. This PR adds logic to automatically remove these temporary files immediately after an indexing operation fails.Linear Issue: SOU-306