Setup tweet cap alert

[dewabisma]

Summary

When we pull tweets we track the usage and give alert if certain threshold has passed, so we know we are running out of quota.

Changes

• Add tweet_pull_usage table
• Add tweet_pull_usage repo
• Add alert service
• Update tweet sync with alert
• Update leaderboard sync with alert

[n13]

should not be checked in

[n13]

Code Review: Feat - Setup Tweet Cap Alert

This PR implements tracking of Twitter API usage and alerting when a threshold is reached. This is crucial for managing API costs/limits.

1. Suggestion: Alert Frequency

In src/services/alert_service.rs, the alert logic triggers whenever the current total exceeds the threshold:

if current_total >= self.config.tweet_sync.alert_threshold {
    // Sends alert...
}

Observation: Once the threshold is reached, every subsequent sync (e.g., every 10 minutes) will trigger another alert until the billing period resets.

Recommendation: Since the interval is 10 minutes, this spam might be acceptable for visibility. However, if you want to reduce noise in the future, you could modify the logic to only alert when the threshold is just crossed, or add a cooldown.

// Optional: Only alert if we crossed the line just now
if current_total >= threshold && previous_total < threshold {
    self.send_twitter_limit_alert(...).await?;
}

2. Period Calculation & Edge Cases

The billing period calculation in TweetPullUsageRepository::calculate_period_for_date is clever in handling short months, but "reset day" logic can be tricky.

• Observation: If reset_day is 31:
  • Feb 28th (non-leap) -> Period "Jan" (Start Jan 31).
  • March 1st -> Period "Feb" (Start Feb 28/29 effective).
  • This effectively shifts the "month label" by one. As long as this is consistent with your billing provider (e.g., X API billing cycle), it is fine. Just verify that the resulting "Period String" (e.g., "2023-01") matches the billing invoice you want to correlate it with.

3. Usage Tracking Resilience

The increment_usage method uses an atomic DB update (SET tweet_count = ... + EXCLUDED.tweet_count), which is excellent for concurrency.

However, consider the crash scenario in TweetSynchronizerService:

1. Tweets are pulled and saved.
2. track_and_alert_usage is called.
3. If Step 2 fails (DB connection blip), the error is logged but the sync function continues (tracing::error!).
4. Since the tweets were saved, last_id will be updated on the next run, so these tweets won't be pulled again.
5. Result: These tweets are never counted in the usage stats.

Recommendation: While perfect accuracy isn't required for an alert system, be aware that this is "at most once" counting behavior in failure modes. If stricter accounting is needed, usage increment should happen in the same transaction as saving the tweets (or relevant_tweets insertion). Given this is for "alerts" and not "billing audits", the current approach is likely acceptable.

4. Minor: Error Handling

In AlertService::send_webhook_alert:

Err(crate::AppError::Server(format!("Webhook alert failed with status {}: {}", status, body)))

This is a good use of the existing AppError. Ensure that AppError::Server is logged appropriately upstream.

Summary

The logic is solid. The alert frequency (every 10 minutes once threshold is met) is noted as acceptable for current operations.

[n13]

Note: I don't think alert spamming is critical when it's only every 10 minutes. It should kinda make noise, and every 10 minutes is not so bad.