Add support for external Redis with embedded Redis disable option

[hoi]

Summary

Add support for using external Redis instead of the embedded Redis server that runs inside the Docker container. This allows users to:

• Avoid unbounded memory growth from embedded Redis (observed 8+ GB in production)
• Share state across multiple Crawl4AI containers, especially for /crawl/job
• Manage their own Redis version for security patching

Related to #1671 (Redis CVE-2025-49844)

List of files changed and why

• Dockerfile - Add entrypoint script, update healthcheck to conditionally skip redis-cli when using external Redis
• deploy/docker/entrypoint.sh - New script to generate supervisord config without Redis program when CRAWL4AI_DISABLE_EMBEDDED_REDIS=true
• deploy/docker/utils.py - Add REDIS_URL environment variable override in load_config()
• deploy/docker/config.yml - Document redis.uri option and external Redis configuration
• deploy/docker/README.md - Add External Redis section to Configuration Recommendations
• tests/docker/test_external_redis.py - Add 13 unit tests for external Redis configuration

How Has This Been Tested?

• 13 unit tests added covering:
  • REDIS_URL environment variable handling in load_config()
  • entrypoint.sh script structure and conditional logic
  • Dockerfile healthcheck conditional Redis check
  • config.yml documentation of new options
• All tests pass locally

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes