Fixing Issue #80: "Images in messages affected by Discord's new Signed Attachment CDN URLs". Adding attachment archiver.

.env.example

@@ -5,9 +5,23 @@ LOG_URL_PREFIX=/logs
 # Listen address and port. Don't change them if you don't know what they do.
 HOST=0.0.0.0
 PORT=8000
-# Whether if the logviewer should use a proxy to view attachments.
-# If set to "no" (default), attachments will expire after 1 day and the logviewer won't be able to show the attachment.
+# Attachments will expire after 1 day and the logviewer won't be able to show the attachment.
 # Please be aware that this may violate Discord TOS and the proxy will have full access to your attachments.
+# Your options are to either use an attachment proxy or archive them to Mongo GridFS.
 # Modmail/Logviewer is not affiliated with the proxy in any way. USE AT YOUR OWN RISK.
 USE_ATTACHMENT_PROXY=no
 ATTACHMENT_PROXY_URL=https://cdn.discordapp.xyz
+# Archive attachments to MongoDB GridFS so they persist beyond Discord's 24h CDN expiry.
+SAVE_ATTACHMENTS=no
+# How often (in seconds) the archiver scans for new unarchived attachments. Default: 600 (10 minutes)
+ARCHIVE_INTERVAL=600
+# Maximum file size in bytes to archive. Files larger than this are skipped. Default: 26214400 (25 MB)
+ARCHIVE_MAX_FILE_SIZE=26214400
+# How long to keep archived attachments. Options: 1w, 1month, 1y, forever. Default: forever
+ARCHIVE_RETENTION=forever
+# Compress images to JPEG before storing. Greatly reduces storage usage. Default: yes
+ARCHIVE_COMPRESS_IMAGES=yes
+# JPEG quality (1-100). Lower = smaller files. 65 is a good balance of quality and size. Default: 65
+ARCHIVE_IMAGE_QUALITY=65
+# Max image resolution (longest edge in pixels). Images larger than this are downscaled. Default: 1920
+ARCHIVE_IMAGE_MAX_RESOLUTION=1920

.gitignore

@@ -406,3 +406,4 @@ dist
 .yarn/build-state.yml
 .yarn/install-state.gz
 .pnp.*
+

README.md

@@ -110,6 +110,51 @@ We recommend setting up a reverse proxy (e.g. Nginx) to port forward external po
 
 To accept requests from a domain instead of your server IP, simply set an `A`/`AAAA` record from your DNS provider to forward your domain to your server IP.
 
+## Preserving Attachments
+
+Discord's CDN URLs for attachments expire after ~24 hours, which means images in your logs will break. There are two ways to fix this:
+
+### Option 1: Attachment Archival (Recommended)
+
+Downloads attachments and stores them directly in your MongoDB using GridFS. No third parties involved.
+
+Add to your `.env` file:
+```
+SAVE_ATTACHMENTS=yes
+```
+
+| Variable | Default | Description |
+|---|---|---|
+| `SAVE_ATTACHMENTS` | `no` | Enable attachment archival (`yes` / `no`) |
+| `ARCHIVE_INTERVAL` | `600` | Seconds between scans for new attachments |
+| `ARCHIVE_MAX_FILE_SIZE` | `26214400` | Max file size in bytes to archive (default 25 MB) |
+| `ARCHIVE_RETENTION` | `forever` | How long to keep files: `1w`, `1month`, `1y`, `forever` |
+| `ARCHIVE_COMPRESS_IMAGES` | `yes` | Compress images to JPEG before storing (`yes` / `no`) |
+| `ARCHIVE_IMAGE_QUALITY` | `65` | JPEG quality 1-100 (lower = smaller files) |
+| `ARCHIVE_IMAGE_MAX_RESOLUTION` | `1920` | Downscale images larger than this (longest edge in px) |
+
+A background task scans your logs collection periodically and downloads any Discord CDN attachment/avatar URLs it finds. Images are compressed to JPEG (configurable) and stored in MongoDB GridFS. When a log page is viewed, archived images are served from `/attachments/` instead of the expired Discord URLs.
+
+> [!NOTE]
+> Attachments that already expired before enabling this feature cannot be recovered. Enable it as soon as possible to archive existing logs while their URLs are still valid.
+
+> [!NOTE]
+> The free MongoDB Atlas tier (M0) has a 512 MB storage limit. With compression enabled, this can hold roughly 2,000-10,000 images depending on size. Consider adjusting `ARCHIVE_IMAGE_QUALITY` and `ARCHIVE_IMAGE_MAX_RESOLUTION` to reduce storage usage, or use a paid tier for more space.
+
+### Option 2: Attachment Proxy
+
+Routes image requests through a third-party proxy service instead of storing them yourself. Simpler setup but relies on an external service.
+
+| Variable | Default | Description |
+|---|---|---|
+| `USE_ATTACHMENT_PROXY` | `no` | Enable the attachment proxy (`yes` / `no`) |
+| `ATTACHMENT_PROXY_URL` | `https://cdn.discordapp.xyz` | The proxy service URL (not your site URL) |
+
+> [!WARNING]
+> The proxy service is not affiliated with Modmail. It will have full access to your attachments. Use at your own risk.
+
+You can enable both options together - archived images take priority, and the proxy is used as a fallback for images not yet archived.
+
 ## Discord OAuth2
 
 Protecting your logs with a login (Discord Oauth2 support) is a premium feature, only available to [Premium members](https://buymeacoffee.com/modmaildev).

app.json

@@ -16,6 +16,31 @@
             "description": "Proxy URL for viewing attachments.",
             "required": false,
             "value": "https://cdn.discordapp.xyz"
+        },
+        "SAVE_ATTACHMENTS": {
+            "description": "Whether to archive attachments to MongoDB GridFS. Set to 'yes' to enable.",
+            "required": false,
+            "value": "no"
+        },
+        "ARCHIVE_INTERVAL": {
+            "description": "How often (in seconds) to scan for unarchived attachments. Default: 600",
+            "required": false,
+            "value": "600"
+        },
+        "ARCHIVE_RETENTION": {
+            "description": "How long to keep archived attachments. Options: 1w, 1month, 1y, forever. Default: forever",
+            "required": false,
+            "value": "forever"
+        },
+        "ARCHIVE_COMPRESS_IMAGES": {
+            "description": "Compress images to JPEG before storing to save space. Default: yes",
+            "required": false,
+            "value": "yes"
+        },
+        "ARCHIVE_IMAGE_QUALITY": {
+            "description": "JPEG quality (1-100). Lower = smaller files. Default: 65",
+            "required": false,
+            "value": "65"
         }
     }
 }

app.py

@@ -3,13 +3,14 @@
 import html
 import os
 
+from bson import ObjectId
 from dotenv import load_dotenv
-from motor.motor_asyncio import AsyncIOMotorClient
+from motor.motor_asyncio import AsyncIOMotorClient, AsyncIOMotorGridFSBucket
 from sanic import Sanic, response
 from sanic.exceptions import NotFound
 from jinja2 import Environment, FileSystemLoader
 
-from core.models import LogEntry
+from core.models import LogEntry, build_archive_lookup
 
 load_dotenv()
 
@@ -61,6 +62,15 @@ def strtobool(val):
         raise ValueError("invalid truth value %r" % (val,))
 
 
+SAVE_ATTACHMENTS = strtobool(os.getenv("SAVE_ATTACHMENTS", "no"))
+ARCHIVE_INTERVAL = int(os.getenv("ARCHIVE_INTERVAL", "600"))
+ARCHIVE_MAX_FILE_SIZE = int(os.getenv("ARCHIVE_MAX_FILE_SIZE", str(25 * 1024 * 1024)))
+ARCHIVE_RETENTION = os.getenv("ARCHIVE_RETENTION", "forever").strip().lower()
+ARCHIVE_COMPRESS_IMAGES = strtobool(os.getenv("ARCHIVE_COMPRESS_IMAGES", "yes"))
+ARCHIVE_IMAGE_QUALITY = int(os.getenv("ARCHIVE_IMAGE_QUALITY", "65"))
+ARCHIVE_IMAGE_MAX_RESOLUTION = int(os.getenv("ARCHIVE_IMAGE_MAX_RESOLUTION", "1920"))
+
+
 @app.listener("before_server_start")
 async def init(app, loop):
     app.ctx.db = AsyncIOMotorClient(MONGO_URI).modmail_bot
@@ -71,6 +81,32 @@ async def init(app, loop):
     else:
         app.ctx.attachment_proxy_url = None
 
+    # Attachment archival setup
+    app.ctx.save_attachments = bool(SAVE_ATTACHMENTS)
+    if app.ctx.save_attachments:
+        app.ctx.fs = AsyncIOMotorGridFSBucket(app.ctx.db, bucket_name="attachments")
+        await app.ctx.db.archived_attachments.create_index("original_url", unique=True)
+        await app.ctx.db.archived_attachments.create_index("status")
+        await app.ctx.db.archived_attachments.create_index("archived_at")
+    else:
+        app.ctx.fs = None
+
+
+@app.listener("after_server_start")
+async def start_archiver(app, loop):
+    if app.ctx.save_attachments:
+        from core.archiver import run_archiver_loop
+        archiver_config = {
+            "interval": ARCHIVE_INTERVAL,
+            "max_file_size": ARCHIVE_MAX_FILE_SIZE,
+            "retention": ARCHIVE_RETENTION,
+            "compress_images": bool(ARCHIVE_COMPRESS_IMAGES),
+            "image_quality": ARCHIVE_IMAGE_QUALITY,
+            "image_max_resolution": ARCHIVE_IMAGE_MAX_RESOLUTION,
+        }
+        app.add_task(run_archiver_loop(app, archiver_config))
+
+
 @app.exception(NotFound)
 async def not_found(request, exc):
     return render_template("not_found")
@@ -89,7 +125,8 @@ async def get_raw_logs_file(request, key):
     if document is None:
         raise NotFound
 
-    log_entry = LogEntry(app, document)
+    archive_lookup = await build_archive_lookup(app, document)
+    log_entry = LogEntry(app, document, archive_lookup=archive_lookup)
 
     return log_entry.render_plain_text()
 
@@ -102,11 +139,43 @@ async def get_logs_file(request, key):
     if document is None:
         raise NotFound
 
-    log_entry = LogEntry(app, document)
+    archive_lookup = await build_archive_lookup(app, document)
+    log_entry = LogEntry(app, document, archive_lookup=archive_lookup)
 
     return log_entry.render_html()
 
 
+@app.get("/attachments/<file_id>/<filename>")
+async def serve_attachment(request, file_id, filename):
+    """Serve an archived attachment from GridFS."""
+    if not app.ctx.save_attachments or app.ctx.fs is None:
+        raise NotFound
+
+    try:
+        oid = ObjectId(file_id)
+    except Exception:
+        raise NotFound
+
+    try:
+        grid_out = await app.ctx.fs.open_download_stream(oid)
+    except Exception:
+        raise NotFound
+
+    content_type = "application/octet-stream"
+    if grid_out.metadata:
+        content_type = grid_out.metadata.get("content_type", content_type)
+
+    data = await grid_out.read()
+    return response.raw(
+        data,
+        content_type=content_type,
+        headers={
+            "Content-Disposition": f'inline; filename="{filename}"',
+            "Cache-Control": "public, max-age=31536000, immutable",
+        },
+    )
+
+
 if __name__ == "__main__":
     app.run(
         host=os.getenv("HOST", "0.0.0.0"),