Fixing Issue #80: "Images in messages affected by Discord's new Signed Attachment CDN URLs". Adding attachment archiver.

[ncjsvr]

What this does

Discord CDN links expire after about 24 hours so any images/attachments in old logs just break. This adds an optional background archiver that downloads them before they expire and stores them in Mongodb GridFS so they always load.

Fixes #80 - #80

Changes

• Added a new archiver module that runs in the background, scans logs for discord cdn urls, downloads them and stores in gridfs. Compresses images to jpeg to save space. Has configurable interval, retention, quality etc
• Updated models so when viewing a log, it checks if attachments have been archived and swaps in the server url instead of the discord one
• Added an /attachments/ route to serve the archived files
• Updated env example, app config, and readme with the new config options
• Updated dockerfile, pipfile, and requirements to add aiohttp and pillow
• Added logging so you can see this process in console.

Config

Off by default, set SAVE_ATTACHMENTS=yes to enable. All the options are in the env example file.

Edit

Just wanted to say I have 72 archived attachments in my mongo database and it uses 3mb currently.

[martinbndr]

In general I like the idea but in my opinion most of the users are relying on the free mongodb tier which has limited data storage. So many of them wouldn't really benefit from this feature as it gets limited quickly by the said storage limitation.

[ncjsvr]

In general I like the idea but in my opinion most of the users are relying on the free mongodb tier which has limited data storage. So many of them wouldn't really benefit from this feature as it gets limited quickly by the said storage limitation.

Understand your point, right now I have 72 archived attachments in my mongo database and it uses 3mb currently. However I did add the compression settings and retention periods.

I think it probably depends how big your server is, this was tested in a 10k member server and we got those 72 images in 3 days. Any server that heavily relies on modmail more may have problems.