Out of control data transfer for assets

[tao]

Bug description

I'm not really sure what to call the title of the bug report because I'm still trying to figure it out. I'll try provide more details soon if I can figure more out but it's a very expensive thing to debug.

Somehow since upgrading to Statamic 6? The queue..? or something transferred 2,851.887 GB of data (2,8 Tb) that cost an extra $256 to our AWS bill. Additionally it did 12 million (PUT,COPY,POST or List) Requests which cost an additional $60.

It took a while to figure out how to read the bucket usage from Amazon S3 but each day it's been charging $18 of usage to my assets.llresearch.dev bucket which is the one I use exclusively for our Statamic CMS assets on the test website.

So I think something went wrong with the queue if it made 12 million requests? I shut down the queue this morning and now it's only charging me $3 today.

It seems like the Statamic\Jobs\HandleEntrySchedule job runs every minute, so I'm not sure if that could be related somehow and causing the 12 million requests?

There are also some jobs in the queue related to Search that reference assets on AWS, the Statamic\Search\InsertMultipleJob one... but I'm not sure if that would be pinging AWS and trying to find or create metadata?

In the example payload for that job you can see it's full of my assets like asset::aws::archive\/images\/101_cover.jpg or asset::aws::audio\/101\/lesson_1.mp3

 id = 190
 queue = default
 payload = {"uuid":"dba5ab36-2152-441d-982b-2e3e9d458e1x","displayName":"Statamic\\Search\\InsertMultipleJob","job":"Illuminate\\Queue\\CallQueuedHandler@call","maxTries":null,"maxExceptions":null,"failOnTimeout":false,"backoff":null,"timeout":null,"retryUntil":null,"data":{"commandName":"Statamic\\Search\\InsertMultipleJob","command":"O:33:\"Statamic\\Search\\InsertMultipleJob\":3:{s:4:\"name\";s:7:\"default\";s:6:\"locale\";N;s:9:\"documents\";O:31:\"Statamic\\Assets\\AssetCollection\":2:{s:8:\"\u0000*\u0000items\";a:100:{i:0;s:30:\"asset::assets::don-quixote.svg\";i:1;s:30:\"asset::assets::icons\/about.svg\";i:2;s:34:\"asset::assets::icons\/beginners.svg\";i:3;s:30:\"asset::assets::icons\/books.svg\";i:4;s:37:\"asset::assets::icons\/carlas-niche.svg\";i:5;s:35:\"asset::assets::icons\/channeling.svg\";i:6;s:34:\"asset::assets::icons\/community.svg\";i:7;s:32:\"asset::assets::icons\/courses.svg\";i:8;s:28:\"asset::assets::icons\/faq.svg\";i:9;s:29:\"asset::assets::icons\/home.svg\";i:10;s:33:\"asset::assets::icons\/lawofone.svg\";i:11;s:32:\"asset::assets::icons\/library.svg\";i:12;s:41:\"asset::assets::icons\/magnifying-glass.svg\";i:13;s:30:\"asset::assets::icons\/media.svg\";i:14;s:36:\"asset::assets::icons\/meditations.svg\";i:15;s:33:\"asset::assets::icons\/navigate.svg\";i:16;s:33:\"asset::assets::icons\/podcasts.svg\";i:17;s:31:\"asset::assets::icons\/search.svg\";i:18;s:32:\"asset::assets::icons\/sidebar.svg\";i:19;s:30:\"asset::assets::icons\/study.svg\";i:20;s:31:\"asset::assets::icons\/topics.svg\";i:21;s:32:\"asset::assets::icons\/updates.svg\";i:22;s:31:\"asset::assets::icons\/videos.svg\";i:23;s:34:\"asset::assets::icons\/wanderers.svg\";i:24;s:29:\"asset::assets::icons\/wiki.svg\";i:25;s:42:\"asset::assets::team-photo-with-unicorn.png\";i:26;s:20:\"asset::aws::404.html\";i:27;s:34:\"asset::aws::archive\/images\/101.jpg\";i:28;s:40:\"asset::aws::archive\/images\/101_cover.jpg\";i:29;s:62:\"asset::aws::archive\/images\/A_Wanderers_Handbook_1024_x_768.jpg\";i:30;s:62:\"asset::aws::archive\/images\/A_Wanderers_Handbook_1152_x_864.jpg\";i:31;s:63:\"asset::aws::archive\/images\/A_Wanderers_Handbook_1280_x_1024.jpg\";i:32;s:63:\"asset::aws::archive\/images\/A_Wanderers_Handbook_1600_x_1200.jpg\";i:33;s:61:\"asset::aws::archive\/images\/A_Wanderers_Handbook_800_x_600.jpg\";i:34;s:57:\"asset::aws::archive\/images\/A_Wanderers_Handbook_image.jpg\";i:35;s:35:\"asset::aws::archive\/images\/Vol1.jpg\";i:36;s:35:\"asset::aws::archive\/images\/Vol2.jpg\";i:37;s:56:\"asset::aws::archive\/images\/a_book_of_days_1024_x_768.jpg\";i:38;s:56:\"asset::aws::archive\/images\/a_book_of_days_1152_x_864.jpg\";i:39;s:57:\"asset::aws::archive\/images\/a_book_of_days_1600_x_1200.jpg\";i:40;s:55:\"asset::aws::archive\/images\/a_book_of_days_800_x_600.jpg\";i:41;s:51:\"asset::aws::archive\/images\/a_book_of_days_cover.jpg\";i:42;s:57:\"asset::aws::archive\/images\/a_book_of_days_cover_image.jpg\";i:43;s:57:\"asset::aws::archive\/images\/a_wanderers_handbook_cover.jpg\";i:44;s:35:\"asset::aws::archive\/images\/abod.jpg\";i:45;s:34:\"asset::aws::archive\/images\/ach.jpg\";i:46;s:39:\"asset::aws::archive\/images\/archives.jpg\";i:47;s:34:\"asset::aws::archive\/images\/awh.jpg\";i:48;s:35:\"asset::aws::archive\/images\/crux.jpg\";i:49;s:46:\"asset::aws::archive\/images\/dialogues_cover.jpg\";i:50;s:56:\"asset::aws::archive\/images\/law_of_one_book_one_cover.jpg\";i:51;s:41:\"asset::aws::archive\/images\/lightlines.jpg\";i:52;s:67:\"asset::aws::archive\/images\/living_the_law_of_one_101_1024_x_768.jpg\";i:53;s:68:\"asset::aws::archive\/images\/living_the_law_of_one_101_1600_x_1200.jpg\";i:54;s:66:\"asset::aws::archive\/images\/living_the_law_of_one_101_800_x_600.jpg\";i:55;s:62:\"asset::aws::archive\/images\/living_the_law_of_one_101_cover.jpg\";i:56;s:64:\"asset::aws::archive\/images\/living_the_law_of_one_101_desktop.jpg\";i:57;s:58:\"asset::aws::archive\/images\/living_the_law_of_one_cover.jpg\";i:58;s:34:\"asset::aws::archive\/images\/loo.jpg\";i:59;s:35:\"asset::aws::archive\/images\/loo1.jpg\";i:60;s:35:\"asset::aws::archive\/images\/loo2.png\";i:61;s:35:\"asset::aws::archive\/images\/loo3.png\";i:62;s:35:\"asset::aws::archive\/images\/loo4.png\";i:63;s:35:\"asset::aws::archive\/images\/loo5.png\";i:64;s:38:\"asset::aws::archive\/images\/mosaics.jpg\";i:65;s:37:\"asset::aws::archive\/images\/poetry.jpg\";i:66;s:67:\"asset::aws::archive\/images\/pub_light_lines_first_25_years_cover.jpg\";i:67;s:61:\"asset::aws::archive\/images\/pub_the_alphabet_mosaics_cover.jpg\";i:68;s:39:\"asset::aws::archive\/images\/quixotic.jpg\";i:69;s:33:\"asset::aws::archive\/images\/rc.png\";i:70;s:34:\"asset::aws::archive\/images\/rc1.png\";i:71;s:41:\"asset::aws::archive\/images\/rc1_german.png\";i:72;s:40:\"asset::aws::archive\/images\/rc1french.png\";i:73;s:34:\"asset::aws::archive\/images\/rc2.png\";i:74;s:41:\"asset::aws::archive\/images\/rc2_german.png\";i:75;s:42:\"asset::aws::archive\/images\/rc_homepage.png\";i:76;s:38:\"asset::aws::archive\/images\/secrets.jpg\";i:77;s:66:\"asset::aws::archive\/images\/the_ll_research_channeling_archives.jpg\";i:78;s:73:\"asset::aws::archive\/images\/the_poetry_of_carla_lisbeth_rueckert_cover.jpg\";i:79;s:38:\"asset::aws::archive\/images\/tilting.jpg\";i:80;s:44:\"asset::aws::archive\/images\/tilting_cover.jpg\";i:81;s:37:\"asset::aws::archive\/images\/voices.png\";i:82;s:37:\"asset::aws::archive\/images\/voices.psd\";i:83;s:43:\"asset::aws::archive\/images\/voices_cover.jpg\";i:84;s:41:\"asset::aws::archive\/images\/whatislove.jpg\";i:85;s:34:\"asset::aws::audio\/101\/lesson_1.mp3\";i:86;s:35:\"asset::aws::audio\/101\/lesson_10.mp3\";i:87;s:35:\"asset::aws::audio\/101\/lesson_11.mp3\";i:88;s:35:\"asset::aws::audio\/101\/lesson_12.mp3\";i:89;s:34:\"asset::aws::audio\/101\/lesson_2.mp3\";i:90;s:34:\"asset::aws::audio\/101\/lesson_3.mp3\";i:91;s:34:\"asset::aws::audio\/101\/lesson_4.mp3\";i:92;s:34:\"asset::aws::audio\/101\/lesson_5.mp3\";i:93;s:34:\"asset::aws::audio\/101\/lesson_6.mp3\";i:94;s:34:\"asset::aws::audio\/101\/lesson_7.mp3\";i:95;s:34:\"asset::aws::audio\/101\/lesson_8.mp3\";i:96;s:34:\"asset::aws::audio\/101\/lesson_9.mp3\";i:97;s:27:\"asset::aws::audio\/jenny.mp3\";i:98;s:44:\"asset::aws::audio\/meditations\/1979.07.11.mp3\";i:99;s:44:\"asset::aws::audio\/meditations\/1981.05.17.mp3\";}s:28:\"\u0000*\u0000escapeWhenCastingToString\";b:0;}}","batchId":null},"createdAt":1771917542,"delay":null}
 attempts = 0
 reserved_at =
 available_at = 1771917542
 created_at = 1771917542

But that's all I can figure out so far, I'm not sure what the problem is yet and I thought because it's costing so much I thought maybe someone else has some ideas on how to debug it or what to investigate more.

I do know there's an issue on Statamic v6 where I can't list my assets on S3 because it just times out? So maybe it's related too? I was able to make some progress on that one by importing the assets into eloquent on my production server, as we have a few thousand PDFs in there. But for the test website we've still been using the file driver instead of eloquent

It just seems to timeout with a 504 error on the assets page, so if it was something related to the queue and it was timing out too or getting a 504 error then maybe the job is being replayed over and over?

I have no clue... any other ideas or things to try?

How to reproduce

I wish I knew

Logs

Environment

Environment
Laravel Version: 12.52.0
PHP Version: 8.4.16
Composer Version: 2.9.5
Environment: testing
Debug Mode: ENABLED
Maintenance Mode: OFF
Timezone: America/Kentucky/Louisville
Locale: en

Cache
Config: NOT CACHED
Events: CACHED
Routes: CACHED
Views: CACHED

Drivers
Broadcasting: pusher
Cache: database
Database: app
Logs: stack / single
Mail: log
Queue: database
Session: database

Storage
public/storage: LINKED

Livewire
Livewire: v4.1.4

Statamic
Addons: 2
Sites: 28 (English, Arabic, Bulgarian, and 25 more)
Stache Watcher: Enabled
Static Caching: Disabled
Version: 6.3.3 PRO

Statamic Addons
mikomagni/statamic-environment: 6.x-dev
statamic/eloquent-driver: 5.2.0

Statamic Eloquent Driver
Addon Settings: file
Asset Containers: file
Assets: file
Blueprints: file
Collection Trees: file
Collections: file
Entries: eloquent
Fieldsets: file
Form Submissions: file
Forms: file
Global Sets: file
Global Variables: file
Navigation Trees: file
Navigations: file
Revisions: eloquent
Sites: file
Taxonomies: file
Terms: file
Tokens: file

Installation

Existing Laravel app

Additional details

I will let you know if I figure anything else out.

[o1y]

so if it was something related to the queue and it was timing out too or getting a 504 error then maybe the job is being replayed over and over?

Are you using assets:* or content in config/statamic/search.php and are you updating the search index regularly?

If you don’t have a custom cache store for assets configured, running cache:clear will wipe the asset meta cache.

When you then update the search index, this dispatches the InsertMultipleJob (introduced in #13126). If no asset meta files are present this will download all assets from your AWS container to re-generate the meta files. If the job times out for some reason, each retry repeats these expensive S3 calls. I also see that you probably run the queue:work without a limit (e.g. --tries=3). This could result in a massive number of repeated requests ..

Just a first thought.. maybe this helps you rule out a few things?

[tao]

The default search driver does have content

'default' => [
    'driver' => 'local',
    'searchables' => 'content',
    'fields' => ['title'],
]

I see from the upgrade guide that content includes assets by default:

Previously, you could set 'searchables' => 'all' on a search index to include entries, terms, assets, users and anything provided by custom searchables.

However, in v6, to split out search between the frontend and the Control Panel, support for 'searchable' => 'all' has been removed.

You can now either use 'searchables' => 'content' - which includes entries, terms and assets (not users)

So that could something that changed during the upgrade to v6 and why assets are in the search job, but I don't know if that specifically is the issue. If that is the cause then there also must be some reason why it's failing and continuously reading or writing to my assets bucket.

I also have a rule on AWS to replicate the production assets bucket to the dev bucket every day so that the assets are in sync, and the production content is synced every day with the CI and a daily deploy. So I don't think it's caused by some kind of mismatch where it tries to search and find assets that are in the production content but not in the test assets bucket.

But the CI does deploy the site every day to sync the new Statamic content and run php please search:update --all but 12 million requests and 2.8tb of data / 21 days since the spike of traffic.

There are a couple of days in between the traffic spike where I might have pause deployments or turned off the queue while doing updates.

But on average with 17 days of (the queue running?) high traffic, it's still 0.7 million requests a day and 165 GB per day, while the total bucket size is 20GB so it's reading the entire bucket ~ 8.25 times per day.

Since the queue has been turned off the traffic spike has stopped though.... but I am also not sure why it's on the dev website and not the production website, which deploys updates from the CI more frequently and has the same php artisan search:update --all command run on each deployment. 😕 🤔

If you don’t have a custom cache store for assets configured, running cache:clear will wipe the asset meta cache.

It is possible that I have php artisan cache:clear in the Forge deploy script in a previous version when I set up the test website for v6, but I can't recall and it's not in the current version. So maybe when I setup the test website it wiped the asset meta data and has been trying to rebuild it since then with some queue jobs? I don't know the details well enough this is outside of my understanding of how things work.

When you then update the search index, this dispatches the InsertMultipleJob (introduced in #13126). If no asset meta files are present this will download all assets from your AWS container to re-generate the meta files. If the job times out for some reason, each retry repeats these expensive S3 calls. I also see that you probably run the queue:work without a limit (e.g. --tries=3). This could result in a massive number of repeated requests ..

I used the default queue worker on Forge, so it might not have included --tries by default, although it sounds like a sensible default to include.

If the job times out for some reason, each retry repeats these expensive S3 calls.

So it could be related to how the assets page on the CMS is not loading when you have a lot of assets, and it's having trouble so it times out and retries... and it has been trying to read the entire bucket 8 times a day?

I think the default queue worker on forge runs this command php8.4 /home/forge/test-website/artisan queue:work 'database' --sleep=3 --daemon --quiet

But it's still not really clear why this only happens on the test website and not the production CMS as the setup on forge is nearly identical, and I only imported my assets to eloquent last week, so the production CMS has also been running the queue and the scheduler without causing a traffic spike.

[o1y]

Do you have any logs we can review to understand why the jobs are failing or hitting a timeout? Before restarting the queue worker, are you clearing all pending jobs with php artisan queue:clear? It would be helpful to know whether a redeploy triggers the same jobs to be dispatched again.

[tao]

I have been trying to debug it a bit more. I cleared all the queues and cache on the test website and set up a new queue worker:

php8.4 /home/forge/site/artisan queue:work 'database' --sleep=3 --daemon --quiet --timeout=60 --tries=1 --verbose

So I've added the verbose flag and the tries=1 flag to stop the jobs from repeating, and the first job failed after timing out:

php artisan queue:listen

   INFO  Processing jobs from the [default] queue.

  2026-02-25 04:52:23 Statamic\Search\InsertMultipleJob .............. RUNNING

In Process.php line 1196:

  The process "'/usr/bin/php8.4' 'artisan' 'queue:work' '--once' '--name=default' '--queue=default' '--backoff=0' '--memory=128' '--sleep=3' '--tries=1'" exceeded the timeout of 60 seconds.

And the log says the same thing that it timed out:

[2026-02-25 04:53:22] testing.ERROR: The process "'/usr/bin/php8.4' 'artisan' 'queue:work' '--once' '--name=default' '--queue=default' '--backoff=0' '--memory=128' '--sleep=3' '--tries=1'" exceeded the timeout of 60 seconds. {"exception":"[object] (Symfony\\Component\\Process\\Exception\\ProcessTimedOutException(code: 0): The process \"'/usr/bin/php8.4' 'artisan' 'queue:work' '--once' '--name=default' '--queue=default' '--backoff=0' '--memory=128' '--sleep=3' '--tries=1'\" exceeded the timeout of 60 seconds. at /home/forge/site/vendor/symfony/process/Process.php:1196)

[2026-02-25 04:54:23] testing.ERROR: Statamic\Search\InsertMultipleJob has been attempted too many times. {"exception":"[object] (Illuminate\\Queue\\MaxAttemptsExceededException(code: 0): Statamic\\Search\\InsertMultipleJob has been attempted too many times. at /home/forge/llresearch-test.on-forge.com/vendor/laravel/framework/src/Illuminate/Queue/MaxAttemptsExceededException.php:24)

And the queue payload for the failed job includes assets in it:

{"uuid":"12b379c9-6735-44e0-8d37-5df4369ce813","displayName":"Statamic\\Search\\InsertMultipleJob","job":"Illuminate\\Queue\\CallQueuedHandler@call","maxTries":null,"maxExceptions":null,"failOnTimeout":false,"backoff":null,"timeout":null,"retryUntil":null,"data":{"commandName":"Statamic\\Search\\InsertMultipleJob","command":"O:33:\"Statamic\\Search\\InsertMultipleJob\":3:{s:4:\"name\";s:7:\"default\";s:6:\"locale\";N;s:9:\"documents\";O:31:\"Statamic\\Assets\\AssetCollection\":2:{s:8:\"\u0000*\u0000items\";a:100:{i:0;s:30:\"asset::assets::don-quixote.svg\";i:1;s:30:\"asset::assets::icons\/about.svg\";

(...)

\"asset::aws::audio\/101\/lesson_6.mp3\";i:94;s:34:\"asset::aws::audio\/101\/lesson_7.mp3\";i:95;s:34:\"asset::aws::audio\/101\/lesson_8.mp3\";i:96;s:34:\"asset::aws::audio\/101\/lesson_9.mp3\";i:97;s:27:\"asset::aws::audio\/jenny.mp3\";i:98;s:43:\"asset::aws::audio\/messages_from_the_ufo.mp3\";i:99;s:37:\"asset::aws::audio\/this_is_the_day.mp3\";}s:28:\"\u0000*\u0000escapeWhenCastingToString\";b:0;}}"

A lot of images and mp3 and things... the payload was quite large.

Then I tried running a queue job manually from the CLI with --timeout=0 and --tries=1

php artisan queue:work --timeout=0 --tries=1 --verbose

   INFO  Processing jobs from the [default] queue.

  2026-02-25 05:10:50 Statamic\Search\InsertMultipleJob 10713 database default .................. RUNNING
  2026-02-25 05:16:16 Statamic\Search\InsertMultipleJob 10713 database default .................. 5m 26s DONE

The job took 5 minutes but it did succeed so there were no logs or anything so it also doesn't tell me much about what it might be doing if it's interacting with the S3 bucket. So I need to see what else I can figure out next.

Maybe if the job is taking 5 minutes then its trying to download all those mp3 and images to generate meta-data about them 🤔

[tao]

Okay, so I tried setting up the background process again on Forge, and if you don't specify a timeout then it defaults to 60 seconds.

So what I think might be happening is this:

1. php artisan search:update --all command tries to update default search index with content and that includes assets
2. 100 assets are included in each job in the batch when running InsertMultipleJob command.
3. The queue worker tries to download the assets to create meta data or figure something out???
4. It tries to download a bunch of MP3s and Podcasts and stuff?
5. The job times out after 60 seconds (but it needs 5 minutes on average to run)
6. The failed job goes back on the queue and tries to run again
7. The job repeatedly tries to download files and times out again and again without a max tries set.

[o1y]

It tries to download a bunch of MP3s and Podcasts and stuff?

Only if for these files no .meta folders are available in your bucket. If they don't exist, this downloads the full file. You could check with: aws s3 ls s3://your-bucket-foo/audio/101/.meta/

Also I would probably set the STATAMIC_STACHE_WATCHER=false. When the stache watcher is enabled this seems to set ttl for the cache to always 0 (= expires immediately) and this triggers many LISTS per job.

cms/src/Assets/AssetContainerContents.php

Lines 300 to 302 in 41850cd

	if (Statamic::isWorker()) {
	$this->cacheStore()->put($this->key(), $files, $this->ttl());
	}

cms/src/Assets/AssetContainerContents.php

Lines 315 to 318 in 41850cd

	private function ttl()
	{
	return Stache::isWatcherEnabled() ? 0 : null;
	}