[SS-63][Oneshot Sources] Fix race condition in Oneshot Ingestion related to multiple Timely Workers

[patrickwwbutler]

This fixes a race condition in which the first timely worker begins to process an ingestion, fails receives an error that cancels the operation very quickly, before the other timely workers actually render the dataflow and begin working. Then, the now-removed code would drop the ingestion, including the PressOnDrop buttons from that worker. This would then result in the other timely workers waiting for results/progress from the first worker forever.

By removing this early ingestion drop logic from the worker, and allowing the StorageController to drop it from all workers when the ingestion is fully completed, we can avoid this race condition.

Verification

This now passes the copy_from_s3_minio.td test with multiple replicas, and larger cluster sizes.

[github-actions]

Pre-merge checklist

• The PR title is descriptive and will make sense in the git log.
• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).