Proposal: Persist trace context in job metadata

[Wouter12]

Idea

To facilitate better observability and bridge the gap between background job execution and distributed tracing, I propose to add more context to the River job's metadata.

The goal is to allow otelriver to automatically inject the TraceID and SpanID into the River job's metadata field. This ensures that when viewing a job in the River UI, the operator can immediately see the associated Trace ID and navigate to their tracing backend.

Proposed Changes:

1. Add a new configuration option PersistTraceContextInMetadata bool to MiddlewareConfig.
2. Update InsertMany to inject the current TraceID and SpanID into the Metadata map of each JobInsertParams.
3. Update Work to ensure the metadata is updated if the context was started within the worker.

[brandur]

I think something like this could work. Actually, I think the original DD middleware pitched to us may have had this feature, and I removed it for the first pass in favour of simplicity.

In terms of trace versus span: would the insert and all attempts of a job share a span and then the insert and each individual attempt of a job would get a unique span ID?

Do you want to try your hand at this change? I think putting it behind a config option is the right move, but I'd opt for one that's a bit less of a mouthful.

[bgentry]

I'm wondering if it'd be broadly useful to have a single config to always propagate or never propagate the trace context. I would expect it'd be largely situation-dependent whether one wants to do this or not, and that in any given app there could be scenarios where you do want this as well as ones where it'd be harmful.

Maybe something more granular would be appropriate?