Allow partial data in tenant-federation

[SungJin1212]

This PR introduces experimental flag -tenant-federation.allow-partial-data for partial responses when using Tenant Federation.

Currently, when querying multiple tenants via federation, a failure in a single tenant causes the entire query to fail. With this change, users can opt-in to a mode where errors from individual tenants are treated as warnings, allowing partial results from successful tenants to be returned.

This is particularly useful when using the regex resolver (e.g., X-Scope-OrgID: user-.*). When dynamically matching a large number of tenants, a single failing tenant will no longer block the entire query result.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[friedrichg]

do we need a warning here?

[friedrichg]

do we need to add a warning ?

[SungJin1212]

@friedrichg

type MetadataQuerier interface {
	MetricsMetadata(ctx context.Context, req *client.MetricsMetadataRequest) ([]scrape.MetricMetadata, error)
}

// ExemplarQuerier provides reading access to time series data.
type ExemplarQuerier interface {
	// Select all the exemplars that match the matchers.
	// Within a single slice of matchers, it is an intersection. Between the slices, it is a union.
	Select(start, end int64, matchers ...[]*labels.Matcher) ([]exemplar.QueryResult, error)
}

I'd love to add a warning here, but the upstream interface doesn't support returning warnings like SeriesSet does. Do we need to log an error at the Warn level?

[friedrichg]

Looking at #6526 I see partial data for metadata was not implemented because prometheus does not support it. Maybe we should do the same here?

Same for exemplars.

@SungJin1212 How does grafana or perses react with errors in these APIs? maybe it just ignores it

[SungJin1212]

@friedrichg
Since we suppress the error, Grafana simply won't receive data for the failed tenants and will display the partial results without any warning.

Yeah, the Prometheus interface doesn't support returning warnings. Should we disable partial data for exemplars and metadata?

[friedrichg]

Ignoring the error is bad at least send a warning to the logs. I feel.