Skip to content

Comments

PRW2:Add bound checking for symbol referencs#7290

Open
SungJin1212 wants to merge 1 commit intocortexproject:masterfrom
SungJin1212:Add-unit/help-ref-bound-checking
Open

PRW2:Add bound checking for symbol referencs#7290
SungJin1212 wants to merge 1 commit intocortexproject:masterfrom
SungJin1212:Add-unit/help-ref-bound-checking

Conversation

@SungJin1212
Copy link
Member

@SungJin1212 SungJin1212 commented Feb 23, 2026

I have found the Distributor gets stuck in panic when UnitRef exceeds the symbols array length in
https://github.com/cortexproject/cortex/blob/master/pkg/util/push/push.go#L220
This PR adds bound checking logic on the server side.

Panic log

Prometheus version: v3.9.1

2026/02/23 00:00:46 http: panic serving 192.168.119.93:60442: runtime error: index out of range [2484] with length 2160
goroutine 10056052 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1943 +0xd3
panic({0x3844be0?, 0xc00daff5a8?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:159 +0x169
panic({0x3844be0?, 0xc00daff5a8?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/cortexproject/cortex/pkg/util/push.convertV2RequestToV1(0xc00c9000c0, 0x0)
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:220 +0xc10
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1.2()
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:114 +0x1e8
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1({0x4139850, 0xc001e8aa80}, 0xc00044a000)
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:192 +0x6f0
net/http.HandlerFunc.ServeHTTP(0xc0006afe00?, {0x4139850?, 0xc001e8aa80?}, 0x413f2c8?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/weaveworks/common/middleware.init.func1.1({0x4139850, 0xc001e8aa80}, 0xc0006afe00)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x108
net/http.HandlerFunc.ServeHTTP(0x0?, {0x4139850?, 0xc001e8aa80?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/cortexproject/cortex/pkg/api.(*API).RegisterRoute.HTTPHeaderMiddleware.Wrap.func1({0x4139850, 0xc001e8aa80}, 0xc0166bf680?)
	/go/src/github.com/cortexproject/cortex/pkg/api/middlewares.go:50 +0x66
net/http.HandlerFunc.ServeHTTP(0xc0006afb80?, {0x4139850?, 0xc001e8aa80?}, 0x2?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0002bb5c0, {0x4139850, 0xc001e8aa80}, 0xc0006af7c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2
github.com/weaveworks/common/middleware.(*Instrument).Wrap.Instrument.Wrap.func1.2({0x4139850?, 0xc001e8aa80?})
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:70 +0x33
github.com/felixge/httpsnoop.(*Metrics).CaptureMetrics(0xc001ba8570, {0x7f081fe2a9b0, 0xc00c900060}, 0xc00098f7f0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:84 +0x1e5
github.com/felixge/httpsnoop.CaptureMetricsFn({0x7f081fe2a9b0, 0xc00c900060}, 0xc00098f7f0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:39 +0x4e
github.com/weaveworks/common/middleware.(*Instrument).Wrap.Instrument.Wrap.func1({0x7f081fe2a9b0, 0xc00c900060}, 0xc0006af7c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:69 +0x28b
net/http.HandlerFunc.ServeHTTP(0x41349b0?, {0x7f081fe2a9b0?, 0xc00c900060?}, 0xc0166bf560?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/weaveworks/common/middleware.(*Log).Wrap.Log.Wrap.func1({0x41349b0, 0xc000476e60}, 0xc0006af7c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:81 +0x209
net/http.HandlerFunc.ServeHTTP(0x41e045?, {0x41349b0?, 0xc000476e60?}, 0x1?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5({0x41326e0, 0xc00544a5a0}, 0xc0006af540)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:163 +0x49d
net/http.HandlerFunc.ServeHTTP(0x41e045?, {0x41326e0?, 0xc00544a5a0?}, 0xc00544a501?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
net/http.serverHandler.ServeHTTP({0x41250c0?}, {0x41326e0?, 0xc00544a5a0?}, 0x1?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc00425d0e0, {0x413f2c8, 0xc000a033b0})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 196
	/usr/local/go/src/net/http/server.go:3493 +0x485
2026/02/23 00:00:46 http: panic serving 192.168.119.93:60438: runtime error: index out of range [2512] with length 2127
goroutine 10056051 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1943 +0xd3
panic({0x3844be0?, 0xc00dafe0c0?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:159 +0x169
panic({0x3844be0?, 0xc00dafe0c0?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/cortexproject/cortex/pkg/util/push.convertV2RequestToV1(0xc0085befc0, 0x0)
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:220 +0xc10
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1.2()
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:114 +0x1e8
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1({0x4139850, 0xc000ef5840}, 0xc00e0f4a00)
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:192 +0x6f0
net/http.HandlerFunc.ServeHTTP(0xc00e0f48c0?, {0x4139850?, 0xc000ef5840?}, 0x413f2c8?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/weaveworks/common/middleware.init.func1.1({0x4139850, 0xc000ef5840}, 0xc00e0f48c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x108
net/http.HandlerFunc.ServeHTTP(0x0?, {0x4139850?, 0xc000ef5840?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/cortexproject/cortex/pkg/api.(*API).RegisterRoute.HTTPHeaderMiddleware.Wrap.func1({0x4139850, 0xc000ef5840}, 0xc000919d40?)
	/go/src/github.com/cortexproject/cortex/pkg/api/middlewares.go:50 +0x66
net/http.HandlerFunc.ServeHTTP(0xc00e0f4640?, {0x4139850?, 0xc000ef5840?}, 0x2?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0002bb5c0, {0x4139850, 0xc000ef5840}, 0xc00e0f4500)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2
github.com/weaveworks/common/middleware.(*Instrument).Wrap.Instrument.Wrap.func1.2({0x4139850?, 0xc000ef5840?})
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:70 +0x33
github.com/felixge/httpsnoop.(*Metrics).CaptureMetrics(0xc0004dbe00, {0x7f081fe2a9b0, 0xc0085bef60}, 0xc0019e77f0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:84 +0x1e5
github.com/felixge/httpsnoop.CaptureMetricsFn({0x7f081fe2a9b0, 0xc0085bef60}, 0xc0019e77f0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:39 +0x4e
github.com/weaveworks/common/middleware.(*Instrument).Wrap.Instrument.Wrap.func1({0x7f081fe2a9b0, 0xc0085bef60}, 0xc00e0f4500)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:69 +0x28b
net/http.HandlerFunc.ServeHTTP(0x41349b0?, {0x7f081fe2a9b0?, 0xc0085bef60?}, 0xc000918d20?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/weaveworks/common/middleware.(*Log).Wrap.Log.Wrap.func1({0x41349b0, 0xc000966820}, 0xc00e0f4500)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:81 +0x209
net/http.HandlerFunc.ServeHTTP(0x41e045?, {0x41349b0?, 0xc000966820?}, 0x1?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5({0x41326e0, 0xc0053f25a0}, 0xc00e0f43c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:163 +0x49d
net/http.HandlerFunc.ServeHTTP(0x41e045?, {0x41326e0?, 0xc0053f25a0?}, 0xc0053f2501?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
net/http.serverHandler.ServeHTTP({0x41250c0?}, {0x41326e0?, 0xc0053f25a0?}, 0x1?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc00425d050, {0x413f2c8, 0xc000a033b0})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 196
	/usr/local/go/src/net/http/server.go:3493 +0x485

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@dosubot
Copy link

dosubot bot commented Feb 23, 2026

Related Documentation

Checked 0 published document(s) in 1 knowledge base(s). No updates required.

How did I do? Any feedback?  Join Discord

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@SungJin1212 SungJin1212 force-pushed the Add-unit/help-ref-bound-checking branch from 1f22d7d to 44c19ef Compare February 23, 2026 00:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant