Invalid UTF-8 string handling in metrics.go
Closed, ResolvedPublic1 Estimated Story Points
Actions

Assigned To

Authored By

	RThomas-WMF
	Jun 26 2025, 12:35 PM

Description

Requests with invalid utf-8 can cause panic in the prometheus metrics.go library

goroutine 24 [running]:
github.com/prometheus/client_golang/prometheus.(*GaugeVec).WithLabelValues(...)
	/go/pkg/mod/github.com/prometheus/client_golang@v1.17.0/prometheus/gauge.go:238
wikimedia-enterprise/api/main/submodules/httputil.(*MetricsRecorder).Serve.func1()
	/app/submodules/httputil/metrics.go:233 +0x539
created by wikimedia-enterprise/api/main/submodules/httputil.(*MetricsRecorder).Serve in goroutine 45
	/app/submodules/httputil/metrics.go:229 +0x26

Replace gcx.Request.URL.Path with gcx.FullPath() that is not prone to invalid utf-8 issues and will ease the high cardinality problem since the path labels will have

path="/v2/structured-contents/*name"

Remove the gcx.ClientIP() and usr.GetUsername() from the labels completely as they are the cause of high cardinality.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T398211 [Business as Usual work Q1-Q2 FY 24/26] On Demand API stability improvements
		Resolved		RThomas-WMF	T397919 Invalid UTF-8 string handling in metrics.go

Event Timeline

RThomas-WMF created this task.Jun 26 2025, 12:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 26 2025, 12:35 PM

RThomas-WMF triaged this task as High priority.Jun 26 2025, 12:35 PM

RThomas-WMF raised the priority of this task from High to Needs Triage.

RThomas-WMF renamed this task from Invalid UTF-8 string in the API requests causes Panic to Invalid UTF-8 string handling in metrics.go.Jun 26 2025, 1:34 PM

RThomas-WMF set the point value for this task to 2.Jun 27 2025, 3:37 AM

RThomas-WMF updated the task description. (Show Details)Jun 27 2025, 1:32 PM

JArguello-WMF added a parent task: T398211: [Business as Usual work Q1-Q2 FY 24/26] On Demand API stability improvements.Jun 30 2025, 3:30 PM

RThomas-WMF removed SGupta-WMF as the assignee of this task.Jul 2 2025, 7:12 AM

RThomas-WMF triaged this task as High priority.

RThomas-WMF added a subscriber: SGupta-WMF.

RThomas-WMF changed the task status from Open to In Progress.Jul 2 2025, 7:15 AM

RThomas-WMF claimed this task.

RThomas-WMF moved this task from Next up to Doing on the Wikimedia-Enterprise-Kanban-On-Call board.Jul 2 2025, 11:23 AM

RThomas-WMF changed the point value for this task from 2 to 1.

RThomas-WMF closed this task as Resolved.Jul 3 2025, 7:40 AM