Page MenuHomePhabricator
Create Task
Maniphest T205870

Fully migrate producers off statsd
Open, MediumPublic
Actions

Description

This task tracks porting statsd metrics traffic to Prometheus. Specifically either porting the applications to use native Prometheus metrics or deploying statsd_exporter to expose Prometheus metrics derived from statsd traffic. The latter approach has been tested successfully for Thumbor in T145867: Test making thumbor statsd metrics available from Prometheus.

An audit can be generated with

timeout 10m ngrep -q -W byline . udp dst port 8125  | grep -v -e '^U ' -e '^$' | cut -f1,2,3 -d.  | pigz -9c > statsd_users_10m.gz

zcat statsd_users_10m.gz | cut -d. -f1 | sort | uniq -dc | sort -rn > top_users_10m

Looking at graphite whisper files mtime a few infrequent statsd producers came up:

deploy
scap
gunicorn

There's a few other producers that go by way of statsd -> statsite (on localhost) -> graphite protocol to graphite1004:

thumbor
swift
zuul

Annotated list of producers above, with plan of action:

statsv-produced metrics, see also T180105

metrics are in Prometheus. grafana dashboards need migrated

  • mw.js.deprecate (generated client-side from mediawiki/extensions/WikimediaEvents)
  • mw.performance
  • browsertime (from WebPageReplay)
  • ve
  • Some metrics in MediaWiki hierarchy, e.g. minerva.WebClientError
  • pagepreviews (top level, PagePreviewsApiFailure/ PagePreviewsApiResponse/ PagePreviewsPreviewShow/)
  • media.thumbnail.client
  • webpagetest (generated by wpt-reporter from Jenkins)
  • wikibase.queryService.ui

navtiming-produced metrics, see also T175087

Prometheus counters for navtiming: https://gerrit.wikimedia.org/r/c/performance/navtiming/+/534771

  • frontend
  • mw.performance.save*
  • eventlogging.client_errors.navigation/paitingtiming
  • performance.survey

TODO

Use global aggregation / percentiles

See also https://wikitech.wikimedia.org/wiki/Prometheus/statsd_k8s for an introduction for service owners on how to write their statsd_exporter mappings (in k8s, but guidelines are generic). Some of the services below use service-runner, for which some statsd metrics will need reconsideration (cfr T222795: Re-evaluate service-runner's (ab)use of statsd timing metric for nodejs GC stats)

Dependent on Service-Runner:

  • aqs PR
  • changeprop/cpjobqueue - scheduled to be moved to k8s PR
  • eventstreams - scheduled to be moved to k8s PR
  • eventgate
  • kartotherian, tilerator, tileratorui -- PR
  • mobileapps - some parts are moving to k8s PR
  • service-template-node PR
  • proton PR
  • recommendation-api PR
  • restbase - (parts?) moving to k8s PR
  • hyperswitch PR
  • citoid -- PR
  • mathoid -- PR

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ext.eventLogging: Pass the value of client_errors_total to mw.trackDLynchmediawiki/extensions/EventLoggingmaster+15 -1
Migrate eventlogging.client_errors to PrometheusAndrea Denissemediawiki/extensions/EventLoggingmaster+1 -0
mediawiki: enable statsd_exporter and add matching rules to appserverCwhiteoperations/puppetproduction+915 -0
varnish: enable statsd_exporter and add matching rulesCwhiteoperations/puppetproduction+164 -0
ci: clean up statsite includesCwhiteoperations/puppetproduction+0 -3
ci: uninstall statsite from ci hostsCwhiteoperations/puppetproduction+1 -0
Migrate growthExperiments.StructuredTask.noSuggestionsDialog to PrometheusAndrea Denissemediawiki/extensions/GrowthExperimentsmaster+1 -0
services: update the rec-api's Docker imageElukeyoperations/deployment-chartsmaster+2 -4
Upgrade to service-runner 3.1.0Elukeymediawiki/services/recommendation-apimaster+3 K -8 K
services: update rec-api's staging Docker imageElukeyoperations/deployment-chartsmaster+1 -1
services: update Docker image and settings for Recommendation APIElukeyoperations/deployment-chartsmaster+2 -7
recommendation-api: update monitoring configElukeyoperations/deployment-chartsmaster+29 -5
Move the only metric produced from Gauge to HistogramElukeymediawiki/services/recommendation-apimaster+4 -3
Use "set" instead of "endTiming" in makeMetricElukeymediawiki/services/recommendation-apimaster+1 -1
update service-runner to 2.8.0 and implement new metrics apiCwhitemediawiki/services/recommendation-apimaster+15 -11
scb: enable statsd_exporter and add matching rulesCwhiteoperations/puppetproduction+37 -0
citoid: switch to native prometheus metricsPipelineBotoperations/deployment-chartsmaster+8 -83
Move remaining metrics to prometheusMvolzmediawiki/services/citoidmaster+85 -22
Update to service-template-node 0.10.0.Cwhitemediawiki/services/citoidmaster+24 -16
Metrics: Wire up MetricsFactory into ServiceWiring and emit stepsCwhitemediawiki/coremaster+60 -0
Metrics: Wire up MetricsFactory into ServiceWiringCwhitemediawiki/coremaster+22 -0
Metrics: Perform MetricsFactory->flush() in emitBufferedStatsdData()Cwhitemediawiki/coremaster+12 -2
Metrics: Implement statsd-exporter compatible Metrics interfaceCwhitemediawiki/coremaster+1 K -0
Metrics: Implement and enable statsd-exporter compatible Metrics interfaceCwhitemediawiki/coremaster+996 -7
Switch to native prometheus latency histogramsAlexandros Kosiarismediawiki/services/mathoidmaster+17 -11
mathoid: Bump deployed versionAlexandros Kosiarisoperations/deployment-chartsmaster+1 -1
update service-runner dependency to 2.8.0 and implement new metrics apiCwhitemediawiki/services/chromium-rendermaster+85 -26
update service-runner to 2.8.0 and implement new metrics apiCwhitemediawiki/services/mobileappsmaster+27 -17
logstash: output webrequest 5xx metricsFilippo Giunchedioperations/puppetproduction+8 -0
profile: disable statsd_exporter relay for oresCwhiteoperations/puppetproduction+3 -1
Use new service-runner metrics for built in prometheus metricsCwhitemediawiki/services/eventstreamsmaster+46 -219
lvs, monitoring: prometheus expects params value as string[] typeCwhiteoperations/puppetproduction+16 -16
monitoring, profile, prometheus: bugfix, prometheus params valuesCwhiteoperations/puppetproduction+26 -18
lvs, monitoring, prometheus: bugfix openapi exportsCwhiteoperations/puppetproduction+13 -9
lvs, prometheus, profile: add blackbox job helper and enable openapi scrapesCwhiteoperations/puppetproduction+211 -83
scb: add graphoid matching rules and deploy statsd exporter to scb clusterCwhiteoperations/puppetproduction+59 -0
update service-runner to 2.8.0 and hyperswitch to 0.14.0Cwhiteanalytics/aqsmaster+2 -2
proton: enable statsd_exporter and add matching rules to profile::protonCwhiteoperations/puppetproduction+79 -0
hiera: update ores to pass statsd through the statsd exporterCwhiteoperations/puppetproduction+2 -0
swift: stop relaying to statsd/statsiteFilippo Giunchedioperations/puppetproduction+0 -0
hiera: update ores to pass statsd through statsd_exporterCwhiteoperations/puppetproduction+2 -0
profile, prometheus, role: install swagger exporter on prometheus nodesCwhiteoperations/puppetproduction+16 -1
logstash: stop relaying to central statsdFilippo Giunchedioperations/puppetproduction+3 -1
hiera: disable statsd_exporter::relay_address on logstash nodesCwhiteoperations/puppetproduction+4 -2
profile: use prometheus for logstash alertingCwhiteoperations/puppetproduction+29 -29
prometheus: make statsd.relay-address toggle-ableCwhiteoperations/puppetproduction+7 -2
thumbor: stop relaying to statsd/statsiteFilippo Giunchedioperations/puppetproduction+6 -2
swift: remove statsiteFilippo Giunchedioperations/puppetproduction+3 -0
swift: stop relaying to statsd/statsiteFilippo Giunchedioperations/puppetproduction+2 -2
swift: port alerts to PrometheusFilippo Giunchedioperations/puppetproduction+64 -64
grafana: use Prometheus swift metrics for dashboardFilippo Giunchedioperations/puppetproduction+89 -94
prometheus: collect swift account/container stats globallyFilippo Giunchedioperations/puppetproduction+3 -0
hiera: fix statsd rulesCwhiteoperations/puppetproduction+9 -9
logstash: update statsd exporter mappings and use exporterCwhiteoperations/puppetproduction+22 -18
add statsd_exporter config to mathoidCwhiteoperations/deployment-chartsmaster+20 -0
profile: enable statsd_exporter and add matching rules to logstash::collectorCwhiteoperations/puppetproduction+11 -0
profile: enable statsd_exporter and add matching rules to ores::workerCwhiteoperations/puppetproduction+204 -0
ci: use statsite for localhost statsd aggregationFilippo Giunchedioperations/puppetproduction+3 -2
statsite: move from role to profileFilippo Giunchedioperations/puppetproduction+6 -6
hieradata: send periodic swift stats to localhostFilippo Giunchedioperations/puppetproduction+3 -0
hieradata: switch all swift statsd traffic to statsd_exporterFilippo Giunchedioperations/puppetproduction+3 -0
swift: add statsd mappings for periodic metricsFilippo Giunchedioperations/puppetproduction+42 -0
swift: turn on statsd_exporter in eqiadFilippo Giunchedioperations/puppetproduction+2 -0
swift: turn on statsd_exporter in codfwFilippo Giunchedioperations/puppetproduction+2 -0
thumbor: relay statsd_exporter metrics to localhostFilippo Giunchedioperations/puppetproduction+3 -1
swift: add statsd_port parameterFilippo Giunchedioperations/puppetproduction+17 -10
hieradata: add statsd_exporter mappings for swift::storageFilippo Giunchedioperations/puppetproduction+187 -2
hieradata: rename swift proxy statsd_exporter mappingFilippo Giunchedioperations/puppetproduction+28 -28
swift: set statsd_exporter to relay to local statsdFilippo Giunchedioperations/puppetproduction+6 -2
hieradata: add statsd_exporter mappings for swift-proxyFilippo Giunchedioperations/puppetproduction+114 -2
swift: enable statsd_exporterFilippo Giunchedioperations/puppetproduction+2 -0
prometheus: set defaults for statsd_exporterFilippo Giunchedioperations/puppetproduction+14 -1
prometheus: add jobs for statsd_exporterFilippo Giunchedioperations/puppetproduction+18 -1
thumbor: add missing statsd_exporter mappingsFilippo Giunchedioperations/puppetproduction+9 -0
thumbor: use statsd_exporterFilippo Giunchedioperations/puppetproduction+3 -0
thumbor: add prometheus-statsd-exporterFilippo Giunchedioperations/puppetproduction+96 -0
thumbor: set name for all statsd_exporter metricsFilippo Giunchedioperations/puppetproduction+1 -0
thumbor: fix missing statsd_exporter mappingsFilippo Giunchedioperations/puppetproduction+8 -0
statsd_exporter: fix commandline flagsFilippo Giunchedioperations/puppetproduction+1 -1
New class: prometheus::statsd_exporterFilippo Giunchedioperations/puppetproduction+76 -0
debian: add patch for inline udp usageFilippo Giunchedioperations/debs/prometheus-statsd-exportermaster+75 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT228380 Tech debt: sunsetting of Graphite
 OpenNoneT205870 Fully migrate producers off statsd
 ResolvedcolewhiteT233089 Export zuul metrics to Prometheus
 Resolved• ACrazeT233448 Review prometheus ORES rules for completeness
 DeclinedcolewhiteT239833 StatsD Exporter drops relayed metrics
 ResolvedcolewhiteT240685 MediaWiki Prometheus support
 ResolvedcolewhiteT249164 RFC: Better interface for generating metrics in MediaWiki
 ResolvedKrinkleT292311 Create project tag for MediaWiki-libs-Metrics
 ResolvedKrinkleT292269 Decouple Profiler class from WebRequest and RequestContext
 ResolvedKrinkleT344748 MediaWiki Core - Review and merge StatsLib patch
 ResolvedherronT344751 Decide on default histogram buckets for MediaWiki timers
 OpenNoneT240995 AQS is not OpenAPI 3 compliant
 Resolved• PcheloloT241176 Review and release service-runner 2.8.0
 ResolvedcolewhiteT247820 Decide on `service-runner` aggregated prometheus metrics and use of `service` label
 ResolvedJgiannelosT277857 Proton metrics broken
 ResolvedKrinkleT175087 Create a navtiming processor for Prometheus
 DeclinedNoneT190936 navtiming.py: When processing metrics, include effectiveConnectionType as a factor
 ResolvedPeterT323124 Replace navtiming Platform tag ("site") with mw_skin
 ResolvedKrinkleT323129 Simulate client dispatch in a single scrape
 OpenNoneT321398 Move performance metrics from Graphite to Prometheus
 OpenNoneT325282 Update navtiming Grafana alerts to use metrics from Prometheus
 ResolvedPeterT325283 Update navtiming dashboards to use Prometheus metrics
 OpenNoneT325284 Update documentation to use Prometheus instead of Graphite
 InvalidNoneT293761 statsd and gunicorn metrics for superset

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Sep 24 2021, 6:41 PM

Change 556420 merged by jenkins-bot:

[mediawiki/services/citoid@master] Update to service-template-node 0.10.0.

https://gerrit.wikimedia.org/r/556420

gerritbot added a comment.Sep 27 2021, 5:06 PM

Change 724129 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/core@master] pass MetricsFactory instance to emitBufferedStatsdData in MWLBFactory

https://gerrit.wikimedia.org/r/724129

• lmata edited projects, added: SRE Observability (FY2021/2022-Q2); removed: SRE Observability (FY2021/2022-Q1).Sep 28 2021, 1:46 PM
gerritbot added a comment.Sep 30 2021, 7:01 PM

Change 721626 merged by jenkins-bot:

[mediawiki/core@master] Metrics: Implement statsd-exporter compatible Metrics interface

https://gerrit.wikimedia.org/r/721626

gerritbot added a comment.Sep 30 2021, 7:13 PM

Change 585032 abandoned by Cwhite:

[mediawiki/core@master] Metrics: Implement and enable statsd-exporter compatible Metrics interface

Reason:

in favor of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/721626/

https://gerrit.wikimedia.org/r/585032

• lmata added a project: Goal.Sep 30 2021, 8:48 PM
• lmata moved this task from Inbox to Up next on the SRE Observability (FY2021/2022-Q2) board.
gerritbot added a comment.Sep 30 2021, 10:43 PM

Change 721629 abandoned by Cwhite:

[mediawiki/core@master] Metrics: Perform MetricsFactory->flush() in emitBufferedStatsdData()

Reason:

in favor of I46f0a09f4dab38fa4c9495aa2da9ecab60376ca7

https://gerrit.wikimedia.org/r/721629

gerritbot added a comment.Sep 30 2021, 10:44 PM

Change 721628 abandoned by Cwhite:

[mediawiki/core@master] Metrics: Wire up MetricsFactory into ServiceWiring

Reason:

in favor of I46f0a09f4dab38fa4c9495aa2da9ecab60376ca7

https://gerrit.wikimedia.org/r/721628

gerritbot added a comment.Oct 7 2021, 3:12 PM

Change 721627 merged by jenkins-bot:

[mediawiki/core@master] Metrics: Wire up MetricsFactory into ServiceWiring and emit steps

https://gerrit.wikimedia.org/r/721627

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.4; 2021-10-12).Oct 7 2021, 4:00 PM
fgiunchedi added a subtask: T175087: Create a navtiming processor for Prometheus.Oct 19 2021, 12:16 PM
• dpifke subscribed.Oct 19 2021, 3:29 PM
Ottomata mentioned this in T222795: Re-evaluate service-runner's (ab)use of statsd timing metric for nodejs GC stats.Dec 17 2021, 7:21 PM
• lmata edited projects, added: SRE Observability (FY2021/2022-Q3); removed: SRE Observability (FY2021/2022-Q2).Jan 12 2022, 9:43 PM
gerritbot added a comment.Mar 9 2022, 2:26 PM

Change 767180 had a related patch set uploaded (by Mvolz; author: Mvolz):

[mediawiki/services/citoid@master] Move remaining metrics to prometheus

https://gerrit.wikimedia.org/r/767180

gerritbot added a comment.Apr 1 2022, 4:21 PM

Change 767180 merged by jenkins-bot:

[mediawiki/services/citoid@master] Move remaining metrics to prometheus

https://gerrit.wikimedia.org/r/767180

gerritbot added a comment.Apr 6 2022, 9:56 AM

Change 776233 had a related patch set uploaded (by Mvolz; author: PipelineBot):

[operations/deployment-charts@master] citoid: switch to native prometheus metrics

https://gerrit.wikimedia.org/r/776233

gerritbot added a comment.Apr 7 2022, 10:20 AM

Change 776233 merged by jenkins-bot:

[operations/deployment-charts@master] citoid: switch to native prometheus metrics

https://gerrit.wikimedia.org/r/776233

Mvolz updated the task description. (Show Details)Apr 7 2022, 11:22 AM
Mvolz added a comment.EditedApr 7 2022, 11:25 AM

This is now deployed for citoid.

I have updated grafana for the most part, however there are a few (minor) metrics this broke which relied on the service-runner native ones; quantiles by status and method are broken, but quantiles overall are still working. I'm not sure how to fix them but I'm not sure how essential those are since it's the same info but just broken down a bit.

Garbage collection metrics have been broken for a while, as have memory pod metrics, and that's not related to this change.

https://grafana-rw.wikimedia.org/d/NJkCVermz/citoid

fgiunchedi added a comment.Apr 7 2022, 12:38 PM
In T205870#7837817, @Mvolz wrote:

This is now deployed for citoid.

This is great to see! Thanks for your help @Mvolz

I have updated grafana for the most part, however there are a few (minor) metrics this broke which relied on the service-runner native ones; quantiles by status and method are broken, but quantiles overall are still working. I'm not sure how to fix them but I'm not sure how essential those are since it's the same info but just broken down a bit.

Garbage collection metrics have been broken for a while, as have memory pod metrics, and that's not related to this change.

ack, please feel free to contact us (SRE o11y) for assistance with the missing/broken metrics if needed

Mvolz added a comment.EditedApr 7 2022, 4:31 PM
In T205870#7838013, @fgiunchedi wrote:
In T205870#7837817, @Mvolz wrote:

This is now deployed for citoid.

This is great to see! Thanks for your help @Mvolz

I have updated grafana for the most part, however there are a few (minor) metrics this broke which relied on the service-runner native ones; quantiles by status and method are broken, but quantiles overall are still working. I'm not sure how to fix them but I'm not sure how essential those are since it's the same info but just broken down a bit.

Garbage collection metrics have been broken for a while, as have memory pod metrics, and that's not related to this change.

ack, please feel free to contact us (SRE o11y) for assistance with the missing/broken metrics if needed

I looked into a bit ago and didn't make any progress, and I'm not going to be able to look at it in the next two weeks either due to going away so if you'd like to have a look, be my guest! GC is broken for mathoid too, a bunch of zotero metrics also don't work (but not sure they ever did as it's not really tooled very well?)

colewhite added a comment.Apr 7 2022, 5:41 PM
In T205870#7838787, @Mvolz wrote:

I looked into a bit ago and didn't make any progress, and I'm not going to be able to look at it in the next two weeks either due to going away so if you'd like to have a look, be my guest! GC is broken for mathoid too, a bunch of zotero metrics also don't work (but not sure they ever did as it's not really tooled very well?)

GC metrics were removed in service-runner 2.9.0

In T205870#7837817, @Mvolz wrote:

I have updated grafana for the most part, however there are a few (minor) metrics this broke which relied on the service-runner native ones; quantiles by status and method are broken, but quantiles overall are still working. I'm not sure how to fix them but I'm not sure how essential those are since it's the same info but just broken down a bit.

I took a stab at fixing citoid quantiles by (method, endpoint, status) as well as total memory, top 5 pods memory, and traffic by http status. Please have a look to see if they're fixed in a way you would expect. If something seems amiss, please feel free to make any modification you deem appropriate.

• lmata edited projects, added: SRE Observability (FY2021/2022-Q4); removed: SRE Observability (FY2021/2022-Q3).Apr 11 2022, 1:00 PM
colewhite edited projects, added: SRE Observability (FY2022/2023-Q1); removed: SRE Observability (FY2021/2022-Q4).Jul 1 2022, 10:59 PM
fgiunchedi added a comment.Aug 18 2022, 1:13 PM

A quick update on "high frequency" statsd producers sampled over 10 minutes on graphite1004. The list is getting shorter and shorter and that's great to see!

546061489 MediaWiki
35414078 restbase
1333791 aqs
 832382 frontend
 182788 kartotherian
   7935 ve
   4908 wikibase
   3380 service_checker
   3185 restbase-dev
   3013 mw
   2061 tilerator
   1915 performance
   1620 Vector
   1554 growthExperiments
    297 tileratorui
    120 gunicorn
     59 Wikidata
     16 eventlogging
     14 cloudvps
fgiunchedi mentioned this in T320620: Port openapi/swagger checks/alerts to Prometheus.Oct 12 2022, 12:01 PM
• lmata edited projects, added: Observability-Metrics; removed: SRE Observability (FY2022/2023-Q1).Nov 7 2022, 12:38 PM
• lmata moved this task from Inbox to Prioritized on the Observability-Metrics board.Jan 16 2023, 5:35 PM
gerritbot added a comment.Apr 14 2023, 9:27 PM

Change 484586 abandoned by Cwhite:

[operations/puppet@production] scb: enable statsd_exporter and add matching rules

Reason:

https://gerrit.wikimedia.org/r/484586

Krinkle removed a project: Performance-Team (Radar).Aug 18 2023, 8:14 PM
gerritbot added a comment.Nov 29 2023, 3:28 PM

Change 558184 merged by jenkins-bot:

[mediawiki/services/recommendation-api@master] update service-runner to 2.8.0 and implement new metrics api

https://gerrit.wikimedia.org/r/558184

elukey mentioned this in rMSRA0ec1466aad8d: update service-runner to 2.8.0 and implement new metrics api.Nov 29 2023, 6:22 PM
elukey subscribed.Dec 9 2023, 9:13 AM

Hi @colewhite! I worked with James to port the Recommendation-api to nodejs 18, and one of the patches that we merged is:

https://gerrit.wikimedia.org/r/c/mediawiki/services/recommendation-api/+/558184

When we deploy the new code, I see this error (and no metrics reported):

{"name":"recommendation-api","hostname":"recommendation-api-production-5459988bb6-g4n7q","pid":17,"level":"ERROR","levelPath":"error/metrics","msg":"endTiming() unsupported for metric type Gauge","time":"2023-12-05T15:18:14.563Z","v":0}

We rolled back, but I am wondering if you have more info (I have limited knowledge about what service-runner does behind the scenes). If we solve this problem we'll be able to deploy anytime :)

gerritbot added a comment.Dec 9 2023, 12:39 PM

Change 981645 had a related patch set uploaded (by Elukey; author: Elukey):

[mediawiki/services/recommendation-api@master] Use "set" instead of "endTiming" in makeMetric

https://gerrit.wikimedia.org/r/981645

gerritbot added a comment.Dec 11 2023, 9:29 AM

Change 981645 merged by Elukey:

[mediawiki/services/recommendation-api@master] Use "set" instead of "endTiming" in makeMetric

https://gerrit.wikimedia.org/r/981645

elukey mentioned this in rMSRA0b555eb148a2: Use "set" instead of "endTiming" in makeMetric.Dec 11 2023, 9:35 AM
gerritbot added a comment.Dec 11 2023, 11:29 AM

Change 982047 had a related patch set uploaded (by Elukey; author: Elukey):

[mediawiki/services/recommendation-api@master] Move the only metric produced from Gauge to Histogram

https://gerrit.wikimedia.org/r/982047

elukey added a comment.Dec 11 2023, 11:34 AM

@colewhite hi again! I added some context to https://gerrit.wikimedia.org/r/c/mediawiki/services/recommendation-api/+/982047, now I have a better idea about what's happening. Lemme know what's best and if I am missing something!

Jdforrester-WMF subscribed.Dec 11 2023, 1:55 PM
gerritbot added a comment.Dec 11 2023, 1:56 PM

Change 982047 merged by jenkins-bot:

[mediawiki/services/recommendation-api@master] Move the only metric produced from Gauge to Histogram

https://gerrit.wikimedia.org/r/982047

elukey mentioned this in rMSRA642bd433d821: Move the only metric produced from Gauge to Histogram.Dec 11 2023, 2:01 PM
gerritbot added a comment.Dec 15 2023, 1:36 PM

Change 983403 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] recommendation-api: update statsd configuration

https://gerrit.wikimedia.org/r/983403

gerritbot added a comment.Dec 18 2023, 11:25 AM

Change 983403 merged by jenkins-bot:

[operations/deployment-charts@master] recommendation-api: update monitoring config

https://gerrit.wikimedia.org/r/983403

gerritbot added a comment.Dec 18 2023, 12:08 PM

Change 983694 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] services: update Docker image and settings for Recommendation API

https://gerrit.wikimedia.org/r/983694

gerritbot added a comment.Dec 18 2023, 2:32 PM

Change 983694 merged by Elukey:

[operations/deployment-charts@master] services: update Docker image and settings for Recommendation API

https://gerrit.wikimedia.org/r/983694

elukey added a comment.Dec 18 2023, 4:35 PM

Tried to deploy rec-api without the statsd exporter, all good but the metrics are still not 100% ok. From a quick look it seems that we define the new metric as "seconds" but in reality the value that it is carries is in ms.

colewhite added a comment.Dec 18 2023, 6:56 PM
In T205870#9413501, @elukey wrote:

Tried to deploy rec-api without the statsd exporter, all good but the metrics are still not 100% ok. From a quick look it seems that we define the new metric as "seconds" but in reality the value that it is carries is in ms.

This was resolved in service-runner 3.1.0. Will recommendation-api work with that version?

gerritbot added a comment.Dec 19 2023, 9:19 AM

Change 984103 had a related patch set uploaded (by Elukey; author: Elukey):

[mediawiki/services/recommendation-api@master] Upgrade to service-runner 3.1.0

https://gerrit.wikimedia.org/r/984103

gerritbot added a comment.Dec 19 2023, 10:03 AM

Change 984103 merged by jenkins-bot:

[mediawiki/services/recommendation-api@master] Upgrade to service-runner 3.1.0

https://gerrit.wikimedia.org/r/984103

gerritbot added a comment.Dec 19 2023, 10:09 AM

Change 984131 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] services: update rec-api's staging Docker image

https://gerrit.wikimedia.org/r/984131

gerritbot added a comment.Dec 19 2023, 10:11 AM

Change 984131 merged by Elukey:

[operations/deployment-charts@master] services: update rec-api's staging Docker image

https://gerrit.wikimedia.org/r/984131

elukey mentioned this in rMSRA12719caf28ff: Upgrade to service-runner 3.1.0.Dec 19 2023, 10:20 AM
elukey added a comment.Dec 19 2023, 10:30 AM
In T205870#9413962, @colewhite wrote:
In T205870#9413501, @elukey wrote:

Tried to deploy rec-api without the statsd exporter, all good but the metrics are still not 100% ok. From a quick look it seems that we define the new metric as "seconds" but in reality the value that it is carries is in ms.

This was resolved in service-runner 3.1.0. Will recommendation-api work with that version?

Done! It is running in staging :)

• DAlangi_WMF subscribed.Jan 24 2024, 3:05 PM
hashar closed subtask T233089: Export zuul metrics to Prometheus as Declined.Mar 29 2024, 2:59 PM
thcipriani reopened subtask T233089: Export zuul metrics to Prometheus as Open.Mar 29 2024, 5:59 PM
gerritbot added a comment.Apr 10 2024, 1:08 PM

Change #1018717 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] services: update the rec-api's Docker image

https://gerrit.wikimedia.org/r/1018717

colewhite closed subtask T240685: MediaWiki Prometheus support as Resolved.Apr 15 2024, 3:12 PM
fgiunchedi mentioned this in T228380: Tech debt: sunsetting of Graphite.May 20 2024, 1:31 PM
gerritbot added a comment.Jun 10 2024, 1:20 PM

Change #1018717 merged by Elukey:

[operations/deployment-charts@master] services: update the rec-api's Docker image

https://gerrit.wikimedia.org/r/1018717

Stashbot added a comment.Jun 10 2024, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-10T13:36:40Z] <elukey> move recommendation-api on wikikube to prometheus metrics (offboarded from statsd) - T205870

elukey added a comment.Jun 10 2024, 4:59 PM

@colewhite o/ I finally deployed recommendation-api, and this time it looks good. I updated also its dashboard:

https://grafana.wikimedia.org/d/Y5wk80oGk/recommendation-api?orgId=1

I see some differences with old/new metrics, but I believe they are due to a better granularity with Prometheus metrics.

This is the snapshot before/after the deployment: https://grafana.wikimedia.org/d/Y5wk80oGk/recommendation-api?orgId=1&from=1718024904966&to=1718029628948

We should be good, if so we can tick-off rec-api :)

fgiunchedi updated the task description. (Show Details)Aug 14 2024, 8:42 AM
colewhite changed the status of subtask T233089: Export zuul metrics to Prometheus from Open to In Progress.Sep 6 2024, 9:56 PM
Aklapper edited projects, added: Patch-Needs-Improvement; removed: Patch-For-Review.Oct 10 2024, 12:35 PM
Aklapper removed subscribers: • dpifke, • Pchelolo.
colewhite updated the task description. (Show Details)Oct 11 2024, 6:30 PM
colewhite updated the task description. (Show Details)Nov 13 2024, 5:07 PM
Aklapper removed colewhite as the assignee of this task.Nov 19 2024, 5:17 PM
Aklapper added a subscriber: colewhite.

@colewhite: Removing task assignee as this open task has been assigned for more than two years - See the email sent to task assignee on October 11th.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!

colewhite closed subtask T233089: Export zuul metrics to Prometheus as Resolved.Dec 19 2024, 12:41 AM
colewhite reopened subtask T233089: Export zuul metrics to Prometheus as Open.Dec 19 2024, 3:48 PM
colewhite closed subtask T233089: Export zuul metrics to Prometheus as Resolved.Dec 19 2024, 8:21 PM
elukey mentioned this in T382408: Prometheus metrics for Kartotherian on k8s.Dec 20 2024, 8:59 AM
fgiunchedi updated the task description. (Show Details)Jan 13 2025, 3:41 PM
fgiunchedi closed subtask T293761: statsd and gunicorn metrics for superset as Invalid.
gerritbot added a comment.Feb 26 2025, 10:26 PM

Change #1123040 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[mediawiki/extensions/EventLogging@master] Migrate eventlogging.client_errors to Prometheus

https://gerrit.wikimedia.org/r/1123040

gerritbot added a project: Patch-For-Review.Feb 26 2025, 10:26 PM
Restricted Application removed a project: Patch-Needs-Improvement. · View Herald TranscriptFeb 26 2025, 10:26 PM
gerritbot added a comment.Feb 26 2025, 10:44 PM

Change #1123041 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[mediawiki/extensions/GrowthExperiments@master] Migrate growthExperiments.StructuredTask.noSuggestionsDialog to Prometheus

https://gerrit.wikimedia.org/r/1123041

gerritbot added a comment.Mar 3 2025, 9:49 PM

Change #1123041 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Migrate growthExperiments.StructuredTask.noSuggestionsDialog to Prometheus

https://gerrit.wikimedia.org/r/1123041

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.19; 2025-03-04).Mar 3 2025, 10:00 PM
gerritbot added a comment.Mar 4 2025, 10:14 PM

Change #1123040 merged by jenkins-bot:

[mediawiki/extensions/EventLogging@master] Migrate eventlogging.client_errors to Prometheus

https://gerrit.wikimedia.org/r/1123040

ReleaseTaggerBot edited projects, added: MW-1.44-notes (1.44.0-wmf.20; 2025-03-11); removed: MW-1.44-notes (1.44.0-wmf.19; 2025-03-04).Mar 4 2025, 11:00 PM
WMDE-leszek mentioned this in T391532: Migrate wikibase.queryService.ui metrics to Prometheus.Apr 9 2025, 9:25 PM
Krinkle closed subtask T175087: Create a navtiming processor for Prometheus as Resolved.Apr 30 2025, 11:29 PM
dancy subscribed.May 20 2025, 4:56 PM

I'm seeing these errors in /var/log/messages on contint1002.wikimedia.org:

May 20 16:52:31 contint1002 statsite@8125[504329]: Traceback (most recent call last):
May 20 16:52:31 contint1002 statsite@8125[504329]:   File "/usr/lib/statsite/sinks/graphite.py", line 94, in <module>
May 20 16:52:31 contint1002 statsite@8125[504329]:     graphite = GraphiteStore(*sys.argv[1:])
May 20 16:52:31 contint1002 statsite@8125[504329]:   File "/usr/lib/statsite/sinks/graphite.py", line 34, in __init__
May 20 16:52:31 contint1002 statsite@8125[504329]:     self.sock = self._create_socket()
May 20 16:52:31 contint1002 statsite@8125[504329]:   File "/usr/lib/statsite/sinks/graphite.py", line 73, in _create_socket
May 20 16:52:31 contint1002 statsite@8125[504329]:     sock.connect((self.host, self.port))
May 20 16:52:31 contint1002 statsite@8125[504329]: TimeoutError: [Errno 110] Connection timed out
May 20 16:52:31 contint1002 statsite@8125[1029]: statsite[1029]: Streaming command exited with status 1
May 20 16:52:31 contint1002 statsite[1029]: Streaming command exited with status 1
root@contint1002:/var/log# systemctl status statsite@8125.service
● statsite@8125.service - statsite (instance 8125)
     Loaded: loaded (/lib/systemd/system/statsite@.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-06-04 22:52:21 UTC; 11 months 14 days ago
   Main PID: 1029 (statsite)
      Tasks: 7 (limit: 154178)
     Memory: 20.3M
        CPU: 9h 6.620s
     CGroup: /system.slice/system-statsite.slice/statsite@8125.service
             ├─  1029 /usr/bin/statsite -f /etc/statsite/8125.ini
             ├─506713 streaming -c python3 /usr/lib/statsite/sinks/graphite.py graphite-in.eqiad.wmnet 2003 ""
             ├─506714 python3 /usr/lib/statsite/sinks/graphite.py graphite-in.eqiad.wmnet 2003
             ├─507203 streaming -c python3 /usr/lib/statsite/sinks/graphite.py graphite-in.eqiad.wmnet 2003 ""
             └─507204 python3 /usr/lib/statsite/sinks/graphite.py graphite-in.eqiad.wmnet 2003
gerritbot added a comment.May 20 2025, 7:48 PM

Change #1148435 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] ci: uninstall statsite from ci hosts

https://gerrit.wikimedia.org/r/1148435

gerritbot added a comment.May 20 2025, 7:48 PM

Change #1148436 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] ci: clean up statsite includes

https://gerrit.wikimedia.org/r/1148436

gerritbot added a comment.May 20 2025, 9:28 PM

Change #1148435 merged by Cwhite:

[operations/puppet@production] ci: uninstall statsite from ci hosts

https://gerrit.wikimedia.org/r/1148435

gerritbot added a comment.May 22 2025, 11:53 PM

Change #1148436 merged by Cwhite:

[operations/puppet@production] ci: clean up statsite includes

https://gerrit.wikimedia.org/r/1148436

gerritbot added a comment.Fri, Sep 25, 3:55 PM

Change #1345159 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/EventLogging@master] ext.eventLogging: Pass the value of client_errors_total to mw.track

https://gerrit.wikimedia.org/r/1345159

gerritbot added a comment.Mon, Oct 5, 7:13 PM

Change #1345159 merged by jenkins-bot:

[mediawiki/extensions/EventLogging@master] ext.eventLogging: Pass the value of client_errors_total to mw.track

https://gerrit.wikimedia.org/r/1345159

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.23; 2026-10-06).Mon, Oct 5, 8:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits