Maniphest T362012

[jobs] Allow configuration of Promethus scraping of a specific endpoint for publication in grafana.wmcloud.org
Open, Needs TriagePublicFeature
Actions

Assigned To

None

Authored By

	bd808
	Apr 6 2024, 9:09 PM

Tags

Referenced Files

None

Subscribers

Tokens

"Hungry Hippo" token, awarded by Don-vip."Meh!" token, awarded by hashar.

Description

As a tool maintainer
I want to expose implementation specific metrics about my tool to Prometheus
So I can track and monitor trends and point-in-time issues with my tool.

Today we have general metrics related to each tool's Kubernetes namespace tracked in Prometheus. This feature request imagines augmenting that information with additional application specific metrics under the control of the tool's maintainers. Once collected such metrics can be used to power dashboards and/or alerts via additional configuration in related services.

Prometheus can be configured to examine annotations to find endpoints to scrape. This could be combined with support in jobs-api and in both clients (jobs-cli and webservice) for adding those annotations to the Deployment to enable collection of custom metrics from a given webservice.

annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8000"

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T53434 Establish an internal system or a recommended external system for monitoring user-created Toolforge web services
		Open	Feature	None	T362012 [jobs] Allow configuration of Promethus scraping of a specific endpoint for publication in grafana.wmcloud.org
		Open		None	T366923 Set up new Prometheus instance for user-created data

Event Timeline

bd808 created this task.Apr 6 2024, 9:09 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 6 2024, 9:09 PM

bd808 added a parent task: T53434: Establish an internal system or a recommended external system for monitoring user-created Toolforge web services.Apr 6 2024, 9:09 PM

• Bandrqaid654 added commits: rMEXT61ac6d61211d: Update git submodules, rMEXT23c20846d0a2: Update git submodules, rECAUe4542cab5f2a: Merge "Migrate to IDatabase::newReplaceQueryBuilder", rECAU6fbec5636448: Migrate to IDatabase::newReplaceQueryBuilder, rMWc968e9425a8f: Merge "jobqueue: combine Logger channels, improve docs, add missing ingroup…, rMWce96bed119bf: jobqueue: combine Logger channels, improve docs, add missing ingroup tags, rMEXTbf63ec8103ae: Update git submodules, rMEXT788d039a719c: Update git submodules, rMEXT6a37ea7e187f: Update git submodules, R2610:ba5cc2e6cb9b: Maven updates, R2610:840b67988f15: Make sure to get proper file extensions after conversion, rMWd6dc9bf5f4c6: Merge "filebackend: simplify "headless" parameter and STREAM_HEADLESS flag", rMEXT756dd6774727: Update git submodules, R2610:f620bdd5b746: Convert Powerpoint files to PDF to import them to Commons, R2610:5754a2fb8466: Convert Powerpoint files to PDF to import them to Commons, R2610:02965379bd24: Convert Powerpoint files to PDF to import them to Commons, R2610:4d466d608106: Convert Powerpoint files to PDF to import them to Commons, R2610:be8e7325b951: Convert Powerpoint files to PDF to import them to Commons, R2610:358f4358d596: Convert Powerpoint files to PDF to import them to Commons, R2610:786a670fa6c1: Convert Powerpoint files to PDF to import them to Commons, R2610:8377504fa0cc: Convert Powerpoint files to PDF to import them to Commons, R2610:6a4cb4f112f4: Convert Powerpoint files to PDF to import them to Commons, R2610:609f596e4836: ERCC: handling of forbidden files, R2610:f3159be106cc: ERCC: improve categorization, R2610:f69127ea0914: ERCC: improve categorization, rMEXTcc341eb88486: Update git submodules, rMWf5bd12e8322a: config: Abstract test matrix for EtcdConfig, rMW4dce45796f27: Merge "config: Abstract test matrix for EtcdConfig", rMWeeabc10b6b76: filebackend: simplify "headless" parameter and STREAM_HEADLESS flag, R2610:88225f982ad0: Make sure to get proper file extensions after conversion, rGTWN6c45c774e3f2: [WikiEduDashboard] Add an optional message, rEMMV2a31c0b39947: Use "optional_services" to inject service MobileContext, rEWED9b7ec7ec4850: Use "optional_services" to inject service MobileContext, rEJSC6298a6f2bf54: Update patch set 6, rEJSCf2c950232811: Update patch set 6, rEJSC3aa23dcc9642: Update patch set 6, rEJSCb4269fd585f2: Replace global configuration variables, rEJSC84839d9917ee: Use HookHandlers and implement CodeEditorGetPageLanguageHook, rJWTH5eb5aa15101c: plugins: Fork summarize plugin from JSDoc, rJWTH9b0eae950d09: Merge "plugins: Fork summarize plugin from JSDoc", R3320:a4690793492d: don't abort if saving fails due to editconflict and retry, rMEXTba404e6304e6: Update git submodules, rMWVDe55893c2d70c: composer.json: Update wikimedia/less.php from 3.1.0 to 3.2.1, rMW7fcd843ec6da: Update git submodules, rMEXTe810c0427f95: Update git submodules, rEOAU87b5fe95ed20: tests: Port ConsumerSubmitControlTest assertion closures to test cases, rMW1cadad226f1a: Update wikimedia/less.php from 3.1.0 to 3.2.1, rEOAU493d45fd6a8a: Merge "tests: Port ConsumerSubmitControlTest assertion closures to test cases", R2610:81b62c89aeea: Flickr: created https://github.com/boncey/Flickr4Java/pull/724.Apr 6 2024, 9:35 PM

taavi removed commits: R2610:81b62c89aeea: Flickr: created https://github.com/boncey/Flickr4Java/pull/724, rEOAU493d45fd6a8a: Merge "tests: Port ConsumerSubmitControlTest assertion closures to test cases", rMW1cadad226f1a: Update wikimedia/less.php from 3.1.0 to 3.2.1, rEOAU87b5fe95ed20: tests: Port ConsumerSubmitControlTest assertion closures to test cases, rMEXTe810c0427f95: Update git submodules, rMW7fcd843ec6da: Update git submodules, rMWVDe55893c2d70c: composer.json: Update wikimedia/less.php from 3.1.0 to 3.2.1, rMEXTba404e6304e6: Update git submodules, R3320:a4690793492d: don't abort if saving fails due to editconflict and retry, rJWTH9b0eae950d09: Merge "plugins: Fork summarize plugin from JSDoc", rJWTH5eb5aa15101c: plugins: Fork summarize plugin from JSDoc, rEJSC84839d9917ee: Use HookHandlers and implement CodeEditorGetPageLanguageHook, rEJSCb4269fd585f2: Replace global configuration variables, rEJSC3aa23dcc9642: Update patch set 6, rEJSCf2c950232811: Update patch set 6, rEJSC6298a6f2bf54: Update patch set 6, rEWED9b7ec7ec4850: Use "optional_services" to inject service MobileContext, rEMMV2a31c0b39947: Use "optional_services" to inject service MobileContext, rGTWN6c45c774e3f2: [WikiEduDashboard] Add an optional message, R2610:88225f982ad0: Make sure to get proper file extensions after conversion, rMWeeabc10b6b76: filebackend: simplify "headless" parameter and STREAM_HEADLESS flag, rMW4dce45796f27: Merge "config: Abstract test matrix for EtcdConfig", rMWf5bd12e8322a: config: Abstract test matrix for EtcdConfig, rMEXTcc341eb88486: Update git submodules, R2610:f69127ea0914: ERCC: improve categorization, R2610:f3159be106cc: ERCC: improve categorization, R2610:609f596e4836: ERCC: handling of forbidden files, R2610:6a4cb4f112f4: Convert Powerpoint files to PDF to import them to Commons, R2610:8377504fa0cc: Convert Powerpoint files to PDF to import them to Commons, R2610:786a670fa6c1: Convert Powerpoint files to PDF to import them to Commons, R2610:358f4358d596: Convert Powerpoint files to PDF to import them to Commons, R2610:be8e7325b951: Convert Powerpoint files to PDF to import them to Commons, R2610:4d466d608106: Convert Powerpoint files to PDF to import them to Commons, R2610:02965379bd24: Convert Powerpoint files to PDF to import them to Commons, R2610:5754a2fb8466: Convert Powerpoint files to PDF to import them to Commons, R2610:f620bdd5b746: Convert Powerpoint files to PDF to import them to Commons, rMEXT756dd6774727: Update git submodules, rMWd6dc9bf5f4c6: Merge "filebackend: simplify "headless" parameter and STREAM_HEADLESS flag", R2610:840b67988f15: Make sure to get proper file extensions after conversion, R2610:ba5cc2e6cb9b: Maven updates, rMEXT6a37ea7e187f: Update git submodules, rMEXT788d039a719c: Update git submodules, rMEXTbf63ec8103ae: Update git submodules, rMWce96bed119bf: jobqueue: combine Logger channels, improve docs, add missing ingroup tags, rMWc968e9425a8f: Merge "jobqueue: combine Logger channels, improve docs, add missing ingroup…, rECAU6fbec5636448: Migrate to IDatabase::newReplaceQueryBuilder, rECAUe4542cab5f2a: Merge "Migrate to IDatabase::newReplaceQueryBuilder", rMEXT23c20846d0a2: Update git submodules, rMEXT61ac6d61211d: Update git submodules.Apr 7 2024, 4:31 PM

The Prometheus side of this seems easy to implement, but should be done on a separate pair of Prometheus VMs instead of on the current tools-prometheus ones.

bd808 updated the task description. (Show Details)Apr 8 2024, 2:59 PM

hashar awarded a token.Apr 9 2024, 7:53 PM

fnegri subscribed.Apr 24 2024, 8:14 AM

dcaro mentioned this in T362452: Taavi knowledge transfer: cloud-vps monitoring.Apr 24 2024, 9:26 AM

Don-vip awarded a token.May 8 2024, 12:49 PM

Don-vip subscribed.

I'm assuming that by the time we get to this we have gotten around completing T348755: [jobs-api,webservice] Run webservices via the jobs framework.

taavi mentioned this in T366923: Set up new Prometheus instance for user-created data.Fri, Jun 7, 4:15 PM