Page MenuHomePhabricator
Create Task
Maniphest T359251

[REPO][SW][GRAFMIGR] (mw.track) Migrate MediaWiki.wikibase.repo.* to statslib
Closed, ResolvedPublic
Actions

Description

Background

Graphite is scheduled to become read-only by March 2023 (Q3-FY2024/2025). The sooner metrics are migrated, the more history and continuity a dashboard will have after the transition. Old data can still be queried from Graphite directly. Consider using "Save as" to create a copy of your Grafana dashboard and rename it to "My Name (old Graphite)" to ease viewing of old data in the future, even after you've updated your dashboard to use Prometheus.

PHP change

Follow the migration process as outlined below.

Secure/Conduct code review(s).
Deploy the changes to production via the train (https://wikitech.wikimedia.org/wiki/Deployments/Train).
Verify that the changes have been successfully implemented.
Update the relevant dashboard(s) by replacing the old Graphite metric(s) with the new Prometheus metric(s).
Please follow the guidelines and standards outlined in the provided documentation:

https://www.mediawiki.org/wiki/Manual:Stats for detailed guidance on the conversion process.
https://drive.google.com/file/d/12yQEuOapkML1vb9MgCaX1QzbLBdXE6X2/view for a video tutorial on the conversion process.
https://docs.google.com/presentation/d/1SZWf_D3mWNX-XHN8PHYI84LDZr6GUQC2AMhZ9mQXCI0/edit#slide=id.g2795460c956_0_23 for slides on the best practices for converting metrics to statslib.

JS change

TODO:

GetEntities.php
  • MediaWiki.wikibase.repo.api.getentities.entities
MediaWikiEditEntityFactory.php
  • MediaWiki.wikibase.repo.EditEntity.timing.EditEntity.attemptSave
  • MediaWiki.wikibase.repo.EditEntity.timing.EditFilterHookRunner.run
  • MediaWiki.wikibase.repo.EditEntity.timing.EntityStore.saveEntity
  • MediaWiki.wikibase.repo.EditEntity.timing.*.*.item
NOTE: Requires updates to StatsdSaveTimeRecordingEditEntity.php, StatsdSaveTimeRecordingEntityStore.php and StatsdTimeRecordingEditFilterHookRunner.php
TermFallbackCacheFactory.php and TermFallbackCacheFactoryTest.php
  • MediaWiki.wikibase.repo.formatterCache
NOTE: Requires updates to newStatsdRecordingCache, which is in TermFallbackCacheServiceFactory.php, TermFallbackServiceFactoryTest.php and TermFallbackCacheFactoryTest.php
EntityParserOutputGeneratorFactory.php
  • MediaWiki.wikibase.repo.ParserOutputGenerator.timing.getParserOutput.html.*
NOTE: Requires updates to StatsdTimeRecordingEntityParserOutputGenerator.php
wikibase.ui.entitysearch.js
  • MediaWiki.wikibase.repo.searchbox.footer.submit
Other
  • MediaWiki.wikibase.repo.term_store.*

source code

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Migrate MediaWiki.wikibase.* statsmediawiki/extensions/Wikibasewmf/1.44.0-wmf.25+798 -581
Add strict types to files touched in previous changemediawiki/extensions/Wikibasemaster+26 -94
EditEntity: Hard-code stats prefixesmediawiki/extensions/Wikibasemaster+12 -66
Migrate MediaWiki.wikibase.* statsmediawiki/extensions/Wikibasemaster+798 -581
add wikibase.repo.searchbox.footer.submit to prometheusmediawiki/extensions/Wikibasemaster+1 -0
Remove unused MediaWiki.wikibase.repo.term_store.* metricmediawiki/extensions/Wikibasemaster+0 -5
Migrate MediaWiki.wikibase.repo.api.getentities.entities.* to statslibmediawiki/extensions/Wikibasemaster+22 -14
Customize query in gerrit

Related Objects
Search...

Event Timeline

• lmata created this task.Mar 6 2024, 3:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2024, 3:59 AM
• lmata mentioned this in T350592: EPIC: migrate in use metrics and dashboards to statslib.Mar 6 2024, 4:04 AM
colewhite updated the task description. (Show Details)Apr 26 2024, 4:13 PM
colewhite mentioned this in T363584: Hackathon 2024: Stats!.
colewhite merged a task: T359254: Migrate MediaWiki.wikidatawiki.wikibase.repo.* to statslib.
Lucas_Werkmeister_WMDE mentioned this in T359253: [CLIENT][SW][GRAFMIGR] Migrate MediaWiki.$prefix.wikibase.client.scribunto.* to statslib.Jun 20 2024, 2:18 PM
Lucas_Werkmeister_WMDE added projects: Wikidata, wmde-wikidata-tech, MediaWiki-extensions-Wikibase-Repo.Jun 21 2024, 10:47 AM
ItamarWMDE added a parent task: T371616: [EPIC][GRAFMIGR] Spruce up Wikidata Grafana Metrics.Aug 1 2024, 3:13 PM
Lucas_Werkmeister_WMDE renamed this task from Migrate MediaWiki.wikibase.repo.* to statslib to [REPO][SW] Migrate MediaWiki.wikibase.repo.* to statslib.Aug 15 2024, 12:42 PM
Lucas_Werkmeister_WMDE moved this task from Incoming to [DOT] By Project on the wmde-wikidata-tech board.
Lucas_Werkmeister_WMDE added a project: [Archived]Wikidata Dev Team.
Lucas_Werkmeister_WMDE moved this task from Incoming to [DOT] Tech Backlog on the [Archived]Wikidata Dev Team board.
karapayneWMDE renamed this task from [REPO][SW] Migrate MediaWiki.wikibase.repo.* to statslib to [REPO][SW][GRAFMIGR] Migrate MediaWiki.wikibase.repo.* to statslib.Aug 29 2024, 8:12 AM
gerritbot added a comment.Nov 27 2024, 8:29 PM

Change #1098611 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[mediawiki/extensions/Wikibase@master] Migrate MediaWiki.wikibase.repo.term_store.* to statslib

https://gerrit.wikimedia.org/r/1098611

gerritbot added a project: Patch-For-Review.Nov 27 2024, 8:29 PM
ItamarWMDE moved this task from [DOT] By Project to Wikidata Omega Triage on the wmde-wikidata-tech board.Dec 4 2024, 11:42 AM
ItamarWMDE edited projects, added wmde-wikidata-tech (Wikidata Omega Triage); removed wmde-wikidata-tech.
ItamarWMDE moved this task from Incoming to Grafana Revamp on the wmde-wikidata-tech (Wikidata Omega Triage) board.Dec 4 2024, 2:06 PM
ItamarWMDE removed a project: wmde-wikidata-tech (Wikidata Omega Triage).Dec 6 2024, 12:03 PM
ItamarWMDE added a project: wmde-wikidata-tech.
ItamarWMDE removed a project: [Archived]Wikidata Dev Team.
ItamarWMDE moved this task from Incoming to Wikidata Omega Triage on the wmde-wikidata-tech board.Dec 10 2024, 1:21 PM
ItamarWMDE edited projects, added wmde-wikidata-tech (Wikidata Omega Triage); removed wmde-wikidata-tech.
andrea.denisse added subscribers: Lucas_Werkmeister_WMDE, andrea.denisse.Dec 17 2024, 4:07 PM

Hi @Lucas_Werkmeister_WMDE , do you have any feedback on https://gerrit.wikimedia.org/r/1098611 ?

gerritbot added a comment.Dec 18 2024, 1:14 AM

Change #1105100 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[mediawiki/extensions/Wikibase@master] Migrate MediaWiki.wikibase.repo.api.getentities.entities.* to statslib

https://gerrit.wikimedia.org/r/1105100

Krinkle renamed this task from [REPO][SW][GRAFMIGR] Migrate MediaWiki.wikibase.repo.* to statslib to [REPO][SW][GRAFMIGR] (mw.track) Migrate MediaWiki.wikibase.repo.* to statslib.Dec 19 2024, 10:59 PM
Krinkle updated the task description. (Show Details)
Krinkle mentioned this in T355837: Add Prometheus support to statsd.js via mw.track().Dec 20 2024, 12:37 AM
andrea.denisse added a comment.Jan 7 2025, 5:09 PM

Hi @Lucas_Werkmeister_WMDE ,

I wanted to follow up on the patch I submitted a while ago, as it's been pending review for quite some time. Your input and approval are essential to move forward with this work, and I would greatly appreciate it if you could take a moment to review it.

If there's any issue or reason it's been delayed, please let me know so we can address it. Alternatively, if you're unable to review it, could you suggest someone else who might be able to take it on?

Looking forward to your response.

andrea.denisse assigned this task to Lucas_Werkmeister_WMDE.Jan 7 2025, 5:10 PM
andrea.denisse added a project: WMDE-Analytics-Engineering.
Lucas_Werkmeister_WMDE added a subscriber: AndrewTavis_WMDE.Jan 8 2025, 12:07 PM

I’ve replied to that Gerrit change, but I don’t know who’s responsible for the rest of this task (there are more metrics to migrate)… IIUC this was in @AndrewTavis_WMDE’s area?

Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this task.Jan 8 2025, 12:07 PM
AndrewTavis_WMDE added a subscriber: karapayneWMDE.Jan 8 2025, 12:24 PM

I'd be happy to help as I can, but then I'm not sure if this falls within the scope of work for T371616: [EPIC][GRAFMIGR] Spruce up Wikidata Grafana Metrics, which is the only epic I have scheduled for Grafana/Graphite/Prometheus related work. Assuming it does, I'd wait for @karapayneWMDE to sign off on further support from my end.

gerritbot added a comment.Jan 9 2025, 11:56 AM

Change #1098611 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Remove unused MediaWiki.wikibase.repo.term_store.* metric

https://gerrit.wikimedia.org/r/1098611

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.12; 2025-01-14).Jan 9 2025, 12:00 PM
andrea.denisse updated the task description. (Show Details)Jan 9 2025, 5:31 PM
gerritbot added a comment.Feb 21 2025, 11:44 PM

Change #1121715 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/Wikibase@master] migrate wikibase.repo.searchbox.footer.submit

https://gerrit.wikimedia.org/r/1121715

AndrewTavis_WMDE updated the task description. (Show Details)Feb 24 2025, 10:50 AM
AndrewTavis_WMDE updated the task description. (Show Details)
gerritbot added a comment.Feb 25 2025, 4:30 PM

Change #1121715 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] add wikibase.repo.searchbox.footer.submit to prometheus

https://gerrit.wikimedia.org/r/1121715

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.19; 2025-03-04); removed MW-1.44-notes (1.44.0-wmf.12; 2025-01-14).Feb 25 2025, 5:00 PM
colewhite updated the task description. (Show Details)Mar 7 2025, 5:22 PM
AndrewTavis_WMDE claimed this task.Mar 10 2025, 2:14 PM
AndrewTavis_WMDE updated the task description. (Show Details)
AndrewTavis_WMDE updated the task description. (Show Details)
gerritbot added a comment.Mar 10 2025, 4:25 PM

Change #1126096 had a related patch set uploaded (by Andrew McAllister (WMDE); author: Andrew McAllister (WMDE)):

[mediawiki/extensions/Wikibase@master] T359251: Migrate MediaWiki.wikibase.repo.* Bug: T359251

https://gerrit.wikimedia.org/r/1126096

AndrewTavis_WMDE added a project: Wikidata Analytics (Kanban).Mar 10 2025, 4:51 PM
AndrewTavis_WMDE moved this task from Incoming to In Progress on the Wikidata Analytics (Kanban) board.
gerritbot added a comment.Mar 18 2025, 1:53 PM

Change #1126096 had a related patch set uploaded (by Andrew McAllister (WMDE); author: Andrew McAllister (WMDE)):

[mediawiki/extensions/Wikibase@master] T359251: Migrate MediaWiki.wikibase.repo.*

https://gerrit.wikimedia.org/r/1126096

AndrewTavis_WMDE moved this task from In Progress to To-Do on the Wikidata Analytics (Kanban) board.Mar 24 2025, 10:28 AM
AndrewTavis_WMDE added a comment.Mar 24 2025, 10:31 AM

Moved to To-Do on our board temporarily as the focus right now is migrating the needed jobs for T377352 :)

AndrewTavis_WMDE moved this task from To-Do to In Progress on the Wikidata Analytics (Kanban) board.Mar 28 2025, 9:27 AM
AndrewTavis_WMDE reassigned this task from AndrewTavis_WMDE to HasanAkgun_WMDE.Apr 4 2025, 2:39 PM
AndrewTavis_WMDE mentioned this in T359252: [GRAFMIGR] Migrate MediaWiki.wikibase.view.* to statslib.
AndrewTavis_WMDE added a comment.Apr 4 2025, 2:44 PM

The patch for T359252 was combined into Wikibase/+/1126096, as my assumption is that we can't do a scoped migration of one sub-directory of WIkibase. I'd done this for Wikibase/view, but instances of StatsdDataFactory were being called from what I assume were other parts of Wikibase.

@HasanAkgun_WMDE is taking over this task from here :)

HasanAkgun_WMDE moved this task from In Progress to In Review on the Wikidata Analytics (Kanban) board.Apr 11 2025, 12:26 PM
gerritbot added a comment.Apr 23 2025, 2:38 PM

Change #1138379 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] EditEntity: Hard-code stats prefixes

https://gerrit.wikimedia.org/r/1138379

gerritbot added a comment.Apr 23 2025, 2:43 PM

Change #1138380 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Add strict types to files touched in previous change

https://gerrit.wikimedia.org/r/1138380

andrea.denisse unsubscribed.Apr 23 2025, 5:59 PM
gerritbot added a comment.Apr 25 2025, 2:08 PM

Change #1126096 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Migrate MediaWiki.wikibase.* stats

https://gerrit.wikimedia.org/r/1126096

gerritbot added a comment.Apr 25 2025, 3:20 PM

Change #1138379 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] EditEntity: Hard-code stats prefixes

https://gerrit.wikimedia.org/r/1138379

gerritbot added a comment.Apr 25 2025, 3:21 PM

Change #1138380 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Add strict types to files touched in previous change

https://gerrit.wikimedia.org/r/1138380

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.27; 2025-04-29); removed MW-1.44-notes (1.44.0-wmf.19; 2025-03-04).Apr 25 2025, 4:00 PM
gerritbot added a comment.Apr 28 2025, 10:19 AM

Change #1139426 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Andrew McAllister (WMDE)):

[mediawiki/extensions/Wikibase@wmf/1.44.0-wmf.25] Migrate MediaWiki.wikibase.* stats

https://gerrit.wikimedia.org/r/1139426

gerritbot added a comment.Apr 28 2025, 1:19 PM

Change #1139426 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.44.0-wmf.25] Migrate MediaWiki.wikibase.* stats

https://gerrit.wikimedia.org/r/1139426

Stashbot added a comment.Apr 28 2025, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-28T13:19:55Z] <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1139426|Migrate MediaWiki.wikibase.* stats (T359251 T359252)]], [[gerrit:1139427|Migrate MediaWiki.$prefix.wikibase.client.scribunto.* to statslib (T359253)]]

Stashbot added a comment.Apr 28 2025, 1:24 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-28T13:24:18Z] <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister-wmde: Backport for [[gerrit:1139426|Migrate MediaWiki.wikibase.* stats (T359251 T359252)]], [[gerrit:1139427|Migrate MediaWiki.$prefix.wikibase.client.scribunto.* to statslib (T359253)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Apr 28 2025, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-28T13:33:48Z] <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1139426|Migrate MediaWiki.wikibase.* stats (T359251 T359252)]], [[gerrit:1139427|Migrate MediaWiki.$prefix.wikibase.client.scribunto.* to statslib (T359253)]] (duration: 13m 52s)

karapayneWMDE closed this task as Resolved.Oct 27 2025, 1:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits