- migrate only up to item id 2000000 (2M)
- script is under /repo/maintenance/rebuildItemTerms.php (no manual execution here)
- run it in batches, say1000 items per batch
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Addshore | T208425 [EPIC] Kill the wb_terms table | |||
| Resolved | ArielGlenn | T226167 audit public tables and make sure we dump them all | |||
| Resolved | Addshore | T219175 [Mega] - Migrate data from wb_terms to new schema | |||
| Resolved | Ladsgroup | T219123 Migrate to and read from new store for item terms | |||
| Resolved | Ladsgroup | T225056 Run Item Terms Rebuild script |
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-releng) [2019-07-31T19:20:15Z] <Amir1> ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki (T225056)
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-09-03T11:35:49Z] <Amir1> ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 1000 --sleep 2 (T225056)
Comment Actions
ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 1000 --sleep 2 Rebuilding Q1 till Q250 [ERROR] Failed to save terms of item: Q29 [ERROR] Failed to save terms of item: Q35 [ERROR] Failed to save terms of item: Q45 [ERROR] Failed to save terms of item: Q55 [ERROR] Failed to save terms of item: Q188 Rebuilding Q251 till Q500 Rebuilding Q501 till Q750 Rebuilding Q751 till Q1000 Done.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-09-04T11:46:47Z] <Amir1> start of ladsgroup@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 2000000 --sleep 2 > ~/rebuildItemTerms.out 2> rebuildItemTerms.err (T225056). This is going to take a while. On screen
Comment Actions
The complete list of things that the script can't build:
[ERROR] Failed to save terms of item: Q29 [ERROR] Failed to save terms of item: Q35 [ERROR] Failed to save terms of item: Q45 [ERROR] Failed to save terms of item: Q55 [ERROR] Failed to save terms of item: Q188 [ERROR] Failed to save terms of item: Q1321 [ERROR] Failed to save terms of item: Q1860 [ERROR] Failed to save terms of item: Q334351 [ERROR] Failed to save terms of item: Q649003 [ERROR] Failed to save terms of item: Q994512 [ERROR] Failed to save terms of item: Q994528 [ERROR] Failed to save terms of item: Q1653380 [ERROR] Failed to save terms of item: Q1676025 [ERROR] Failed to save terms of item: Q1676026
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-09-06T10:59:01Z] <Amir1> ladsgroup@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=testwikidatawiki (T225056)
Comment Actions
Change 534790 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mediawiki: Add rebuildItemTerms for Wikidata
Comment Actions
Change 534790 merged by Marostegui:
[operations/puppet@production] mediawiki: Add rebuildItemTerms for Wikidata
Comment Actions
Merged and as requested at :
root@mwmaint1002:~# cat /var/log/wikidata/wikidata-rebuildItemTerms.log Rebuilding Q501 till Q2000000 root@mwmaint1002:~# ls -lh /var/log/wikidata/wikidata-rebuildItemTerms.log -rw-r--r-- 1 www-data www-data 30 Sep 10 08:57 /var/log/wikidata/wikidata-rebuildItemTerms.log
Those permissions are the same as the other logs on that directory have.
Comment Actions
Change 535526 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mediawiki: Start rebuildItermTerms for wikidatawiki
Comment Actions
Done:
root@mwmaint1002:/var/log/wikidata# cat /var/log/wikidata/wikidata-rebuildItemTerms.log Rebuilding Q501 till Q2000000 root@mwmaint1002:/var/log/wikidata# ls -lh /var/log/wikidata/wikidata-rebuildItemTerms.log -rw-r--r-- 1 www-data www-data 30 Sep 16 09:45 /var/log/wikidata/wikidata-rebuildItemTerms.log
Comment Actions
Change 535526 merged by Marostegui:
[operations/puppet@production] mediawiki: Start rebuildItermTerms for wikidatawiki
Comment Actions
It's running:
ladsgroup@mwmaint1002:~$ tail /var/log/wikidata/wikidata-rebuildItemTerms.log Rebuilding Q2001750 till Q2001999 Rebuilding Q2002000 till Q2002249 Rebuilding Q2002250 till Q2002499 Rebuilding Q2002500 till Q2002749 Rebuilding Q2002750 till Q2002999 Rebuilding Q2003000 till Q2003249 Rebuilding Q2003250 till Q2003499 Rebuilding Q2003500 till Q2003749 Rebuilding Q2003750 till Q2003999 Rebuilding Q2004000 till Q2004249
Comment Actions
at around 6:50 UTC this morning we began seeing this:
icinga-wm: PROBLEM - MediaWiki eqiad exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops
Probably the interaction between a script and a bot. The candidates for fast editing bots at the time were KrBot, LargeDatasetBot, ԱշբոտՏՆՂ At the time I looked, I saw about 35 edits a minute from KrBot, and about that many exceptions a minute. The bot has slowed down to about 10 edits a minute some time later and the number of fatals decreased as well: https://logstash.wikimedia.org/goto/8fa30aac1e6166f08bdfb0f906b2c14f
Comment Actions
https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539498/ was merged in response and kicked in about 10 minutes ago, with good results on the graph.
Comment Actions
We done running the manual rebuild of first 2m items
@ArielGlenn in case of more issues related to migartion, let's report them in the parent task instead. Thanks!