Page MenuHomePhabricator

Run Item Terms Rebuild script
Closed, ResolvedPublic

Description

  • migrate only up to item id 2000000 (2M)
  • script is under /repo/maintenance/rebuildItemTerms.php (no manual execution here)
  • run it in batches, say1000 items per batch

Event Timeline

alaa_wmde set Due Date to Jul 1 2019, 10:00 PM.
alaa_wmde updated the task description. (Show Details)
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptJun 5 2019, 6:54 AM
alaa_wmde changed Due Date from Jul 1 2019, 10:00 PM to Jul 9 2019, 10:00 PM.Jun 13 2019, 6:00 PM
alaa_wmde changed Due Date from Jul 9 2019, 10:00 PM to Jul 2 2019, 10:00 PM.

Mentioned in SAL (#wikimedia-releng) [2019-07-31T19:20:15Z] <Amir1> ladsgroup@deployment-deploy01:/srv/mediawiki-staging/php-master$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki (T225056)

Mentioned in SAL (#wikimedia-operations) [2019-09-03T11:35:49Z] <Amir1> ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 1000 --sleep 2 (T225056)

ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 1000 --sleep 2
Rebuilding Q1 till Q250
[ERROR] Failed to save terms of item: Q29
[ERROR] Failed to save terms of item: Q35
[ERROR] Failed to save terms of item: Q45
[ERROR] Failed to save terms of item: Q55
[ERROR] Failed to save terms of item: Q188
Rebuilding Q251 till Q500
Rebuilding Q501 till Q750
Rebuilding Q751 till Q1000
Done.

Mentioned in SAL (#wikimedia-operations) [2019-09-04T11:46:47Z] <Amir1> start of ladsgroup@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 2000000 --sleep 2 > ~/rebuildItemTerms.out 2> rebuildItemTerms.err (T225056). This is going to take a while. On screen

The complete list of things that the script can't build:

[ERROR] Failed to save terms of item: Q29
[ERROR] Failed to save terms of item: Q35
[ERROR] Failed to save terms of item: Q45
[ERROR] Failed to save terms of item: Q55
[ERROR] Failed to save terms of item: Q188
[ERROR] Failed to save terms of item: Q1321
[ERROR] Failed to save terms of item: Q1860
[ERROR] Failed to save terms of item: Q334351
[ERROR] Failed to save terms of item: Q649003
[ERROR] Failed to save terms of item: Q994512
[ERROR] Failed to save terms of item: Q994528
[ERROR] Failed to save terms of item: Q1653380
[ERROR] Failed to save terms of item: Q1676025
[ERROR] Failed to save terms of item: Q1676026

Mentioned in SAL (#wikimedia-operations) [2019-09-06T10:59:01Z] <Amir1> ladsgroup@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=testwikidatawiki (T225056)

Change 534790 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mediawiki: Add rebuildItemTerms for Wikidata

https://gerrit.wikimedia.org/r/534790

Change 534790 merged by Marostegui:
[operations/puppet@production] mediawiki: Add rebuildItemTerms for Wikidata

https://gerrit.wikimedia.org/r/534790

Merged and as requested at :

root@mwmaint1002:~# cat /var/log/wikidata/wikidata-rebuildItemTerms.log
Rebuilding Q501 till Q2000000
root@mwmaint1002:~# ls -lh /var/log/wikidata/wikidata-rebuildItemTerms.log
-rw-r--r-- 1 www-data www-data 30 Sep 10 08:57 /var/log/wikidata/wikidata-rebuildItemTerms.log

Those permissions are the same as the other logs on that directory have.

Change 535526 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mediawiki: Start rebuildItermTerms for wikidatawiki

https://gerrit.wikimedia.org/r/535526

Merged and as requested at :

root@mwmaint1002:~# cat /var/log/wikidata/wikidata-rebuildItemTerms.log
Rebuilding Q501 till Q2000000
root@mwmaint1002:~# ls -lh /var/log/wikidata/wikidata-rebuildItemTerms.log
-rw-r--r-- 1 www-data www-data 30 Sep 10 08:57 /var/log/wikidata/wikidata-rebuildItemTerms.log

Those permissions are the same as the other logs on that directory have.

Since we haven't started yet. We need to do it again since logrotate ate it.

Let me know when you want me to touch it again.

Done:

root@mwmaint1002:/var/log/wikidata# cat /var/log/wikidata/wikidata-rebuildItemTerms.log
Rebuilding Q501 till Q2000000
root@mwmaint1002:/var/log/wikidata# ls -lh /var/log/wikidata/wikidata-rebuildItemTerms.log
-rw-r--r-- 1 www-data www-data 30 Sep 16 09:45 /var/log/wikidata/wikidata-rebuildItemTerms.log

Change 535526 merged by Marostegui:
[operations/puppet@production] mediawiki: Start rebuildItermTerms for wikidatawiki

https://gerrit.wikimedia.org/r/535526

It's running:

ladsgroup@mwmaint1002:~$ tail /var/log/wikidata/wikidata-rebuildItemTerms.log
Rebuilding Q2001750 till Q2001999
Rebuilding Q2002000 till Q2002249
Rebuilding Q2002250 till Q2002499
Rebuilding Q2002500 till Q2002749
Rebuilding Q2002750 till Q2002999
Rebuilding Q2003000 till Q2003249
Rebuilding Q2003250 till Q2003499
Rebuilding Q2003500 till Q2003749
Rebuilding Q2003750 till Q2003999
Rebuilding Q2004000 till Q2004249

It's done up to Q10Mio now. Just heads up

at around 6:50 UTC this morning we began seeing this:

icinga-wm: PROBLEM - MediaWiki eqiad exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops

Probably the interaction between a script and a bot. The candidates for fast editing bots at the time were KrBot, LargeDatasetBot, ԱշբոտՏՆՂ At the time I looked, I saw about 35 edits a minute from KrBot, and about that many exceptions a minute. The bot has slowed down to about 10 edits a minute some time later and the number of fatals decreased as well: https://logstash.wikimedia.org/goto/8fa30aac1e6166f08bdfb0f906b2c14f

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539498/ was merged in response and kicked in about 10 minutes ago, with good results on the graph.

We done running the manual rebuild of first 2m items

@ArielGlenn in case of more issues related to migartion, let's report them in the parent task instead. Thanks!