Page MenuHomePhabricator

"refreshLinks.php --dfn-only" can't reliably make it through a full enwiki run
Closed, ResolvedPublic

Description

As reported in RT #2355 by Daniel, a run of "refreshLinks.php --dfn-only" fails on enwiki:
Retrieving illegal entries from pagelinks... 0..100..200..300..400..500..600..700..800..900..1000..1100..1200..1300..1400..1500..1600..1700..1800..1900..2000..2100..2200..2300..2400..2500..2600..2700..2800..2900..3000..3100..3200..3300..3400..3500..3600..3700..3800..3900..4000..4100..4200..4300..4400..4500..4600..4700..4800..4900..5000..5100..5200..5300..5400..5500..5600..5700..5800..5900..6000..6100..6200..6300..6400..6500..6600..6700..6800..6900..7000..7100..7200..7300..7400..7500..7600..7700..7800..7900..8000..8100..8200..8300..8400..8500..8600..8700..8800..8900..9000..9100..9200..9300..9400..9500..9600..9700..9800..9900..10000..10100..10200..10300..10400..10500..10600..10700..10800..10900..11000..11100..11200..11300..11400..11500..11600..11700..11800..11900..12000..12100..12200..12300..12400..12500..12600..12700..Error in fetchObject(): Lost connection to MySQL server during query (10.0.6.46)

Is it possible to split this up into chunks? This blocks full resolution of bug 16112.


Version: wmf-deployment
Severity: normal

Details

Reference
bz36195

Event Timeline

bzimport raised the priority of this task from to Normal.Nov 22 2014, 12:21 AM
bzimport set Reference to bz36195.
bzimport added a subscriber: Unknown Object (MLST).

per Tim it can resume where it stopped. running it manually in screen now to see..

You can control batchsize and max lag for this maintenance script with default script parameter. Maybe it helps to set the batchsize from 100 to a lower number and try again.

greg added a comment.Apr 1 2013, 5:15 PM

Daniel: Is this the script that powers the wikidata cronjob (at least partially)? If so, is this bug resolved (enough)?

Dzahn added a comment.Apr 2 2013, 1:46 AM

It is puppetized in: operations/puppet repo

./manifests/misc/maintenance.php

class misc::maintenance::refreshlinks

51 cron { "cron-refreshlinks-${name}":
52 command => "/usr/local/bin/mwscriptwikiset refreshLinks.php ${cluster}.dblist --dfn-only > /home/mwdeploy/refreshLinks/${name}.log 2>&1",

65 # add cron jobs - usage: <cluster>@<day of month> (these are just needed monthly) (note: s1 is temp. deactivated)
66 cronjob { ['s2@2', 's3@3', 's4@4', 's5@5', 's6@6', 's7@7']: }

<-- as you can see in line 66 above, it is not applied to "s1" which would be enwiki. So it is just running for all other clusters because it never worked for s1. How to make it work on s1, i don't know, sorry.

This bug is about enable the script on s1, but that needs a fix of the maintenance script, which is tracked under bug 42180

Change 199989 had a related patch set uploaded (by PleaseStand):
refreshLinks.php: Limit rows scanned per query in deleteLinksFromNonexistent()

https://gerrit.wikimedia.org/r/199989

Change 199989 merged by jenkins-bot:
refreshLinks.php: Limit rows scanned per query in deleteLinksFromNonexistent()

https://gerrit.wikimedia.org/r/199989

The above change is in 1.25wmf24, which has been deployed to all Wikipedias. There may still be some performance issues, though the script should now be able to run to completion.

Nemo_bis closed this task as Resolved.Apr 9 2015, 6:54 AM
Nemo_bis claimed this task.

The above change is in 1.25wmf24, which has been deployed to all Wikipedias. There may still be some performance issues, though the script should now be able to run to completion.

Thanks! Tentatively closing.