Page MenuHomePhabricator

E7. Run populateContentModel.php for page on all WMF wikis
Closed, ResolvedPublic1 Story Points

Event Timeline

Mattflaschen-WMF raised the priority of this task from to Needs Triage.
Mattflaschen-WMF updated the task description. (Show Details)
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2015, 12:55 AM
DannyH triaged this task as Normal priority.Jun 30 2015, 5:18 PM
DannyH set Security to None.
DannyH edited a custom field.
DannyH renamed this task from Run populateContentModel.php for page and archive for all WMF wikis to E7. Run populateContentModel.php for page and archive for all WMF wikis.Jun 30 2015, 9:02 PM
Catrope assigned this task to Legoktm.Jul 1 2015, 5:44 PM

Why do we need to populate archive? Shouldn't we only do that when we plan on converting a namespace?

I think that is likely to be forgotten. I would much rather it be consistent across all the wikis.

It shouldn't be overly expensive if we don't do revision.

Legoktm renamed this task from E7. Run populateContentModel.php for page and archive for all WMF wikis to E7. Run populateContentModel.php for page on all WMF wikis.Jul 13 2015, 6:44 PM

All small wikis and test* wikis done, will do medium after lunch.

All small wikis and test* wikis done, will do medium after lunch.

Did this get done?

All small wikis and test* wikis done, will do medium after lunch.

Did this get done?

Oops, nope. It's running right now for medium.

Medium finished. I'll let it sit for a few hours and start large tomorrow morning.

[10:38:27] <legoktm> !log running foreachwikiindblist /home/legoktm/largebutnotenwiki.dblist populateContentModel.php --ns=all --table=page

[10:38:27] <legoktm> !log running foreachwikiindblist /home/legoktm/largebutnotenwiki.dblist populateContentModel.php --ns=all --table=page

For the record, commonswiki has more pages than enwiki...oh well.

[14:57:50] <legoktm> !log running mwscript populateContentModel.php --wiki=enwiki --ns=all --table=page

Killed it...there's a memory issue somewhere:

Legoktm added a subscriber: hoo.Jul 26 2015, 10:54 PM

@hoo found the culprit, Flow's ContentHandlerDefaultModelFor hook invoked LinkCache, which will grow indefinitely: T106998: LinkCache should be LRU based to avoid indefinitely growing and causing memory issues.

Restarted the script with LinkCache::singleton()->clear() statements added to avoid OOM'ing again. Not going to backport the LinkCache fix since it has scary perf implications.

Legoktm closed this task as Resolved.Jul 28 2015, 5:11 PM

Finished on enwiki.