rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort
Closed, ResolvedPublic

Description

It seems after https://github.com/wikimedia/mediawiki-extensions-Wikibase/commit/6e32ab5d3fde0c368831f53f6e81f5c154bf20f7 was merged... Wikidatawiki on beta is causing the db updates to not finish in 45 minutes

As it has 335,601 items... It's gonna take a while

Reedy created this task.Jun 16 2017, 3:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 16 2017, 3:09 AM

Mentioned in SAL (#wikimedia-releng) [2017-06-16T03:10:03Z] <Reedy> running mwscript extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki in screen as root on deployment-tin for T168036

Reedy added a comment.Jun 16 2017, 3:10 AM

That didn't take long to run.. And then logged itself as completed...

Mentioned in SAL (#wikimedia-releng) [2017-06-16T03:13:50Z] <Reedy> running mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki in screen as root on deployment-tin for T168036

Reedy added a comment.Jun 16 2017, 3:13 AM

Ah, seems it's actually rebuildTermSqlIndex.php

And the script doesn't have any sort of resuming functionality... So it gets so far through, and then when the beta job dies, it loses progress, and starts again on the next run...

Reedy renamed this task from RebuildEntityPerPage causing beta-update-databases-eqiad to abort to rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort.Jun 16 2017, 3:15 AM
Reedy triaged this task as High priority.Jun 16 2017, 6:45 PM

So I ran the script... And update.php for wikidatawiki on beta...

...have term_full_entity_id field in wb_terms table.
        Processed up to page 1026 (Q1014)
        Processed up to page 2026 (Q2015)

Something is severely broken here

Reedy added a comment.Jun 16 2017, 6:57 PM

Or is it?

It's hard to tell what's running. That syntax seems to be used by multiple scripts

Reedy added a comment.Jun 18 2017, 7:20 PM

So after nuking hundreds of thousands of items and properties...

...index term_search already set on wb_terms table.
...have term_full_entity_id field in wb_terms table.
        Processed up to page 509402 (Q350016)
        Processed up to page 510931 (Q351021)
        Processed up to page 512461 (Q352026)
        Processed up to page 513986 (Q353031)
        Processed up to page 515516 (Q354036)
        Processed up to page 517045 (Q355041)
        Processed up to page 518567 (Q356046)
        Processed up to page 520117 (Q357051)
        Processed up to page 521626 (Q358056)
        Processed up to page 522645 (Q358707)
        Done rebuilding item terms

It then did properties immediately after...

Done in 9 min 47 s.

If I immediately then run update.php again...

...have term_full_entity_id field in wb_terms table.
        Processed up to page 509402 (Q350016)

There's a major problem of this task not being logged as complete; it shouldn't be being run on EVERY run of update.php. Plus, it shouldn't need to update every row multiple times?

Mentioned in SAL (#wikimedia-releng) [2017-06-18T19:25:18Z] <Reedy> A lot of items on beta wikidatawiki deleted T168036 T167981

Mentioned in SAL (#wikimedia-releng) [2017-06-18T19:26:53Z] <Reedy> Re-enabled beta-update-databases-eqiad as wikidatawiki takes < 10 minutes T168036 T167981

@Reedy sorry for being silent here for a while. Not commenting here does not mean we didn't work on this. I filed T168708 about fixing the issue here. Relevant patch(es) should follow very soon.

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 24 2017, 11:02 AM
greg added a subscriber: greg.Jul 26 2017, 6:58 PM

@Reedy sorry for being silent here for a while. Not commenting here does not mean we didn't work on this. I filed T168708 about fixing the issue here. Relevant patch(es) should follow very soon.

per ^ what's the status with this task?

Should be "resolved" IMO. @Reedy: can you confirm?

greg closed this task as Resolved.Jul 27 2017, 5:24 PM

Reopen if not :)