Page MenuHomePhabricator

Deletion of als and mo wikis causes halting errors in https://tools.wmflabs.org/meta/userpages
Closed, ResolvedPublic

Description

It seems some databases were recently dropped (T181923: Delete als.wik(ibooks|iquote|tionary), mo.wik(ipedia|tionary)) and this is complaining:

https://tools.wmflabs.org/meta/userpages/Evolution+and+evolvability

( ! )  Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[42000]: Syntax error or access violation: 1049 Unknown database 'alswikibooks_p'' in /mnt/nfs/labstore-secondary-tools-project/meta/git/wikimedia-contrib/tool-labs/backend/modules/Database.php on line 282
( ! ) PDOException: SQLSTATE[42000]: Syntax error or access violation: 1049 Unknown database 'alswikibooks_p' in /mnt/nfs/labstore-secondary-tools-project/meta/git/wikimedia-contrib/tool-labs/backend/modules/Database.php on line 282

@jcrespo also mentions that meta_p tables seems to be wrong.

So, questions are:

  • should the code be updated?
  • should meta_p be updated?
  • both, any?

A github issue exists, some links:

Event Timeline

aborrero created this task.Jan 8 2018, 2:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 8 2018, 2:03 PM
chasemp added a subscriber: chasemp.Jan 8 2018, 2:05 PM

To do a clean sweep and rebuild of meta_p

/usr/local/sbin/maintain-meta_p --all-databases [--debug]

Wiki13 added a subscriber: Wiki13.EditedJan 8 2018, 2:15 PM

Tool-wise the database code for this project is also shared (after looking at the individual index.php files) with other tools under the tools.meta project. See github here above. It could be very well possible the oudated meta_p causes for issues, as it probably assumes its still there.

To do a clean sweep and rebuild of meta_p

/usr/local/sbin/maintain-meta_p --all-databases [--debug]

Doing it right now in labsdb1009.eqiad.wmnet, labsdb1010.eqiad.wmnet, labsdb1011.eqiad.wmnet.

@jcrespo , could you check if the meta_p tables looks fine for you? The PHP code is still showing the errors.

jcrespo added a comment.EditedJan 8 2018, 2:36 PM
labsdb1009:~$ my meta_p -e "SELECT * FROM meta_p.wiki WHERE dbname like 'als%'"
+---------------+------+-----------+------------+----------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
| dbname        | lang | name      | family     | url                        | size | slice     | is_closed | has_echo | has_flaggedrevs | has_visualeditor | has_wikidata | is_sensitive |
+---------------+------+-----------+------------+----------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
| alswiki       | gsw  | Wikipedia | wikipedia  | https://als.wikipedia.org  |    2 | s3.labsdb |         0 |        1 |               1 |                1 |            1 |            0 |
| alswikibooks  | en   | None      | wikibooks  | https://als.wikibooks.org  |    1 | s3.labsdb |         1 |        1 |               0 |                0 |            1 |            0 |
| alswikiquote  | en   | None      | wikiquote  | https://als.wikiquote.org  |    1 | s3.labsdb |         1 |        1 |               0 |                0 |            1 |            0 |
| alswiktionary | en   | None      | wiktionary | https://als.wiktionary.org |    1 | s3.labsdb |         1 |        1 |               0 |                0 |            0 |            1 |
+---------------+------+-----------+------------+----------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+

I am not commenting because I don't know if those should be there or not, or if the code should take into account the is_closed. I honestly don't know, but if I had to guess, I would assume they shouln't be if they are deleted?

I ran sudo /usr/local/sbin/maintain-meta_p --all-databases --purge --debug (the --purge part is important) on labsdb1009, labsdb1010 and labsdb1011.

$ sql meta "SELECT * FROM meta_p.wiki WHERE dbname like 'als%'"
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
| dbname  | lang | name      | family    | url                       | size | slice     | is_closed | has_echo | has_flaggedrevs | has_visualeditor | has_wikidata | is_sensitive |
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
| alswiki | gsw  | Wikipedia | wikipedia | https://als.wikipedia.org |    2 | s3.labsdb |         0 |        1 |               1 |                1 |            1 |            0 |
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
$ sql --cluster=web meta "SELECT * FROM meta_p.wiki WHERE dbname like 'als%'"
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
| dbname  | lang | name      | family    | url                       | size | slice     | is_closed | has_echo | has_flaggedrevs | has_visualeditor | has_wikidata | is_sensitive |
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
| alswiki | gsw  | Wikipedia | wikipedia | https://als.wikipedia.org |    2 | s3.labsdb |         0 |        1 |               1 |                1 |            1 |            0 |
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
$ sql --cluster=analytics meta "SELECT * FROM meta_p.wiki WHERE dbname like 'als%'"
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
| dbname  | lang | name      | family    | url                       | size | slice     | is_closed | has_echo | has_flaggedrevs | has_visualeditor | has_wikidata | is_sensitive |
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+
| alswiki | gsw  | Wikipedia | wikipedia | https://als.wikipedia.org |    2 | s3.labsdb |         0 |        1 |               1 |                1 |            1 |            0 |
+---------+------+-----------+-----------+---------------------------+------+-----------+-----------+----------+-----------------+------------------+--------------+--------------+

The is_closed records are now gone and only the active alswiki remains.

https://tools.wmflabs.org/meta/userpages/Evolution+and+evolvability is now complaining about missing mowiki_p database, but that db has also properly been purged from the meta_p.wiki table:

(u3518@s7.labsdb) [meta_p]> SELECT * FROM meta_p.wiki WHERE dbname like 'mo%';
Empty set (0.01 sec)

(u3518@s7.web.db.svc.eqiad.wmflabs) [meta_p]> SELECT * FROM meta_p.wiki WHERE dbname like 'mo%';
Empty set (0.01 sec)

(u3518@s7.analytics.db.svc.eqiad.wmflabs) [meta_p]> SELECT * FROM meta_p.wiki WHERE dbname like 'mo%';
Empty set (0.00 sec)
bd808 added a subscriber: Pathoschild.
  • should meta_p be updated?

{{done}}

@Pathoschild: remaining issues are possibly in the app itself, but do ping us if you find that there is something about meta_p that is causing this.

bd808 renamed this task from tools: missing databases creates some issues to Deletion of als and mo wikis causes halting errors in https://tools.wmflabs.org/meta/userpages.Jan 8 2018, 7:19 PM
bd808 triaged this task as Normal priority.
bd808 updated the task description. (Show Details)
EddieGP added a subscriber: EddieGP.Jan 8 2018, 7:23 PM

Bryan ran maintain-meta_p --all-databases --purge command (as documented on https://wikitech.wikimedia.org/wiki/Delete_a_wiki) during the deletion process (sf T181925#3879345), which didn't work, because the deletion of wiki replica views predates the deletion in prod, which means the patch which removed the wikis from various dblists and added it to deleted.dblists wasn't applied yet at the time Bryan ran the command, but maintain-meta_p relies on the dblists files to build the table.

@aborrero ran maintain-meta_p --all-databases above, which didn't work, because it misses the --purge flag. That flag is used to truncate the table before proceeding. By default maintain-meta_p inserts all known database names and their properties into meta_p.wiki. For each wiki one INSERT sql operation is performed (none are performed for the wikis in deleted.dblists though). Hence this command would add new wikis, but not remove any.

tl;dr: Please run maintain-meta_p --all-databases --purge on all replicas.

Or ignore me, Bryan seems to have done this while I wrote my comment ;-)

@bd808 I don't get an error with userpages/Evolution+and+evolvability. It may have been a cache issue; can you retry and post the error text if it still happens?

tl;dr: Please run maintain-meta_p --all-databases --purge on all replicas.

Or ignore me, Bryan seems to have done this while I wrote my comment ;-)

Yes, I ran it again. For folks from the future: we should not drop the Wiki Replica views until after the databases are placed in deleted.dblist. It makes everything easier.

bd808 closed this task as Resolved.Jan 8 2018, 7:29 PM
bd808 claimed this task.

@bd808 I don't get an error with userpages/Evolution+and+evolvability. It may have been a cache issue; can you retry and post the error text if it still happens?

Working for me now too as is https://tools.wmflabs.org/meta/userpages/BDavis+%28WMF%29. Both of these were failing at the time I wrote T184433#3883935, so likely it was some meta_p caching related issue.

For folks from the future: we should not drop the Wiki Replica views until after the databases are placed in deleted.dblist. It makes everything easier.

That's what I was thinking, updated the docs: https://wikitech.wikimedia.org/w/index.php?title=Delete_a_wiki&diff=1779772&oldid=1779676

238482n375 removed bd808 as the assignee of this task.Jun 15 2018, 8:02 AM
238482n375 lowered the priority of this task from Normal to Lowest.
238482n375 moved this task from Next Up to In Code Review on the Analytics-Kanban board.
238482n375 edited subscribers, added: 238482n375; removed: Aklapper.

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpUYWdzCgogICAgQW5hbHl0aWNzLUthbmJhbiAoSW4gUHJvZ3Jlc3MpCgpTdWJzY3JpYmVycwpBa2xhcHBlciwgSkFsbGVtYW5kb3UKQXNzaWduZWQgVG8KSkFsbGVtYW5kb3UKQXV0aG9yZWQgQnkKSkFsbGVtYW5kb3UsIEZyaSwgSnVuIDE1CkRlc2NyaXB0aW9uCgpPb3ppZSBqb2JzIGhhdmUgYmVlbiBmYWlsaW5nIGF0IGxlYXN0IGEgZmV3IHRpbWVzIGVhY2guIE1vcmUgaW52ZXN0aWdhdGlvbiBuZWVkZWQuCkpBbGxlbWFuZG91IGNyZWF0ZWQgdGhpcyB0YXNrLkZyaSwgSnVuIDE1LCA3OjIxIEFNCkhlcmFsZCBhZGRlZCBhIHN1YnNjcmliZXI6IEFrbGFwcGVyLiC3IFZpZXcgSGVyYWxkIFRyYW5zY3JpcHRGcmksIEp1biAxNSwgNzoyMSBBTQpKQWxsZW1hbmRvdSBjbGFpbWVkIHRoaXMgdGFzay5GcmksIEp1biAxNSwgNzoyMiBBTQpKQWxsZW1hbmRvdSB1cGRhdGVkIHRoZSB0YXNrIGRlc2NyaXB0aW9uLiAoU2hvdyBEZXRhaWxzKQpKQWxsZW1hbmRvdSBhZGRlZCBhIHByb2plY3Q6IEFuYWx5dGljcy1LYW5iYW4uCkpBbGxlbWFuZG91IG1vdmVkIHRoaXMgdGFzayBmcm9tIE5leHQgVXAgdG8gSW4gUHJvZ3Jlc3Mgb24gdGhlIEFuYWx5dGljcy1LYW5iYW4gYm9hcmQuCkNoYW5nZSBTdWJzY3JpYmVycwpDaGFuZ2UgUHJpb3JpdHkKQXNzaWduIC8gQ2xhaW0KTW92ZSBvbiBXb3JrYm9hcmQKQ2hhbmdlIFByb2plY3QgVGFncwpBbmFseXRpY3MtS2FuYmFuCtcKU2VjdXJpdHkK1wpXaWtpbWVkaWEtVkUtQ2FtcGFpZ25zIChTMi0yMDE4KQrXClNjYXAK1wpTY2FwIChTY2FwMy1BZG9wdGlvbi1QaGFzZTIpCtcKQWJ1c2VGaWx0ZXIK1wpEYXRhLXJlbGVhc2UK1wpIYXNodGFncwrXCkxhYnNEQi1BdWRpdG9yCtcKTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2kK1wpMYW5ndWFnZS0yMDE4LUFwci1KdW5lCtcKTGFuZ3VhZ2UtMjAxOC1KYW4tTWFyCtcKSEhWTQrXCkhBV2VsY29tZQrXCkJvbGQKSXRhbGljcwpNb25vc3BhY2VkCkxpbmsKQnVsbGV0ZWQgTGlzdApOdW1iZXJlZCBMaXN0CkNvZGUgQmxvY2sKUXVvdGUKVGFibGUKVXBsb2FkIEZpbGUKTWVtZQpQcmV2aWV3CkhlbHAKRnVsbHNjcmVlbiBNb2RlClBpbiBGb3JtIE9uIFNjcmVlbgoyMzg0ODJuMzc1IGFkZGVkIHByb2plY3RzOiBTZWN1cml0eSwgV2lraW1lZGlhLVZFLUNhbXBhaWducyAoUzItMjAxOCksIFNjYXAgKFNjYXAzLUFkb3B0aW9uLVBoYXNlMiksIEFidXNlRmlsdGVyLCBEYXRhLXJlbGVhc2UsIEhhc2h0YWdzLCBMYWJzREItQXVkaXRvciwgTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2ksIExhbmd1YWdlLTIwMTgtQXByLUp1bmUsIExhbmd1YWdlLTIwMTgtSmFuLU1hciwgSEhWTSwgSEFXZWxjb21lLlBSRVZJRVcKMjM4NDgybjM3NSBtb3ZlZCB0aGlzIHRhc2sgZnJvbSBJbiBQcm9ncmVzcyB0byBJbiBDb2RlIFJldmlldyBvbiB0aGUgQW5hbHl0aWNzLUthbmJhbiBib2FyZC4KMjM4NDgybjM3NSByZW1vdmVkIEpBbGxlbWFuZG91IGFzIHRoZSBhc3NpZ25lZSBvZiB0aGlzIHRhc2suCjIzODQ4Mm4zNzUgdHJpYWdlZCB0aGlzIHRhc2sgYXMgTG93ZXN0IHByaW9yaXR5LgoyMzg0ODJuMzc1IHJlbW92ZWQgc3Vic2NyaWJlcnM6IEFrbGFwcGVyLCBKQWxsZW1hbmRvdS4KQ29udGVudCBsaWNlbnNlZCB1bmRlciBDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uLVNoYXJlQWxpa2UgMy4wIChDQy1CWS1TQSkgdW5sZXNzIG90aGVyd2lzZSBub3RlZDsgY29kZSBsaWNlbnNlZCB1bmRlciBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSAoR1BMKSBvciBvdGhlciBvcGVuIHNvdXJjZSBsaWNlbnNlcy4gQnkgdXNpbmcgdGhpcyBzaXRlLCB5b3UgYWdyZWUgdG8gdGhlIFRlcm1zIG9mIFVzZSwgUHJpdmFjeSBQb2xpY3ksIGFuZCBDb2RlIG9mIENvbmR1Y3QuILcgV2lraW1lZGlhIEZvdW5kYXRpb24gtyBQcml2YWN5IFBvbGljeSC3IENvZGUgb2YgQ29uZHVjdCC3IFRlcm1zIG9mIFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu

238482n375 set Security to Software security bug.Jun 15 2018, 8:06 AM
238482n375 changed the visibility from "Public (No Login Required)" to "Custom Policy".

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu

Restricted Application added a project: Security. · View Herald TranscriptJun 15 2018, 2:05 PM
Aklapper added a project: Tools.
Aklapper removed a project: Security.
Aklapper changed the visibility from "Custom Policy" to "Public (No Login Required)".