Page MenuHomePhabricator

user_properties_anon view not being created/maintained consistently on wikireplicas due to lack of meta_p in all sections
Closed, ResolvedPublicBUG REPORT

Description

Based on maintain-views.yaml, I would expect the user_properties_anon view to be created for all wikis in small.dblist and medium.dblist. But that doesn't seem to be the case.

[16:09:27] <AntiComposite>	 anyone know how the user_properties_anon view is supposed to work?
[16:10:09] <AntiComposite>	 based on maintain-views.yaml I'd expect it to be available for wikis in medium.dblist and small.dblist, but that doesn't appear to be the case (at least not consistently)
[16:10:42] <AntiComposite>	 https://quarry.wmcloud.org/query/59576 it works on huwiki, which is in large.dblist 
[16:11:16] <AntiComposite>	 but not thwiki (medium, https://quarry.wmcloud.org/query/59573) or thwikinews_p (small, https://quarry.wmcloud.org/query/59578)
[16:11:33] <AntiComposite>	 or enwiki (large, https://quarry.wmcloud.org/query/59577)
[16:15:17] <bd808>	 AntiComposite: I think this is the config that builds the view -- https://github.com/wikimedia/puppet/blob/5b9e59a44ed3432c23227e7b1a64562c02047afe/modules/profile/templates/wmcs/db/wikireplicas/maintain-views.yaml#L833-L837 -- but I don't remember what the "limit: 2" part there means.
[16:15:34] <AntiComposite>	 should be https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/wmcs/db/wikireplicas/views/maintain-views.py#194
[16:15:52] <AntiComposite>	 at least, that's the only place where limit is mentioned in maintain-views.py
[16:16:39] <bd808>	 *nod*
[16:22:49] <bd808>	 that size is set to `db_info.get("size", None)` -- https://github.com/wikimedia/puppet/blob/5b9e59a44ed3432c23227e7b1a64562c02047afe/modules/profile/files/wmcs/db/wikireplicas/views/maintain-views.py#L495 which should I think be the value from https://github.com/wikimedia/puppet/blob/5b9e59a44ed3432c23227e7b1a64562c02047afe/modules/profile/templates/wmcs/db/wikireplicas/maintain-views.yaml#L24-L31 depending on which dblist the wikidb was
[16:22:49] <bd808>	  found in.
[16:23:26] <bd808>	 So yeah, I agree that in theory this should be small.dblist + medium.dblist dbs
[16:25:15] <bd808>	 I wonder if it has something to do with stale meta_p data. See https://github.com/wikimedia/puppet/commit/e492c0a9933d553548983d230ce58c49b3e1fce0 for the other bit from the config about the allow list of props to expose
[16:25:59] <AntiComposite>	 huwiki has been in large.dblist since 2015 https://phabricator.wikimedia.org/rOMWC77560147969945121697192eab093c840f7a8976
[16:26:03] <bd808>	 meta_p is local to each wiki replica instance and not replicated from elsewhere
[16:26:43] <bd808>	 so if wiki replica nodes are added or rebuilt the meta_p data they have can drift or be completely missing
[16:27:11] <AntiComposite>	 if properties_anon_whitelist was empty, or irrelevant, or whatever, wouldn't the view still get created but with no data available?
[16:29:50] <bd808>	 maybe? there are several bare returns to exit from do_customview -- https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/wmcs/db/wikireplicas/views/maintain-views.py#188 -- I haven't reasoned them all out for this case
[16:33:12] <bd808>	 I'm actually now wondering if we are only provisioning meta_p on the s7 replica nodes in the latest wiki replica cluster design...
[16:34:37] <bd808>	 https://github.com/wikimedia/operations-cookbooks/blob/84d104d0f2f0246d5a3a23a8357493ce8871840a/cookbooks/sre/wikireplicas/add-wiki.py is the new hotness for maintaining that table and it looks to be restricting it to s7.
[16:35:00] <bd808>	 This could really be an "action at a distance" problem that we didn't notice when building the latest cluster.
[16:36:06] <bd808>	 AntiComposite: that's as far into the rabbit hole as I can go right now. sounds like it would be worth a phab task and maybe a cut-n-paste of some of this initial tracing
[16:36:11] <AntiComposite>	 yup, and since the table is actually missing, it fails check_customview_source and the view doesn't get created
[16:36:28] <AntiComposite>	 I'll write one up

Event Timeline

Bebiezaza rescinded a token.
Bebiezaza awarded a token.
Bebiezaza subscribed.

I just checked every wiki in s7.dblist, and the view was created for all of them, even though they're all in large.dblist.

• bd808 renamed this task from user_properties_anon view not being created/maintained consistently on wikireplicas to user_properties_anon view not being created/maintained consistently on wikireplicas due to lack of meta_p in all sections.Oct 29 2021, 5:44 PM
• odimitrijevic raised the priority of this task from Low to Medium.

Possible fixes:

  • provision meta_p in all slices so that the existing view creation logic which involves reading data from meta_p works
  • change the view creation script so that the list of exposed columns is read from yaml config instead of meta_p
  • change the view creation script so that the list of exposed columns is read from yaml config instead of meta_p

This seems to be consensus as easiest/best going forward from WMCS team discussion.

Change 739680 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/puppet@production] wikireplicas: remove dependency on meta_p for user_properties_anon view

https://gerrit.wikimedia.org/r/739680

Change 739680 merged by Andrew Bogott:

[operations/puppet@production] wikireplicas: remove dependency on meta_p for user_properties_anon view

https://gerrit.wikimedia.org/r/739680

Change 743472 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wiki replicas: depool clouddb1013-clouddb1016

https://gerrit.wikimedia.org/r/743472

Change 743473 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wiki replicas: repool clouddb1013-clouddb1016, depool clouddb1017-1020

https://gerrit.wikimedia.org/r/743473

Change 743474 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wiki replicas: repool clouddb1017-1020

https://gerrit.wikimedia.org/r/743474

Change 743472 merged by Andrew Bogott:

[operations/puppet@production] wiki replicas: depool clouddb1013-clouddb1016

https://gerrit.wikimedia.org/r/743472

Change 743473 merged by Andrew Bogott:

[operations/puppet@production] wiki replicas: repool clouddb1013-clouddb1016, depool clouddb1017-1020

https://gerrit.wikimedia.org/r/743473

Change 743474 merged by Andrew Bogott:

[operations/puppet@production] wiki replicas: repool clouddb1017-1020

https://gerrit.wikimedia.org/r/743474

Andrew subscribed.

I applied this and ran maintain-views everywhere; would love it if someone else confirms this did what was hoped for.

AntiCompositeNumber reassigned this task from AntiCompositeNumber to • bd808.

Exists:

Does not exist:

Re-reading maintain-views.py again, this is expected behavior. The limit applies to the minimum size, not the maximum (if limit > db_size: return) . This does make sense in the context of an anonymized view, but the comments and log message didn't exactly help my understanding.

Thanks for the work to actually roll this out @Andrew. And thank you @AntiCompositeNumber for noticing the regression, helping figure out what led to the issue, and verifying the fix. It takes a village to keep the replicas working. :)