Page MenuHomePhabricator

maintain-views table filter not working for custom views on multiple tables
Closed, ResolvedPublic


Coming from T305300 there seems to be an issue with maintain-views regenerating the view for centralauth_p.localuser.
The view in puppet is:

    - localuser
    - globaluser
  view: >
    select lu_wiki, lu_name, lu_attached_timestamp, lu_attached_method, lu_attachment_method,
    lu_local_id, lu_global_id
  where: lu_global_id = gu_id AND gu_hidden_level=0

Please note that lu_attachment_method is there.
The current view is:

root@clouddb1021.eqiad.wmnet[centralauth_p]> show create table localuser\G
*************************** 1. row ***************************
                View: localuser
         Create View: CREATE ALGORITHM=UNDEFINED DEFINER=`viewmaster`@`%` SQL SECURITY DEFINER VIEW `localuser` AS select `centralauth`.`localuser`.`lu_wiki` AS `lu_wiki`,`centralauth`.`localuser`.`lu_name` AS `lu_name`,`centralauth`.`localuser`.`lu_attached_timestamp` AS `lu_attached_timestamp`,`centralauth`.`localuser`.`lu_attached_method` AS `lu_attached_method`,`centralauth`.`localuser`.`lu_local_id` AS `lu_local_id`,`centralauth`.`localuser`.`lu_global_id` AS `lu_global_id` from (`centralauth`.`localuser` join `centralauth`.`globaluser`) where `centralauth`.`localuser`.`lu_global_id` = `centralauth`.`globaluser`.`gu_id` and `centralauth`.`globaluser`.`gu_hidden_level` = 0
character_set_client: utf8
collation_connection: utf8_general_ci
1 row in set (0.001 sec)

Regenerating the view doesn't seem to include the new column lu_attachment_method

root@clouddb1021:~#  maintain-views --database centralauth --table localuser --replace --debug
2022-06-29 05:54:39,753 DEBUG Removing 0 dbs as sensitive
2022-06-29 05:54:39,769 INFO Full views for centralauth:
2022-06-29 05:54:39,770 INFO Custom views for centralauth:

The maintain-views script on clouddb1021 is also up-to-date and has that column:

root@clouddb1021:/etc# cat ./maintain-views.yaml | grep -w lu_attachment_method
      select lu_wiki, lu_name, lu_attached_timestamp, lu_attached_method, lu_attachment_method,

For some reason regenerating the view doesn't include the new column.

The column is there on the table under the view:

root@clouddb1021.eqiad.wmnet[centralauth_p]> show create table centralauth.localuser\G
*************************** 1. row ***************************
       Table: localuser
Create Table: CREATE TABLE `localuser` (
  `lu_wiki` varbinary(255) NOT NULL DEFAULT '',
  `lu_name` varbinary(255) NOT NULL DEFAULT '',
  `lu_attached_timestamp` binary(14) DEFAULT NULL,
  `lu_attached_method` enum('primary','empty','mail','password','admin','new','login') DEFAULT NULL,
  `lu_local_id` int(10) unsigned DEFAULT NULL,
  `lu_global_id` int(10) unsigned DEFAULT NULL,
  `lu_attachment_method` tinyint(3) unsigned DEFAULT NULL,
  PRIMARY KEY (`lu_wiki`,`lu_name`),
  KEY `lu_name` (`lu_name`,`lu_wiki`)
1 row in set (0.001 sec)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I'm guessing the table filtering code

if args.table:
    fullviews = [t for t in config["fullviews"] if t == args.table]

    customviews = {}
    for view, meta in config["customviews"].items():
        if meta["source"] == args.table:
            customviews[view] = config["customviews"][view]

is getting confused because localuser is configured to have multiple source tables:

    - localuser
    - globaluser

I guess this could be fixed with something like if isinstance(meta["source"], list) and args.table in meta["source"], but that would also make --table globaluser refresh the localuser view and I'm not sure if that's expected or not. Alternatively we could check if args.table is the first element in the list. Thoughts?

taavi renamed this task from maintain-views not regenerating centralauth views to maintain-views table filter not working for custom views on multiple tables.Jun 30 2022, 1:49 PM

I don't think refreshing the localuser view on each of those run is a problem (as it should be mostly a noop).

I believe that I have been struck by this bug as well. T315426#8903257

The situation for me was that I added multiple sources to the abuse_filter_history view, then ran sudo maintain-views --all-databases --table abuse_filter_history --replace-all
It skipped the replacement of the view.

If I omit the --table abuse_filter_history option and replace all tables, then it correctly recreates the view.

I can submit a patch to maintain-views, based on @taavi's proposed fix in T311588#8040308
There are a few more instances of multi-source views in the config, so we should perhaps evaluate whether the same fix will be ok for those as well.

Change 927723 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the maintain-views script to improve the table selection option

Change 927723 merged by Btullis:

[operations/puppet@production] Update the maintain-views script to improve the table selection option

taavi claimed this task.