We can probably start by running a pt-show-grants on each host, compare them and make sure they are all the same. Once done that, we can start fixing redundant stuff and escaping things.
One thing I've always noticed was:
GRANT ALL PRIVILEGES ON '%\_p'.* already covers every _p database, making these three specific ones redundant:

• heartbeat\_p — subsumed by %\_p
• meta\_p — subsumed by %\_p
• centralauth\_p — subsumed by %\_p
• %wik%\_p — also subsumed by %\_p

But we have to double check everything.

I did a cumin run:

fnegri@cumin1003:~$ sudo cumin clouddb* 'for s in $(ls /run/mysqld/mysqld*sock); do pt-show-grants --socket $s --only=maintainviews; done'

This shows that clouddb1013, clouddb1017, clouddb1022 and clouddb1023 have the incorrect "double escape" with \\_ instead of \_. I will fix it manually on those 4 hosts, then check again.

Note: I couldn't check clouddb1019 as it's currently down: T422813: clouddb1019 down.

Manual commands I'm running on the affected hosts to fix the double escape:

GRANT ALL PRIVILEGES ON `meta\_p`.* TO `maintainviews`@`localhost`;
GRANT ALL PRIVILEGES ON `heartbeat\_p`.* TO `maintainviews`@`localhost`;
GRANT ALL PRIVILEGES ON `centralauth\_p`.* TO `maintainviews`@`localhost`;
GRANT ALL PRIVILEGES ON `%wik%\_p`.* TO `maintainviews`@`localhost`;
GRANT ALL PRIVILEGES ON `%\_p`.* TO `maintainviews`@`localhost`;

REVOKE ALL PRIVILEGES ON `meta\\_p`.* FROM maintainviews@localhost;
REVOKE ALL PRIVILEGES ON `heartbeat\\_p`.* FROM maintainviews@localhost;
REVOKE ALL PRIVILEGES ON `centralauth\\_p`.* FROM maintainviews@localhost;
REVOKE ALL PRIVILEGES ON `%wik%\\_p`.* FROM maintainviews@localhost;
REVOKE ALL PRIVILEGES ON `%\\_p`.* FROM maintainviews@localhost;

Done, and verified that now the grants for maintainviews are in sync across all clouddbs (except clouddb1019 that I could not verify).

I will prepare a patch to remove the redundant grants from wiki-replicas.sql.

Change #1270464 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] mariadb: wiki-replicas: remove redundant grants

https://gerrit.wikimedia.org/r/1270464

Change #1270465 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] mariadb: wiki-replicas: add grants for %_maintain

https://gerrit.wikimedia.org/r/1270465

These grants are present on all clouddb hosts but are not listed in wiki-replicas.sql:

GRANT BINLOG ADMIN, BINLOG REPLAY, CONNECTION ADMIN, FEDERATED ADMIN, READ_ONLY ADMIN, REPLICATION SLAVE ADMIN, SET USER, SUPER ON *.* TO `maintainviews`@`localhost` IDENTIFIED BY PASSWORD '***';

I don't think they are required, I am using clouddb1017@s3 to test if they can be removed by recreating the user with:

DROP USER maintainviews@localhost;
CREATE USER maintainviews@localhost IDENTIFIED BY '***';

Two of the grants are actually needed to create views. This seems to be enough:

GRANT SET USER, READ_ONLY ADMIN ON *.* TO `maintainviews`@`localhost` IDENTIFIED BY PASSWORD '***';

Change #1270891 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] mariadb: wiki-replicas: add missing grants

https://gerrit.wikimedia.org/r/1270891

Two of the grants are actually needed to create views.

Added in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1270891

Change #1270464 merged by FNegri:

[operations/puppet@production] mariadb: wiki-replicas: remove redundant grants

https://gerrit.wikimedia.org/r/1270464

Change #1270465 merged by FNegri:

[operations/puppet@production] mariadb: wiki-replicas: add grants for %_maintain

https://gerrit.wikimedia.org/r/1270465

Change #1270891 merged by FNegri:

[operations/puppet@production] mariadb: wiki-replicas: add missing grants

https://gerrit.wikimedia.org/r/1270891

I merged the patches but they need to be applied manually. So far I've only applied the new grants to clouddb1017, I will do the other clouddbs next week.