I think I did it.

<3

apache in people1004 is still serving the old omg.json file, otherwise, you could have seen the where is left. I can wait for a bit for cache to expire. I'll bug it if it's not done by tomorrow.

Excellent! I think then we are done. db-mysql works fine on cumin1002 thanks to @ABran-WMF so we should start moving all our stuff to cumin1002 and make sure it is all fine there.

zarcillo database (db1215) was missing. I have fixed,but I am wondering if there could be others?

The random hosts is something that happens all the time, usually connection errors. I think I explicitly excluded clouddb for wikiuser rotation and that was left here. I'll remove that.

The misc and backups are probably because these sections are not added to omg. At least for this specific case, they should be added.

(sorry for the mess, the original script was written for rotation of wikiuser/wikiadmin)

Not a mess! It has already simplified a lot all the process. If you need some help, let me know!
In the case of cumin...of course we need them to be deployed everywhere, which is of course different from wikiuser/wikiadmin. Maybe we can add that feature to the script.

I re-ran the script again and added the users and grants, let's see what's missing now.

clouddb* hosts seem to be failing as well as db1208, db2102, dbstore1009:3350
The rest looks good!

okay, These seems to be something weird going on wrt these hosts.

The grants are there from what I'm seeing:

root@clouddb1021.eqiad.wmnet[(none)]> show grants for root@10.64.48.98;
+------------------------------------------------------------------------------------------------------------------------+
| Grants for root@10.64.48.98                                                                                            |
+------------------------------------------------------------------------------------------------------------------------+
| GRANT `labsdbuser` TO `root`@`10.64.48.98` WITH ADMIN OPTION                                                           |
| GRANT ALL PRIVILEGES ON *.* TO `root`@`10.64.48.98` IDENTIFIED BY PASSWORD '<redacted>' |
+------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.001 sec)

But in cumin1002, I get this:

ladsgroup@cumin1002:~$ sudo db-mysql clouddb1021:3311
ERROR 1045 (28000): Access denied for user 'root'@'10.64.48.98' (using password: YES)

It can be the ssl issue but not sure.

Might be the password in the old format too, I can check in a bit

clouddb* hosts fixed.
I'll check the other two

db2102 fixed.
db1208 seems to be analytics related with matomo database - I am taking a look

omg got updated as well: https://people.wikimedia.org/~ladsgroup/omg/

(The issue was that I was pushing to people host in eqiad while the active one is codfw).