Page MenuHomePhabricator
Create Task
Maniphest T231280

Remove grants for the old dbproxy hosts from the misc databases
Closed, ResolvedPublic
Actions

Description

Once the old proxies are not active, we need to remove those grants from the databases.
The list of proxies and IPs:

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
production-m3.sql: Remove dbproxy1003 grantsoperations/puppetproduction+0 -43
production-m3.sql: Remove grants for dbproxy1008operations/puppetproduction+0 -45
production-m1.sql: Remove grants for dbproxy1001operations/puppetproduction+0 -42
report_users: Remove dbproxy1004,dbproxy1009operations/softwaremaster+3 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui triaged this task as Medium priority.Aug 27 2019, 9:20 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Stashbot added a comment.Aug 27 2019, 9:21 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T09:21:30Z] <marostegui> Remove grants for dbproxy1004 and dbproxy1009 from m4 hosts (db1107 and db1108) - T231280

Marostegui added a comment.Aug 27 2019, 9:24 AM

m4: Grants removed on db1107 and db1108 for dbproxy1004 and dbproxy1009

dbproxy1004.eqiad.wmnet has address 10.64.16.148
dbproxy1009.eqiad.wmnet has address 10.64.32.158

The users were eventlog and haproxy.

root@db1107.eqiad.wmnet[mysql]> select user,host from user where host='10.64.32.148' or host='10.64.32.158';
Empty set (0.00 sec)

root@db1107.eqiad.wmnet[mysql]>
Marostegui updated the task description. (Show Details)Aug 27 2019, 9:24 AM
Marostegui added a subtask: T231520: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011.Aug 29 2019, 7:31 AM
Marostegui added a subtask: T231967: Decommission dbproxy1005.eqiad.wmnet.Sep 4 2019, 7:59 AM
Stashbot added a comment.Sep 5 2019, 5:42 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-05T05:42:56Z] <marostegui> Remove grants for dbproxy1005 T231280 T231967

Stashbot mentioned this in T231967: Decommission dbproxy1005.eqiad.wmnet.Sep 5 2019, 5:42 AM
Marostegui updated the task description. (Show Details)Sep 5 2019, 6:01 AM

I have removed the following grants that were for dbproxy1005, from m5 databases (and also checked they don't exist anywhere else):

# host 10.64.16.155
155.16.64.10.in-addr.arpa domain name pointer dbproxy1005.eqiad.wmnet.
+----------------+--------------+
| user           | host         |
+----------------+--------------+
| designate      | 10.64.16.155 |
| haproxy        | 10.64.16.155 |
| labsdbaccounts | 10.64.16.155 |
| root           | 10.64.16.155 |
| ssastry        | 10.64.16.155 |
| testreduce     | 10.64.16.155 |
+----------------+--------------+
6 rows in set (0.00 sec)
root@db1133.eqiad.wmnet[(none)]> select user,host from mysql.user where host='10.64.16.155';
Empty set (0.00 sec)
Marostegui updated the task description. (Show Details)Sep 5 2019, 6:03 AM
Marostegui mentioned this in T233207: Decommission dbproxy1006.eqiad.wmnet.Sep 18 2019, 11:42 AM
Marostegui updated the task description. (Show Details)
Stashbot added a comment.Oct 7 2019, 7:09 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-07T07:09:58Z] <marostegui> Remove grants for dbproxy1006 on m1 databases - T231280

Marostegui added a comment.Oct 7 2019, 7:12 AM

The following grants for dbproxy1006 IP's have been removed from m1 databases:

# host dbproxy1006
dbproxy1006.eqiad.wmnet has address 10.64.16.159


root@db1135.eqiad.wmnet[mysql]> select user,host from user where host='10.64.16.159';
+--------------+--------------+
| user         | host         |
+--------------+--------------+
| bacula       | 10.64.16.159 |
| etherpadlite | 10.64.16.159 |
| haproxy      | 10.64.16.159 |
| librenms     | 10.64.16.159 |
| rddmarc      | 10.64.16.159 |
| rt           | 10.64.16.159 |
+--------------+--------------+
6 rows in set (0.00 sec)

After removing them:

root@db1135.eqiad.wmnet[mysql]> select user,host from user where host='10.64.16.159';
Empty set (0.00 sec)
Marostegui updated the task description. (Show Details)Oct 7 2019, 7:12 AM
gerritbot added a comment.Oct 22 2019, 5:06 AM

Change 545142 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] report_users: Remove dbproxy1004,dbproxy1009

https://gerrit.wikimedia.org/r/545142

gerritbot added a project: Patch-For-Review.Oct 22 2019, 5:06 AM
gerritbot added a comment.Oct 22 2019, 5:09 AM

Change 545142 merged by jenkins-bot:
[operations/software@master] report_users: Remove dbproxy1004,dbproxy1009

https://gerrit.wikimedia.org/r/545142

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2019, 5:10 AM
Marostegui moved this task from In progress to Pending comment on the DBA board.Nov 19 2019, 1:41 PM
RobH closed subtask T231967: Decommission dbproxy1005.eqiad.wmnet as Resolved.Dec 19 2019, 11:09 PM
Marostegui mentioned this in T244463: Decommission dbproxy1001.eqiad.wmnet.Feb 11 2020, 6:45 AM
Stashbot added a comment.Feb 11 2020, 6:48 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-11T06:48:57Z] <marostegui> Remove grants in m1 for dbproxy1001 - T231280

Marostegui added a comment.Feb 11 2020, 6:52 AM

Grants removed in db1135 with replication

root@db1135.eqiad.wmnet[(none)]> select user from mysql.user where host like '10.64.0.165';
+--------------+
| user         |
+--------------+
| bacula       |
bacula
| bacula9      |
| bloguser     |
| bugs         |
| contacts     |
| designate    |
| etherpadlite |
| haproxy      |
| librenms     |
| pdns         |
| pdns_admin   |
| rddmarc      |
| rt           |
+--------------+

root@db1135.eqiad.wmnet[(none)]> select user from mysql.user where host like '10.64.0.165';
Empty set (0.01 sec)

root@db1135.eqiad.wmnet[(none)]>
Marostegui updated the task description. (Show Details)Feb 11 2020, 6:53 AM
gerritbot added a comment.Feb 11 2020, 6:56 AM

Change 571424 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m1.sql: Remove grants for dbproxy1001

https://gerrit.wikimedia.org/r/571424

gerritbot added a project: Patch-For-Review.Feb 11 2020, 6:56 AM
gerritbot added a comment.Feb 11 2020, 6:59 AM

Change 571424 merged by Marostegui:
[operations/puppet@production] production-m1.sql: Remove grants for dbproxy1001

https://gerrit.wikimedia.org/r/571424

Maintenance_bot removed a project: Patch-For-Review.Feb 11 2020, 7:10 AM
Marostegui updated the task description. (Show Details)Feb 17 2020, 6:34 AM
Marostegui added subtasks: T245384: decommission dbproxy1002.eqiad.wmnet, T245385: decommission dbproxy1007.eqiad.wmnet.Feb 17 2020, 6:40 AM
Marostegui mentioned this in T245384: decommission dbproxy1002.eqiad.wmnet.
Marostegui mentioned this in T245385: decommission dbproxy1007.eqiad.wmnet.
Marostegui updated the task description. (Show Details)Feb 17 2020, 9:47 AM
Marostegui updated the task description. (Show Details)Feb 18 2020, 6:20 AM
Marostegui moved this task from Pending comment to In progress on the DBA board.Feb 18 2020, 6:52 AM
Stashbot added a comment.Feb 19 2020, 8:50 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-19T08:50:48Z] <marostegui> Remove dbproxy1007 grants from m2 - T231280

Marostegui updated the task description. (Show Details)Feb 19 2020, 8:53 AM
Papaul closed subtask T233207: Decommission dbproxy1006.eqiad.wmnet as Resolved.Mar 20 2020, 3:22 PM
Marostegui added a subtask: T248944: Decommission dbproxy1010.eqiad.wmnet.Mar 31 2020, 5:49 AM
Stashbot added a comment.Apr 2 2020, 9:47 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-02T09:47:32Z] <marostegui> Remove haproxy@10.64.37.14 from labsdb hosts - T231280 T248944

Stashbot mentioned this in T248944: Decommission dbproxy1010.eqiad.wmnet.Apr 2 2020, 9:47 AM
Marostegui updated the task description. (Show Details)Apr 2 2020, 9:49 AM
Marostegui added a comment.Apr 2 2020, 10:00 AM
root@cumin1001:~# host dbproxy1010
dbproxy1010.eqiad.wmnet has address 10.64.37.14
root@cumin1001:~# for i in labsdb1009 labsdb1010 labsdb1011 labsdb1012; do mysql.py -h$i -e "show grants for 'haproxy'@'10.64.37.14';";done
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.37.14'
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.37.14'
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.37.14'
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.37.14'
Marostegui updated the task description. (Show Details)Apr 2 2020, 10:01 AM
Marostegui mentioned this in T249590: decommission dbproxy1011.eqiad.wmnet.Apr 7 2020, 8:03 AM
Marostegui added a subtask: T249590: decommission dbproxy1011.eqiad.wmnet.
Stashbot added a comment.Apr 13 2020, 8:15 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-13T08:15:06Z] <marostegui> Remove grants for haproxy@10.64.37.15 from labsdb hosts T231280

Marostegui added a comment.Apr 13 2020, 8:17 AM

dbproxy1011 grants removed

root@cumin1001:/home/marostegui# host dbproxy1011
dbproxy1011.eqiad.wmnet has address 10.64.37.15
root@cumin1001:/home/marostegui# for i in labsdb1009 labsdb1010 labsdb1011 labsdb1012; do mysql.py -h$i -e "show grants for 'haproxy'@'10.64.37.15';";done
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.37.15'
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.37.15'
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.37.15'
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.37.15'
Marostegui updated the task description. (Show Details)Apr 13 2020, 8:17 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T231520: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 as Resolved.Apr 14 2020, 12:24 PM
• Cmjohnson closed subtask T249590: decommission dbproxy1011.eqiad.wmnet as Resolved.May 13 2020, 6:33 PM
• Cmjohnson closed subtask T245384: decommission dbproxy1002.eqiad.wmnet as Resolved.May 13 2020, 7:03 PM
• Cmjohnson closed subtask T244463: Decommission dbproxy1001.eqiad.wmnet as Resolved.
• Cmjohnson closed subtask T245385: decommission dbproxy1007.eqiad.wmnet as Resolved.May 13 2020, 7:08 PM
• Cmjohnson closed subtask T248944: Decommission dbproxy1010.eqiad.wmnet as Resolved.May 18 2020, 3:41 PM
Marostegui added a subtask: T255406: decommission dbproxy1008.eqiad.wmnet.Jun 15 2020, 9:30 AM
Marostegui updated the task description. (Show Details)
Marostegui mentioned this in T255406: decommission dbproxy1008.eqiad.wmnet.
Stashbot added a comment.Jun 24 2020, 5:14 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-24T05:14:56Z] <marostegui> Remove grants from dbproxy1008 - T231280 T255406

Marostegui added a comment.Jun 24 2020, 5:16 AM

dbproxy1008 grants removed from m3 (and also checked all the other mX sections):

root@cumin2001:/home/marostegui# ./section m3 | while read host port; do echo $host; mysql.py -h$host:$port -e "show grants for 'haproxy'@'10.64.32.157';"; done
db2134.codfw.wmnet
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.32.157'
db2078.codfw.wmnet
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.32.157'
db1128.eqiad.wmnet
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.32.157'
db1117.eqiad.wmnet
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.32.157'
gerritbot added a comment.Jun 24 2020, 5:19 AM

Change 607407 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m3.sql: Remove grants for dbproxy1008

https://gerrit.wikimedia.org/r/607407

gerritbot added a project: Patch-For-Review.Jun 24 2020, 5:19 AM
gerritbot added a comment.Jun 24 2020, 5:27 AM

Change 607407 merged by Marostegui:
[operations/puppet@production] production-m3.sql: Remove grants for dbproxy1008

https://gerrit.wikimedia.org/r/607407

Marostegui updated the task description. (Show Details)Jun 24 2020, 5:28 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 24 2020, 6:10 AM
Marostegui mentioned this in T256216: Decommission dbproxy1003.eqiad.wmnet.Jun 24 2020, 9:59 AM
Marostegui added a subtask: T256216: Decommission dbproxy1003.eqiad.wmnet.
Marostegui updated the task description. (Show Details)Jun 24 2020, 10:27 AM
gerritbot added a comment.Jul 8 2020, 8:25 AM

Change 610237 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m3.sql: Remove dbproxy1003 grants

https://gerrit.wikimedia.org/r/610237

gerritbot added a project: Patch-For-Review.Jul 8 2020, 8:25 AM
Stashbot added a comment.Jul 8 2020, 8:28 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-08T08:28:50Z] <marostegui> Remove dbproxy1003 grants from misc hosts T231280

Marostegui added a comment.Jul 8 2020, 8:40 AM

Grants for dbproxy1003's IP removed:

for i in db1132 db1128 db1133 db1080; do echo $i; mysql.py -h$i -e "select user,host from mysql.user where host like '10.64.0.198'";done
db1132
db1128
db1133
db1080
Marostegui closed this task as Resolved.Jul 8 2020, 8:40 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Jul 8 2020, 8:41 AM

Change 610237 merged by Marostegui:
[operations/puppet@production] production-m3.sql: Remove dbproxy1003 grants

https://gerrit.wikimedia.org/r/610237

Marostegui updated the task description. (Show Details)Jul 8 2020, 8:45 AM
• Cmjohnson closed subtask T256216: Decommission dbproxy1003.eqiad.wmnet as Resolved.Aug 10 2020, 6:11 PM
• Cmjohnson closed subtask T255406: decommission dbproxy1008.eqiad.wmnet as Resolved.Aug 10 2020, 6:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits