Page MenuHomePhabricator
Create Task
Maniphest T183249

Reimage and upgrade to stretch all dbproxies
Closed, ResolvedPublic
Actions

Description

Current proxies are on jessie and have bad configuration (socket location, lack of good firewall configuration). We need to upgrade them before database failover.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo triaged this task as Medium priority.Dec 19 2017, 2:11 PM
jcrespo created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 19 2017, 2:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
jcrespo added parent tasks: Restricted Task, T104699: Firewall configurations for database hosts.Dec 19 2017, 2:12 PM
jcrespo moved this task from Triage to In progress on the DBA board.
gerritbot subscribed.Dec 19 2017, 2:14 PM

Change 399164 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Preparing reimage of dbproxy1001 and setup proxy firewall

https://gerrit.wikimedia.org/r/399164

gerritbot added a project: Patch-For-Review.Dec 19 2017, 2:14 PM
jcrespo renamed this task from Reimage and upgrade to stretch all proxies to Reimage and upgrade to stretch all dbproxies.Dec 19 2017, 2:26 PM
gerritbot added a comment.Dec 19 2017, 2:32 PM

Change 399164 merged by Jcrespo:
[operations/puppet@production] mariadb: Preparing reimage of dbproxy1001 and setup proxy firewall

https://gerrit.wikimedia.org/r/399164

ops-monitoring-bot subscribed.Dec 19 2017, 2:37 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1001.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712191436_jynus_20842.log.

jcrespo added a comment.Dec 19 2017, 5:46 PM

dbproxy1001 has been successfully reimaged, which joins the already upgraded to stretch dbproxy1004 and dbproxy1009 (although these one have to yet be reconfigured its socket and firewall). All others are on jessie still, have to be reimaged one by one.

gerritbot added a comment.Dec 20 2017, 7:17 AM

Change 399347 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Preparing to reimage dbproxy1004

https://gerrit.wikimedia.org/r/399347

gerritbot added a comment.Dec 20 2017, 7:21 AM

Change 399347 merged by Jcrespo:
[operations/puppet@production] dbproxy: Change socket location for dbproxy1004

https://gerrit.wikimedia.org/r/399347

gerritbot added a comment.Dec 20 2017, 7:28 AM

Change 399349 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Preparing dbproxy1005 for reimage

https://gerrit.wikimedia.org/r/399349

gerritbot added a comment.Dec 20 2017, 7:29 AM

Change 399349 merged by Jcrespo:
[operations/puppet@production] dbproxy: Preparing dbproxy1005 for reimage

https://gerrit.wikimedia.org/r/399349

ops-monitoring-bot added a comment.Dec 20 2017, 7:37 AM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1005.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712200736_jynus_12971.log.

gerritbot added a comment.Dec 20 2017, 7:54 AM

Change 399350 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Fix dbproxy1005 and prepare dbproxy1007 for reimage

https://gerrit.wikimedia.org/r/399350

gerritbot added a comment.Dec 20 2017, 7:54 AM

Change 399350 merged by Jcrespo:
[operations/puppet@production] dbproxy: Fix dbproxy1005 and prepare dbproxy1007 for reimage

https://gerrit.wikimedia.org/r/399350

ops-monitoring-bot added a comment.Dec 20 2017, 7:57 AM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1007.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712200756_jynus_16585.log.

gerritbot added a comment.Dec 20 2017, 8:03 AM

Change 399351 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Prepare dbproxy1008 for reimage

https://gerrit.wikimedia.org/r/399351

ops-monitoring-bot added a comment.Dec 20 2017, 8:19 AM

Completed auto-reimage of hosts:

['dbproxy1005.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Dec 20 2017, 8:23 AM

Change 399351 merged by Jcrespo:
[operations/puppet@production] dbproxy: Prepare dbproxy1008 for reimage

https://gerrit.wikimedia.org/r/399351

ops-monitoring-bot added a comment.Dec 20 2017, 8:25 AM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1008.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712200824_jynus_23607.log.

jcrespo added a comment.Dec 20 2017, 8:26 AM

All proxies reimaged except the active ones:

dbproxy1002.eqiad.wmnet
dbproxy1003.eqiad.wmnet
dbproxy1006.eqiad.wmnet
dbproxy1009.eqiad.wmnet
dbproxy1010.eqiad.wmnet
dbproxy1011.eqiad.wmnet
ops-monitoring-bot added a comment.Dec 20 2017, 8:40 AM

Completed auto-reimage of hosts:

['dbproxy1007.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 20 2017, 9:08 AM

Completed auto-reimage of hosts:

['dbproxy1008.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Dec 20 2017, 9:10 AM

Change 399359 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] haproxy: Update haproxy systemd.unit to that of stretch

https://gerrit.wikimedia.org/r/399359

gerritbot added a comment.Dec 20 2017, 9:25 AM

Change 399359 merged by Jcrespo:
[operations/puppet@production] haproxy: Update haproxy systemd.unit to that of stretch

https://gerrit.wikimedia.org/r/399359

jcrespo mentioned this in T183469: Setup newer machines and replace all old misc (m*) and x1 eqiad machines.Dec 21 2017, 2:59 PM

Very related (blocked on failovers of): T183469

jcrespo changed the task status from Open to Stalled.Jan 10 2018, 6:20 PM
jcrespo removed jcrespo as the assignee of this task.
• Marostegui moved this task from In progress to Pending comment on the DBA board.Feb 9 2018, 12:18 PM
jcrespo changed the task status from Stalled to Open.Mar 14 2018, 3:05 PM
jcrespo claimed this task.
jcrespo moved this task from Pending comment to In progress on the DBA board.
ops-monitoring-bot added a comment.Mar 14 2018, 3:08 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803141507_jynus_29676.log.

ops-monitoring-bot added a comment.Mar 14 2018, 3:09 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1006.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803141508_jynus_29917.log.

ops-monitoring-bot added a comment.Mar 14 2018, 3:34 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1006.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803141534_jynus_4121.log.

ops-monitoring-bot added a comment.Mar 14 2018, 3:41 PM

Completed auto-reimage of hosts:

['dbproxy1002.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 14 2018, 4:07 PM

Completed auto-reimage of hosts:

['dbproxy1006.eqiad.wmnet']

and were ALL successful.

jcrespo added a comment.Mar 14 2018, 5:58 PM

All proxies are now on stretch except the ones for labsdbs (10 and 11).

jcrespo removed jcrespo as the assignee of this task.Mar 14 2018, 5:58 PM
jcrespo moved this task from In progress to Pending comment on the DBA board.
gerritbot added a comment.Mar 16 2018, 8:43 AM

Change 419985 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] dbproxy: Change m4-master from dbproxy1009 to dbproxy1004

https://gerrit.wikimedia.org/r/419985

gerritbot added a comment.Mar 16 2018, 9:14 AM

Change 419985 merged by Jcrespo:
[operations/dns@master] dbproxy: Change m4-master from dbproxy1009 to dbproxy1004

https://gerrit.wikimedia.org/r/419985

gerritbot added a comment.Mar 16 2018, 10:35 AM

Change 419999 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] Switchover temporarily wikireplica-web to dbproxy1010

https://gerrit.wikimedia.org/r/419999

gerritbot added a comment.Mar 16 2018, 10:40 AM

Change 419999 merged by Jcrespo:
[operations/dns@master] Switchover temporarily wikireplica-web to dbproxy1010

https://gerrit.wikimedia.org/r/419999

gerritbot added a comment.Mar 16 2018, 1:23 PM

Change 420021 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] cloud-dns: Point wikireplica-web to dbproxy10010

https://gerrit.wikimedia.org/r/420021

gerritbot added a comment.Mar 16 2018, 1:25 PM

Change 420022 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy1011: Adapt syntax to strech and fix socket location

https://gerrit.wikimedia.org/r/420022

gerritbot added a comment.Mar 16 2018, 1:54 PM

Change 420021 merged by Jcrespo:
[operations/puppet@production] cloud-dns: Point wikireplica-web to dbproxy1010

https://gerrit.wikimedia.org/r/420021

aborrero subscribed.Mar 16 2018, 2:16 PM

I just update wikireplica DNS records:

root@labcontrol1001:~# /usr/local/sbin/wikireplica_dns --aliases -v --zone web.db.svc.eqiad.wmflabs.
2018-03-16T14:02:41Z keystoneclient.utils WARNING : list takes at most 1 positional argument (2 given)
2018-03-16T14:02:41Z mwopenstackclients.DnsManager INFO    : Updating s8.web.db.svc.eqiad.wmflabs.
2018-03-16T14:02:41Z py.warnings  WARNING : /usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:97: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
2018-03-16T14:02:42Z mwopenstackclients.DnsManager INFO    : Updating s3.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:17Z mwopenstackclients.DnsManager INFO    : Updating s2.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:23Z mwopenstackclients.DnsManager INFO    : Updating s1.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:24Z mwopenstackclients.DnsManager INFO    : Updating s7.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:28Z mwopenstackclients.DnsManager INFO    : Updating s6.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:29Z mwopenstackclients.DnsManager INFO    : Updating s5.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:30Z mwopenstackclients.DnsManager INFO    : Updating s4.web.db.svc.eqiad.wmflabs.
ops-monitoring-bot added a comment.Mar 16 2018, 2:49 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1011.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803161449_jynus_16161.log.

gerritbot added a comment.Mar 16 2018, 2:58 PM

Change 420022 merged by Jcrespo:
[operations/puppet@production] dbproxy1011: Adapt syntax to strech and fix socket location

https://gerrit.wikimedia.org/r/420022

ops-monitoring-bot added a comment.Mar 16 2018, 3:23 PM

Completed auto-reimage of hosts:

['dbproxy1011.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Mar 16 2018, 3:48 PM

Change 420055 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] cloud-dns: Point all wikireplicas to dbproxy1011

https://gerrit.wikimedia.org/r/420055

gerritbot added a comment.Mar 16 2018, 4:01 PM

Change 420058 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] Switchover temporarily wikireplica-web to dbproxy1011

https://gerrit.wikimedia.org/r/420058

gerritbot added a comment.Mar 16 2018, 4:21 PM

Change 420058 merged by Jcrespo:
[operations/dns@master] Switchover temporarily wikireplica-web to dbproxy1011

https://gerrit.wikimedia.org/r/420058

gerritbot added a comment.Mar 16 2018, 4:23 PM

Change 420055 merged by Jcrespo:
[operations/puppet@production] cloud-dns: Point all wikireplicas to dbproxy1011

https://gerrit.wikimedia.org/r/420055

gerritbot added a comment.Mar 16 2018, 4:57 PM

Change 420071 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] dbproxy-wikirreplicas: Revert to the original proxy configuration

https://gerrit.wikimedia.org/r/420071

gerritbot added a comment.Mar 16 2018, 5:27 PM

Change 420079 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Update dbproxy1010 to the latest socket path and config

https://gerrit.wikimedia.org/r/420079

gerritbot added a comment.Mar 16 2018, 5:27 PM

Change 420079 merged by Jcrespo:
[operations/puppet@production] dbproxy: Update dbproxy1010 to the latest socket path and config

https://gerrit.wikimedia.org/r/420079

ops-monitoring-bot added a comment.Mar 16 2018, 5:33 PM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1010.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803161733_jynus_20625.log.

gerritbot added a comment.Mar 16 2018, 6:06 PM

Change 420071 merged by Jcrespo:
[operations/dns@master] dbproxy-wikirreplicas: Revert to the original proxy configuration

https://gerrit.wikimedia.org/r/420071

ops-monitoring-bot added a comment.Mar 16 2018, 6:07 PM

Completed auto-reimage of hosts:

['dbproxy1010.eqiad.wmnet']

and were ALL successful.

jcrespo closed this task as Resolved.Mar 16 2018, 6:20 PM
jcrespo claimed this task.

With todays reimage/restart of dbproxy1009, 10 and 11, this should be now 100% done.

• Marostegui added a comment.Mar 16 2018, 6:48 PM

Very nice work! :-)

• Marostegui awarded a token.Mar 16 2018, 6:49 PM
gerritbot added a comment.Mar 28 2018, 11:23 AM

Change 422386 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] haproxy: Remove older templates (haproxy<1.7)

https://gerrit.wikimedia.org/r/422386

gerritbot added a comment.Mar 28 2018, 12:40 PM

Change 422386 merged by Jcrespo:
[operations/puppet@production] haproxy: Remove older templates (haproxy<1.7)

https://gerrit.wikimedia.org/r/422386

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits