Page MenuHomePhabricator

Reimage and upgrade to stretch all dbproxies
Closed, ResolvedPublic

Description

Current proxies are on jessie and have bad configuration (socket location, lack of good firewall configuration). We need to upgrade them before database failover.

Event Timeline

jcrespo triaged this task as Medium priority.Dec 19 2017, 2:11 PM
jcrespo created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 19 2017, 2:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
jcrespo moved this task from Triage to In progress on the DBA board.

Change 399164 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Preparing reimage of dbproxy1001 and setup proxy firewall

https://gerrit.wikimedia.org/r/399164

jcrespo renamed this task from Reimage and upgrade to stretch all proxies to Reimage and upgrade to stretch all dbproxies.Dec 19 2017, 2:26 PM

Change 399164 merged by Jcrespo:
[operations/puppet@production] mariadb: Preparing reimage of dbproxy1001 and setup proxy firewall

https://gerrit.wikimedia.org/r/399164

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1001.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712191436_jynus_20842.log.

dbproxy1001 has been successfully reimaged, which joins the already upgraded to stretch dbproxy1004 and dbproxy1009 (although these one have to yet be reconfigured its socket and firewall). All others are on jessie still, have to be reimaged one by one.

Change 399347 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Preparing to reimage dbproxy1004

https://gerrit.wikimedia.org/r/399347

Change 399347 merged by Jcrespo:
[operations/puppet@production] dbproxy: Change socket location for dbproxy1004

https://gerrit.wikimedia.org/r/399347

Change 399349 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Preparing dbproxy1005 for reimage

https://gerrit.wikimedia.org/r/399349

Change 399349 merged by Jcrespo:
[operations/puppet@production] dbproxy: Preparing dbproxy1005 for reimage

https://gerrit.wikimedia.org/r/399349

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1005.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712200736_jynus_12971.log.

Change 399350 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Fix dbproxy1005 and prepare dbproxy1007 for reimage

https://gerrit.wikimedia.org/r/399350

Change 399350 merged by Jcrespo:
[operations/puppet@production] dbproxy: Fix dbproxy1005 and prepare dbproxy1007 for reimage

https://gerrit.wikimedia.org/r/399350

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1007.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712200756_jynus_16585.log.

Change 399351 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Prepare dbproxy1008 for reimage

https://gerrit.wikimedia.org/r/399351

Completed auto-reimage of hosts:

['dbproxy1005.eqiad.wmnet']

and were ALL successful.

Change 399351 merged by Jcrespo:
[operations/puppet@production] dbproxy: Prepare dbproxy1008 for reimage

https://gerrit.wikimedia.org/r/399351

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1008.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712200824_jynus_23607.log.

All proxies reimaged except the active ones:

dbproxy1002.eqiad.wmnet
dbproxy1003.eqiad.wmnet
dbproxy1006.eqiad.wmnet
dbproxy1009.eqiad.wmnet
dbproxy1010.eqiad.wmnet
dbproxy1011.eqiad.wmnet

Completed auto-reimage of hosts:

['dbproxy1007.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['dbproxy1008.eqiad.wmnet']

and were ALL successful.

Change 399359 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] haproxy: Update haproxy systemd.unit to that of stretch

https://gerrit.wikimedia.org/r/399359

Change 399359 merged by Jcrespo:
[operations/puppet@production] haproxy: Update haproxy systemd.unit to that of stretch

https://gerrit.wikimedia.org/r/399359

jcrespo changed the task status from Open to Stalled.Jan 10 2018, 6:20 PM
jcrespo removed jcrespo as the assignee of this task.
Marostegui moved this task from In progress to Next on the DBA board.Feb 9 2018, 12:18 PM
jcrespo changed the task status from Stalled to Open.Mar 14 2018, 3:05 PM
jcrespo claimed this task.
jcrespo moved this task from Next to In progress on the DBA board.

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803141507_jynus_29676.log.

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1006.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803141508_jynus_29917.log.

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1006.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803141534_jynus_4121.log.

Completed auto-reimage of hosts:

['dbproxy1002.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['dbproxy1006.eqiad.wmnet']

and were ALL successful.

All proxies are now on stretch except the ones for labsdbs (10 and 11).

jcrespo removed jcrespo as the assignee of this task.Mar 14 2018, 5:58 PM
jcrespo moved this task from In progress to Next on the DBA board.

Change 419985 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] dbproxy: Change m4-master from dbproxy1009 to dbproxy1004

https://gerrit.wikimedia.org/r/419985

Change 419985 merged by Jcrespo:
[operations/dns@master] dbproxy: Change m4-master from dbproxy1009 to dbproxy1004

https://gerrit.wikimedia.org/r/419985

Change 419999 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] Switchover temporarily wikireplica-web to dbproxy1010

https://gerrit.wikimedia.org/r/419999

Change 419999 merged by Jcrespo:
[operations/dns@master] Switchover temporarily wikireplica-web to dbproxy1010

https://gerrit.wikimedia.org/r/419999

Change 420021 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] cloud-dns: Point wikireplica-web to dbproxy10010

https://gerrit.wikimedia.org/r/420021

Change 420022 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy1011: Adapt syntax to strech and fix socket location

https://gerrit.wikimedia.org/r/420022

Change 420021 merged by Jcrespo:
[operations/puppet@production] cloud-dns: Point wikireplica-web to dbproxy1010

https://gerrit.wikimedia.org/r/420021

I just update wikireplica DNS records:

root@labcontrol1001:~# /usr/local/sbin/wikireplica_dns --aliases -v --zone web.db.svc.eqiad.wmflabs.
2018-03-16T14:02:41Z keystoneclient.utils WARNING : list takes at most 1 positional argument (2 given)
2018-03-16T14:02:41Z mwopenstackclients.DnsManager INFO    : Updating s8.web.db.svc.eqiad.wmflabs.
2018-03-16T14:02:41Z py.warnings  WARNING : /usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:97: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
2018-03-16T14:02:42Z mwopenstackclients.DnsManager INFO    : Updating s3.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:17Z mwopenstackclients.DnsManager INFO    : Updating s2.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:23Z mwopenstackclients.DnsManager INFO    : Updating s1.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:24Z mwopenstackclients.DnsManager INFO    : Updating s7.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:28Z mwopenstackclients.DnsManager INFO    : Updating s6.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:29Z mwopenstackclients.DnsManager INFO    : Updating s5.web.db.svc.eqiad.wmflabs.
2018-03-16T14:07:30Z mwopenstackclients.DnsManager INFO    : Updating s4.web.db.svc.eqiad.wmflabs.

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1011.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803161449_jynus_16161.log.

Change 420022 merged by Jcrespo:
[operations/puppet@production] dbproxy1011: Adapt syntax to strech and fix socket location

https://gerrit.wikimedia.org/r/420022

Completed auto-reimage of hosts:

['dbproxy1011.eqiad.wmnet']

and were ALL successful.

Change 420055 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] cloud-dns: Point all wikireplicas to dbproxy1011

https://gerrit.wikimedia.org/r/420055

Change 420058 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] Switchover temporarily wikireplica-web to dbproxy1011

https://gerrit.wikimedia.org/r/420058

Change 420058 merged by Jcrespo:
[operations/dns@master] Switchover temporarily wikireplica-web to dbproxy1011

https://gerrit.wikimedia.org/r/420058

Change 420055 merged by Jcrespo:
[operations/puppet@production] cloud-dns: Point all wikireplicas to dbproxy1011

https://gerrit.wikimedia.org/r/420055

Change 420071 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] dbproxy-wikirreplicas: Revert to the original proxy configuration

https://gerrit.wikimedia.org/r/420071

Change 420079 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Update dbproxy1010 to the latest socket path and config

https://gerrit.wikimedia.org/r/420079

Change 420079 merged by Jcrespo:
[operations/puppet@production] dbproxy: Update dbproxy1010 to the latest socket path and config

https://gerrit.wikimedia.org/r/420079

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['dbproxy1010.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803161733_jynus_20625.log.

Change 420071 merged by Jcrespo:
[operations/dns@master] dbproxy-wikirreplicas: Revert to the original proxy configuration

https://gerrit.wikimedia.org/r/420071

Completed auto-reimage of hosts:

['dbproxy1010.eqiad.wmnet']

and were ALL successful.

jcrespo closed this task as Resolved.Mar 16 2018, 6:20 PM
jcrespo claimed this task.

With todays reimage/restart of dbproxy1009, 10 and 11, this should be now 100% done.

Very nice work! :-)

Change 422386 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] haproxy: Remove older templates (haproxy<1.7)

https://gerrit.wikimedia.org/r/422386

Change 422386 merged by Jcrespo:
[operations/puppet@production] haproxy: Remove older templates (haproxy<1.7)

https://gerrit.wikimedia.org/r/422386