Page MenuHomePhabricator
Create Task
Maniphest T147194

reimage maps-test* servers
Closed, ResolvedPublic
Actions

Description

Disks have been filled on maps-test* servers (see T146848). This most probably result in data corruption for cassandra and / or postgresql. Those servers needs to be reimaged anyway to align them with other maps servers and the improved automation that has been put in place on those.

Details

SubjectRepoBranchLines +/-
maps / postgresql: use replication user for monitoringoperations/puppetproduction+8 -10
maps / postgresql: corrected hiera key for replication passwordoperations/puppetproduction+1 -1
maps / postgresql: corrected hiera key for replication passwordoperations/puppetproduction+1 -2
maps / postgresql: new configuration format for slavesoperations/puppetproduction+35 -22
Maps - cleanup postgres user creationoperations/puppetproduction+53 -42
maps - adding dummy monitoring password for postgresqllabs/privatemaster+1 -0
maps - change structure of slaves according to https://gerrit.wikimedia.org/r/#/c/315271/labs/privatemaster+1 -6
maps - adding dummy passwords for postgresql monitoring and replicationlabs/privatemaster+2 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Gehel created this task.Oct 3 2016, 1:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 3 2016, 1:05 PM
Stashbot subscribed.Oct 3 2016, 1:09 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-03T13:09:17Z] <gehel> shutting down services on maps-test* servers prior to reimage -T147194

Stashbot added a comment.Oct 3 2016, 1:13 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-03T13:13:41Z] <gehel> reimage of maps-test2001 - T147194

ops-monitoring-bot subscribed.Oct 3 2016, 2:12 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610031410_gehel_29303.log.

ops-monitoring-bot added a comment.Oct 3 2016, 2:50 PM

Completed auto-reimage of hosts:

['maps-test2001.codfw.wmnet']

Those hosts were successful:

['maps-test2001.codfw.wmnet']
ops-monitoring-bot added a comment.Oct 5 2016, 2:50 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610051449_gehel_18275.log.

Gehel moved this task from Backlog to In progress on the Maps-Sprint board.Oct 5 2016, 2:57 PM
ops-monitoring-bot added a comment.Oct 5 2016, 3:24 PM

Completed auto-reimage of hosts:

['maps-test2002.codfw.wmnet']

Those hosts were successful:

['maps-test2002.codfw.wmnet']
Gehel mentioned this in T147334: Unhandled(?) exceptions in scap3.Oct 5 2016, 8:22 PM
Gehel triaged this task as High priority.Oct 5 2016, 8:43 PM
Stashbot added a comment.Oct 7 2016, 8:53 AM

Mentioned in SAL (#wikimedia-operations) [2016-10-07T08:53:48Z] <gehel> reimaging maps-test2003 - T147194

ops-monitoring-bot added a comment.Oct 7 2016, 8:55 AM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2003.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610070855_gehel_10452.log.

ops-monitoring-bot added a comment.Oct 7 2016, 9:17 AM

Completed auto-reimage of hosts:

['maps-test2003.codfw.wmnet']

Those hosts were successful:

[]
Stashbot added a comment.Oct 7 2016, 9:20 AM

Mentioned in SAL (#wikimedia-operations) [2016-10-07T09:20:08Z] <gehel> reimaging maps-test2004 - T147194

ops-monitoring-bot added a comment.Oct 7 2016, 9:21 AM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610070920_gehel_14871.log.

ops-monitoring-bot added a comment.Oct 7 2016, 9:48 AM

Completed auto-reimage of hosts:

['maps-test2004.codfw.wmnet']

Those hosts were successful:

[]
Stashbot added a comment.Oct 7 2016, 4:25 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-07T16:25:04Z] <gehel> reimage maps-test2001 - T147194

ops-monitoring-bot added a comment.Oct 7 2016, 4:26 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610071625_gehel_13919.log.

ops-monitoring-bot added a comment.Oct 7 2016, 5:58 PM

Completed auto-reimage of hosts:

['maps-test2001.codfw.wmnet']

Those hosts were successful:

['maps-test2001.codfw.wmnet']
Stashbot added a comment.Oct 10 2016, 1:12 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-10T13:12:26Z] <gehel> reimage maps-test2002 - T147194

ops-monitoring-bot added a comment.Oct 10 2016, 1:12 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610101311_gehel_19322.log.

ops-monitoring-bot added a comment.Oct 10 2016, 1:57 PM

Completed auto-reimage of hosts:

['maps-test2002.codfw.wmnet']

Those hosts were successful:

[]
Stashbot added a comment.Oct 10 2016, 2:04 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-10T14:03:31Z] <gehel> reimage maps-test200[34] - T147194

ops-monitoring-bot added a comment.Oct 10 2016, 2:04 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2003.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610101403_gehel_27871.log.

ops-monitoring-bot added a comment.Oct 10 2016, 2:04 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610101403_gehel_27863.log.

ops-monitoring-bot added a comment.Oct 10 2016, 2:09 PM

Completed auto-reimage of hosts:

['maps-test2004.codfw.wmnet']

Those hosts were successful:

[]
ops-monitoring-bot added a comment.Oct 10 2016, 2:17 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610101416_gehel_29045.log.

ops-monitoring-bot added a comment.Oct 10 2016, 2:40 PM

Completed auto-reimage of hosts:

['maps-test2003.codfw.wmnet']

Those hosts were successful:

['maps-test2003.codfw.wmnet']
ops-monitoring-bot added a comment.Oct 10 2016, 2:53 PM

Completed auto-reimage of hosts:

['maps-test2004.codfw.wmnet']

Those hosts were successful:

['maps-test2004.codfw.wmnet']
gerritbot subscribed.Oct 11 2016, 2:24 PM

Change 315271 had a related patch set uploaded (by Gehel):
Maps - cleanup postgres user creation

https://gerrit.wikimedia.org/r/315271

gerritbot added a project: Patch-For-Review.Oct 11 2016, 2:24 PM
Gehel created subtask T148031: Maps - error when doing initial tiles generation: "Error: could not create converter for SQL_ASCII"".Oct 13 2016, 1:23 PM
gerritbot added a comment.Oct 14 2016, 2:20 PM

Change 315959 had a related patch set uploaded (by Gehel):
maps - adding dummy passwords for postgresql monitoring and replication

https://gerrit.wikimedia.org/r/315959

gerritbot added a comment.Oct 14 2016, 2:29 PM

Change 315959 merged by Gehel:
maps - adding dummy passwords for postgresql monitoring and replication

https://gerrit.wikimedia.org/r/315959

gerritbot added a comment.Oct 18 2016, 9:27 AM

Change 316536 had a related patch set uploaded (by Gehel):
maps - change structure of slaves according to https://gerrit.wikimedia.org/r/#/c/315271/

https://gerrit.wikimedia.org/r/316536

gerritbot added a comment.Oct 18 2016, 9:28 AM

Change 316536 merged by Gehel:
maps - change structure of slaves according to https://gerrit.wikimedia.org/r/#/c/315271/

https://gerrit.wikimedia.org/r/316536

gerritbot added a comment.Oct 18 2016, 11:48 AM

Change 316549 had a related patch set uploaded (by Gehel):
maps - adding dummy monitoring password for postgresql

https://gerrit.wikimedia.org/r/316549

gerritbot added a comment.Oct 18 2016, 11:54 AM

Change 316549 merged by Gehel:
maps - adding dummy monitoring password for postgresql

https://gerrit.wikimedia.org/r/316549

Gehel added a parent task: T137939: Increase frequency of OSM replication.Oct 19 2016, 6:38 PM
ops-monitoring-bot added a comment.Oct 21 2016, 4:01 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610211601_gehel_23846.log.

ops-monitoring-bot added a comment.Oct 21 2016, 4:02 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610211601_gehel_24043.log.

ops-monitoring-bot added a comment.Oct 21 2016, 4:33 PM

Completed auto-reimage of hosts:

['maps-test2001.codfw.wmnet']

Of which those FAILED:

set(['maps-test2001.codfw.wmnet'])
ops-monitoring-bot added a comment.Oct 21 2016, 4:42 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610211642_gehel_529.log.

ops-monitoring-bot added a comment.Oct 21 2016, 4:42 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2003.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610211642_gehel_662.log.

ops-monitoring-bot added a comment.Oct 21 2016, 4:42 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610211642_gehel_723.log.

ops-monitoring-bot added a comment.Oct 21 2016, 4:47 PM

Completed auto-reimage of hosts:

['maps-test2002.codfw.wmnet']

Of which those FAILED:

set(['maps-test2002.codfw.wmnet'])
ops-monitoring-bot added a comment.Oct 21 2016, 4:48 PM

Completed auto-reimage of hosts:

['maps-test2003.codfw.wmnet']

Of which those FAILED:

set(['maps-test2003.codfw.wmnet'])
ops-monitoring-bot added a comment.Oct 21 2016, 4:48 PM

Completed auto-reimage of hosts:

['maps-test2004.codfw.wmnet']

Of which those FAILED:

set(['maps-test2004.codfw.wmnet'])
ops-monitoring-bot added a comment.Oct 21 2016, 5:11 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610211709_gehel_7440.log.

ops-monitoring-bot added a comment.Oct 21 2016, 5:14 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2003.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610211713_gehel_8323.log.

ops-monitoring-bot added a comment.Oct 21 2016, 5:21 PM

Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201610211720_gehel_9640.log.

ops-monitoring-bot added a comment.Oct 21 2016, 5:36 PM

Completed auto-reimage of hosts:

['maps-test2003.codfw.wmnet']

Of which those FAILED:

set(['maps-test2003.codfw.wmnet'])
ops-monitoring-bot added a comment.Oct 21 2016, 5:47 PM

Completed auto-reimage of hosts:

['maps-test2002.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Oct 21 2016, 5:51 PM

Completed auto-reimage of hosts:

['maps-test2004.codfw.wmnet']

Of which those FAILED:

set(['maps-test2004.codfw.wmnet'])
gerritbot added a comment.Oct 27 2016, 12:40 PM

Change 315271 merged by Gehel:
Maps - cleanup postgres user creation

https://gerrit.wikimedia.org/r/315271

gerritbot added a comment.Oct 27 2016, 12:50 PM

Change 318282 had a related patch set uploaded (by Gehel):
maps / postgresql: new configuration format for slaves

https://gerrit.wikimedia.org/r/318282

gerritbot added a comment.Oct 27 2016, 12:51 PM

Change 318282 merged by Gehel:
maps / postgresql: new configuration format for slaves

https://gerrit.wikimedia.org/r/318282

gerritbot added a comment.Oct 27 2016, 12:56 PM

Change 318283 had a related patch set uploaded (by Gehel):
maps / postgresql: corrected hiera key for replication password

https://gerrit.wikimedia.org/r/318283

gerritbot added a comment.Oct 27 2016, 12:57 PM

Change 318283 merged by Gehel:
maps / postgresql: corrected hiera key for replication password

https://gerrit.wikimedia.org/r/318283

gerritbot added a comment.Oct 27 2016, 12:59 PM

Change 318286 had a related patch set uploaded (by Gehel):
maps / postgresql: corrected hiera key for replication password

https://gerrit.wikimedia.org/r/318286

gerritbot added a comment.Oct 27 2016, 1:00 PM

Change 318286 merged by Gehel:
maps / postgresql: corrected hiera key for replication password

https://gerrit.wikimedia.org/r/318286

Stashbot added a comment.Oct 27 2016, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-27T13:33:39Z] <gehel> postgres replication checks in error after deployment of https://gerrit.wikimedia.org/r/#/c/315271/ (T147194) - replication is working, only check is failing - icinga is silenced

Stashbot added a comment.Oct 27 2016, 1:34 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-27T13:34:05Z] <gehel> maps / postgres replication checks in error after deployment of https://gerrit.wikimedia.org/r/#/c/315271/ (T147194) - replication is working, only check is failing - icinga is silenced

gerritbot added a comment.Oct 28 2016, 10:00 AM

Change 318511 had a related patch set uploaded (by Gehel):
maps / postgresql: use replication user for monitoring

https://gerrit.wikimedia.org/r/318511

gerritbot added a comment.Oct 28 2016, 11:49 AM

Change 318511 merged by Gehel:
maps / postgresql: use replication user for monitoring

https://gerrit.wikimedia.org/r/318511

Gehel closed subtask T148031: Maps - error when doing initial tiles generation: "Error: could not create converter for SQL_ASCII"" as Resolved.Oct 28 2016, 4:27 PM
Gehel closed this task as Resolved.Oct 31 2016, 1:41 PM

Re-image is complete, initial tile generation is in progress and working fine, but we are going to switch it to Cassandra. I already mark this task as resovled.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits