Page MenuHomePhabricator

Thumbor upgrade to stretch plan
Closed, ResolvedPublic

Description

  • puppet changes to pull from component/thumbor repo
  • test changes on deployment-prep

Testing

  • override thumbor config on a Stretch host to avoid saving output to Swift
  • run a script locally on the Stretch machine that will compare output of random thumbnails between a prod Jessie host and the Stretch one, looking at DSSIM scores

Roll upgrade to stretch:
--> use raid1-lvm-ext4-srv-dualboot.cfg

  • thumbor2002
  • thumbor2001
  • thumbor2003
  • thumbor2004
  • thumbor1001
  • thumbor1002
  • thumbor1003
  • thumbor1004

Event Timeline

jijiki created this task.Jan 24 2019, 4:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2019, 4:10 PM
jijiki updated the task description. (Show Details)Feb 6 2019, 4:04 PM
CDanis added a subscriber: CDanis.Feb 6 2019, 4:05 PM
jijiki moved this task from Backlog/Radar to St on the User-jijiki board.Feb 8 2019, 10:55 AM
jijiki renamed this task from Thumbor upgrade plan (TBA) to Thumbor upgrade to stretch plan.Feb 12 2019, 11:31 AM
jijiki triaged this task as Normal priority.
jijiki added projects: Thumbor, Operations, serviceops.
jijiki updated the task description. (Show Details)
jijiki added subscribers: Muehlenhoff, fgiunchedi, Gilles, Joe.

Change 490405 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] thumbor: add support for debian stretch

https://gerrit.wikimedia.org/r/490405

jijiki moved this task from St to In Progress on the User-jijiki board.Feb 14 2019, 11:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-14T11:02:50Z] <jijiki> Disabling puppet on thumbor* servers - T214597

Change 490405 merged by Effie Mouzeli:
[operations/puppet@production] thumbor: add support for debian stretch

https://gerrit.wikimedia.org/r/490405

Change 490583 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] thumbor: add support for debian stretch

https://gerrit.wikimedia.org/r/490583

Change 490583 merged by Effie Mouzeli:
[operations/puppet@production] thumbor: add support for debian stretch

https://gerrit.wikimedia.org/r/490583

Mentioned in SAL (#wikimedia-operations) [2019-02-14T14:12:38Z] <jijiki> Enabling puppet on thumbor* servers - T214597

jijiki updated the task description. (Show Details)Feb 14 2019, 2:34 PM

Change 490610 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Upgrade thumbor2002 to stretch

https://gerrit.wikimedia.org/r/490610

Change 490610 merged by Effie Mouzeli:
[operations/puppet@production] Upgrade thumbor2002 to stretch

https://gerrit.wikimedia.org/r/490610

Mentioned in SAL (#wikimedia-operations) [2019-02-18T10:54:02Z] <jijiki> Reimaging thumbor2002 to stretch - T214597

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181130_jiji_3443_thumbor2002_codfw_wmnet.log.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']

Of which those FAILED:

['thumbor2002.codfw.wmnet']

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181139_jiji_6639_thumbor2002_codfw_wmnet.log.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']

Of which those FAILED:

['thumbor2002.codfw.wmnet']

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181141_jiji_6965_thumbor2002_codfw_wmnet.log.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']

Of which those FAILED:

['thumbor2002.codfw.wmnet']

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181144_jiji_8607_thumbor2002_codfw_wmnet.log.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']

Of which those FAILED:

['thumbor2002.codfw.wmnet']

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902181215_jiji_16296_thumbor2002_codfw_wmnet.log.

Let me know when the host is ready for testing

@Gilles It looks like we have some issues with <code>thumbor2002</code>, we are investigating if we can continue the upgrade with other host.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']

Of which those FAILED:

['thumbor2002.codfw.wmnet']

Mentioned in SAL (#wikimedia-operations) [2019-02-18T13:25:48Z] <jijiki> Depooling thumbor1004 to check if the rest of our hosts can handle the load without it - T214597

Change 491277 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Upgrade thumbor1004 to stretch

https://gerrit.wikimedia.org/r/491277

Change 491277 merged by Effie Mouzeli:
[operations/puppet@production] Upgrade thumbor1004 to stretch

https://gerrit.wikimedia.org/r/491277

Mentioned in SAL (#wikimedia-operations) [2019-02-18T17:49:54Z] <jijiki> Reimaging thumbor1004 to stretc - T214597

Mentioned in SAL (#wikimedia-operations) [2019-02-18T17:49:59Z] <jijiki> Reimaging thumbor1004 to stretch - T214597

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['thumbor1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201902181750_jiji_246664.log.

Completed auto-reimage of hosts:

['thumbor1004.eqiad.wmnet']

Of which those FAILED:

['thumbor1004.eqiad.wmnet']

Mentioned in SAL (#wikimedia-operations) [2019-02-19T15:47:29Z] <jijiki> Reimaging thumbor2002 to stretch - T214597

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

['thumbor2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201902191553_jiji_10924.log.

Completed auto-reimage of hosts:

['thumbor2002.codfw.wmnet']

Of which those FAILED:

['thumbor2002.codfw.wmnet']
Gilles updated the task description. (Show Details)Feb 19 2019, 8:21 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-22T10:15:29Z] <jijiki> Pooling thumbor1004 after upgrade - T214597

Mentioned in SAL (#wikimedia-operations) [2019-02-22T11:32:53Z] <jijiki> Pooling thumbor2002 after upgrade - T214597

Mentioned in SAL (#wikimedia-operations) [2019-02-25T10:16:19Z] <jijiki> Depooling thumbor1001 to reimage - T214597

Change 492642 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Upgrade all Thumbor servers to stretch

https://gerrit.wikimedia.org/r/492642

Change 492642 merged by Effie Mouzeli:
[operations/puppet@production] Upgrade all Thumbor servers to stretch

https://gerrit.wikimedia.org/r/492642

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

thumbor1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902251118_jiji_23066_thumbor1001_eqiad_wmnet.log.

Mentioned in SAL (#wikimedia-operations) [2019-02-25T11:19:38Z] <jijiki> Reimageing thumbor1001 - T214597

Completed auto-reimage of hosts:

['thumbor1001.eqiad.wmnet']

Of which those FAILED:

['thumbor1001.eqiad.wmnet']

Mentioned in SAL (#wikimedia-operations) [2019-02-25T16:21:58Z] <jijiki> Depooling and reimaging thumbor2001 - T214597

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

['thumbor2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201902251627_jiji_595.log.

Mentioned in SAL (#wikimedia-operations) [2019-02-25T17:59:35Z] <jijiki> Depooling and reimaging thumbor1002 to stretch - T214597

Completed auto-reimage of hosts:

['thumbor2001.codfw.wmnet']

and were ALL successful.

jijiki updated the task description. (Show Details)Feb 25 2019, 6:19 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

thumbor1002.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902251825_jiji_93856_thumbor1002_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['thumbor1002.eqiad.wmnet']

and were ALL successful.

jijiki updated the task description. (Show Details)Feb 26 2019, 6:43 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-26T06:50:52Z] <jijiki> Depool and reimage thumbor1003 and thumbor2003 - T214597

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

thumbor1003.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902260705_jiji_200381_thumbor1003_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2003.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902260708_jiji_25581_thumbor2003_codfw_wmnet.log.

Completed auto-reimage of hosts:

['thumbor2003.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-02-26T08:07:25Z] <jijiki> Pooling thumbor2003 - T214597

Mentioned in SAL (#wikimedia-operations) [2019-02-26T08:08:27Z] <jijiki> Depool and reimage thumbor2004 - T214597

Script wmf-auto-reimage was launched by jiji on cumin2001.codfw.wmnet for hosts:

thumbor2004.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902260827_jiji_15458_thumbor2004_codfw_wmnet.log.

jijiki updated the task description. (Show Details)Feb 26 2019, 8:43 AM

Completed auto-reimage of hosts:

['thumbor1003.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['thumbor2004.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2019-02-26T11:12:08Z] <jijiki> Pooling thumbor2004 - T214597

jijiki updated the task description. (Show Details)Feb 26 2019, 12:16 PM
jijiki closed this task as Resolved.Feb 26 2019, 12:19 PM

All servers have been upgraded to stretch, next episode on T216815 🍾