Page MenuHomePhabricator
Create Task
Maniphest T224583

Migrate labstore1006/1007 to Stretch/Buster
Closed, ResolvedPublic
Actions

Description

These are currently running jessie:

  • labstore1006.wikimedia.org
  • labstore1007.wikimedia.org

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.May 29 2019, 1:32 PM
MoritzMuehlenhoff mentioned this in T224549: Track remaining jessie systems in production.
• Bstorm edited projects, added cloud-services-team (Kanban); removed cloud-services-team.May 31 2019, 9:08 PM
• Bstorm subscribed.
ArielGlenn triaged this task as Medium priority.Jun 11 2019, 7:53 AM
ArielGlenn subscribed.
bd808 added a parent task: T232677: Remove support for Debian Jessie in Cloud Services.Oct 2 2019, 9:14 PM
bd808 added a project: Cloud-VPS (Debian Jessie Deprecation).Oct 24 2019, 2:51 AM
bd808 moved this task from Backlog to Hardware on the Cloud-VPS (Debian Jessie Deprecation) board.Oct 24 2019, 2:52 AM
gerritbot added a comment.Mar 12 2020, 12:16 AM

Change 579095 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] dumps-distribution: move all traffic to labstore1006

https://gerrit.wikimedia.org/r/579095

gerritbot added a project: Patch-For-Review.Mar 12 2020, 12:16 AM
• Bstorm claimed this task.Mar 12 2020, 12:17 AM
• Bstorm moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
• Bstorm added a subscriber: elukey.Mar 12 2020, 12:21 AM

@elukey Are there any special considerations for Kerberos/HDFS/Hadoop stuff if I aim to upgrade this pair in place to Stretch and then possibly Buster?

elukey added a comment.EditedMar 12 2020, 7:40 AM

@Bstorm the only thing to keep in mind is that Buster ships with java 11 only, and hadoop for the moment works only with 8, so we'd need a special config like profile::java::analytics in case :)

Edit: nevermind, already deployed on the labstores, all good then!

MoritzMuehlenhoff added a comment.Mar 12 2020, 7:44 AM

profile::java::analytics uses the Java 8 forward port on Buster, so that part should be fine. Wrt Kerberos there should also be no real issues I can think of.

• Bstorm added a comment.Mar 12 2020, 3:42 PM

In that case it might be worth saving some future work and moving along to buster on these.

MoritzMuehlenhoff added a comment.Mar 12 2020, 3:53 PM

@elukey: I don't even think we need additional changes? E.g. an-launcher1001 is on Buster and uses profile::hadoop::common with OpenJDK 8, so this is already all implemented.

elukey added a comment.Mar 12 2020, 3:56 PM
In T224583#5964337, @MoritzMuehlenhoff wrote:

@elukey: I don't even think we need additional changes? E.g. an-launcher1001 is on Buster and uses profile::hadoop::common with OpenJDK 8, so this is already all implemented.

Yes I edited my comment above, everything is already implemented for labstores, no real precautions needed for both Stretch or Buster.

• Bstorm added a comment.Mar 12 2020, 7:19 PM

I'm going through the process on virtual machines before I proceed on labstore1007 so at least I'm not surprised by some of the ways it won't go well.

gerritbot added a comment.Mar 12 2020, 10:06 PM

Change 579095 merged by Bstorm:
[operations/puppet@production] dumps-distribution: move all traffic to labstore1006

https://gerrit.wikimedia.org/r/579095

Stashbot added a comment.Mar 12 2020, 10:07 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-12T22:07:54Z] <bstorm_> moving all nfs traffic off labstore1007 and to labstore1006 for upgrades T224583

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2020, 10:10 PM
Stashbot added a comment.Mar 13 2020, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-13T16:04:35Z] <bstorm_> rebooting labstore1007 for first cycle of upgrades T224583

Stashbot added a comment.Mar 13 2020, 4:51 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-13T16:51:32Z] <bstorm_> rebooting labstore1007 for stretch upgrade T224583

• Bstorm added a comment.Mar 13 2020, 4:59 PM

labstore1007 is serving NFS and web fine on stretch now.

• Bstorm added a comment.Mar 13 2020, 4:59 PM

Mind you, clients are still all pointed at 1006. Now to try buster.

Stashbot added a comment.Mar 13 2020, 7:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-13T19:30:51Z] <bstorm_> rebooting labstore1007 for upgrade to buster T224583

• Bstorm updated the task description. (Show Details)Mar 13 2020, 7:41 PM
gerritbot added a comment.Mar 13 2020, 8:01 PM

Change 579616 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] Revert "dumps-distribution: move all traffic to labstore1006"

https://gerrit.wikimedia.org/r/579616

gerritbot added a project: Patch-For-Review.Mar 13 2020, 8:01 PM
gerritbot added a comment.Mar 13 2020, 8:05 PM

Change 579616 merged by Bstorm:
[operations/puppet@production] Revert "dumps-distribution: move all traffic to labstore1006"

https://gerrit.wikimedia.org/r/579616

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2020, 8:11 PM
gerritbot added a comment.Mar 13 2020, 8:26 PM

Change 579621 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] dumps-distribution: Fix smartmontools so it doesn't trigger alerts

https://gerrit.wikimedia.org/r/579621

gerritbot added a project: Patch-For-Review.Mar 13 2020, 8:26 PM
gerritbot added a comment.Mar 13 2020, 8:51 PM

Change 579621 merged by Bstorm:
[operations/puppet@production] dumps-distribution: Fix smartmontools so it doesn't trigger alerts

https://gerrit.wikimedia.org/r/579621

gerritbot added a comment.Mar 13 2020, 9:05 PM

Change 579626 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] dumps-distribution: move all NFS traffic to labstore1007

https://gerrit.wikimedia.org/r/579626

gerritbot added a comment.Mar 13 2020, 9:05 PM

Change 579627 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] dumps-distribution: switch which host does acme

https://gerrit.wikimedia.org/r/579627

gerritbot added a comment.Mar 13 2020, 9:07 PM

Change 579628 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/dns@master] dumps-distribution: set the TTL to 5M for dumps.wikimedia.org

https://gerrit.wikimedia.org/r/579628

gerritbot added a comment.Mar 13 2020, 9:09 PM

Change 579628 merged by Bstorm:
[operations/dns@master] dumps-distribution: set the TTL to 5M for dumps.wikimedia.org

https://gerrit.wikimedia.org/r/579628

gerritbot added a comment.Mar 13 2020, 9:17 PM

Change 579642 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/dns@master] dumps-distribution: fail over to labstore1007 for dumps.wikimedia.org

https://gerrit.wikimedia.org/r/579642

gerritbot added a comment.Mar 13 2020, 9:19 PM

Change 579626 merged by Bstorm:
[operations/puppet@production] dumps-distribution: move all NFS traffic to labstore1007

https://gerrit.wikimedia.org/r/579626

gerritbot added a comment.Mar 13 2020, 10:09 PM

Change 579642 merged by Bstorm:
[operations/dns@master] dumps-distribution: fail over to labstore1007 for dumps.wikimedia.org

https://gerrit.wikimedia.org/r/579642

gerritbot added a comment.Mar 13 2020, 10:16 PM

Change 579627 merged by Bstorm:
[operations/puppet@production] dumps-distribution: switch which host does acme

https://gerrit.wikimedia.org/r/579627

Stashbot added a comment.Mar 13 2020, 10:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-13T22:21:14Z] <bstorm_> downtimed labstore1006 for upgrades T224583

• Bstorm added a comment.Mar 13 2020, 10:26 PM

Noting here as well: when doing these in-place upgrades, facter 3 causes a funny thing to happen where it caches the last OS version and screws with your puppet runs.

You have to run sudo rm /opt/puppetlabs/facter/cache/cached_facts/operating\ system to clear it out of the cache if that happens.

Stashbot added a comment.Mar 13 2020, 10:27 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-13T22:27:34Z] <bstorm_> rebooting labstore1006 T224583

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2020, 11:10 PM
Stashbot added a comment.Mar 13 2020, 11:12 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-13T23:12:44Z] <bstorm_> rebooting labstore1006 for upgrade to stretch T224583

• Bstorm added a comment.Mar 14 2020, 12:12 AM

Ok, labstore1006 is now buster. Failing things back to their steady state.

Dzahn added a parent task: T247045: Migrate all of production metal and VMs to Buster or later.Mar 14 2020, 12:12 AM
gerritbot added a comment.Mar 14 2020, 12:13 AM

Change 579670 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] Revert "dumps-distribution: move all NFS traffic to labstore1007"

https://gerrit.wikimedia.org/r/579670

gerritbot added a project: Patch-For-Review.Mar 14 2020, 12:13 AM

Change 579671 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/dns@master] Revert "dumps-distribution: fail over to labstore1007 for dumps.wikimedia.org"

https://gerrit.wikimedia.org/r/579671

gerritbot added a comment.Mar 14 2020, 12:14 AM

Change 579672 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] Revert "dumps-distribution: switch which host does acme"

https://gerrit.wikimedia.org/r/579672

gerritbot added a comment.Mar 14 2020, 12:15 AM

Change 579670 merged by Bstorm:
[operations/puppet@production] Revert "dumps-distribution: move all NFS traffic to labstore1007"

https://gerrit.wikimedia.org/r/579670

gerritbot added a comment.Mar 14 2020, 12:17 AM

Change 579671 merged by Bstorm:
[operations/dns@master] Revert "dumps-distribution: fail over to labstore1007 for dumps.wikimedia.org"

https://gerrit.wikimedia.org/r/579671

gerritbot added a comment.Mar 14 2020, 12:18 AM

Change 579672 merged by Bstorm:
[operations/puppet@production] Revert "dumps-distribution: switch which host does acme"

https://gerrit.wikimedia.org/r/579672

• Bstorm updated the task description. (Show Details)Mar 14 2020, 12:21 AM
gerritbot added a comment.Mar 14 2020, 12:23 AM

Change 579674 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/dns@master] Revert "dumps-distribution: set the TTL to 5M for dumps.wikimedia.org"

https://gerrit.wikimedia.org/r/579674

gerritbot added a comment.Mar 14 2020, 12:24 AM

Change 579674 merged by Bstorm:
[operations/dns@master] Revert "dumps-distribution: set the TTL to 5M for dumps.wikimedia.org"

https://gerrit.wikimedia.org/r/579674

• Bstorm closed this task as Resolved.Mar 14 2020, 12:26 AM

Everything is failed back to how it normally is except it's now buster.

Maintenance_bot removed a project: Patch-For-Review.Mar 14 2020, 1:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL