| • EChetty | |
| Feb 10 2023, 11:29 AM |
| F54214206: image.png | |
| Thu, May 23, 10:00 AM |
We have five stat servers that are still running buster, namely: stat100[4-8]
However, stat100[4-7] are currently scheduled to be decommisisoned in T353785: Decom EOL stats servers stat100[4-7]
Therefore, the only remaining host that needs to be upgraded is: stat1008
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Open | None | T288804 Upgrade the Data Engineering infrastructure to Debian Bullseye | |||
| Resolved | BTullis | T329360 Upgrade stat1008 to bullseye |
Hi folks! In T295661 I updated the AMD ROCm stack for our GPU to 5.4, but I added support only for Bullseye. When you migrate stat100[5,8] it should be sufficient to bump the ROCM version to 54 in puppet's regex.yaml. Please ping me in case something doesn't work!
Can we revisit this? Primarily for using recent ROCM versions when working with GPUs as mentioned by Luca above, but it would also be nice to add the ability to run docker containers on the stat clients (T275551).
Hi @fkaelin - I believe that we will be tackling T336040: Bring stat1010 into service with GPU from stat1005 within the next couple of weeks, which should at least get you ROCm 5.4 on bullseye. That's as soon as T336036: Bring stat1009 into service is finished, which will be our first bullseye based stats server. (cc @Stevemunene)
Unfortunately, I don't think that getting the docker CLI on the stat servers is going to happen any time soon. I know that @elukey has made some good progress recently with getting access to GPUs on the dse-k8s cluster (see: T333009#8808370) so perhaps you could build on that work, somehow.
Hi folks! Would it be possible to have either stat1005 or stat1008 (the ones with GPUs) on bookworm? I am asking since we'd have a place with python 3.11 and the latest packages, very useful for debugging etc.. I can help in adjusting puppet where needed to accommodate bullseye -> bookworm if needed!
It's a great idea, but we haven't even finished getting a stat server running on bullseye yet. We are about to start work on T336040: Bring stat1010 into service with GPU from stat1005 so if you like we could change the target from that from bullseye to bookworm.
I've also just got a good bit of work done with rebuilding hadoop for bullseye T337465: Rebuild all hadoop packages for bullseye with different distribution suffix mechanism and https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Bigtop_Packages so it should be easier for us to try building hadoop etc for bookworm.
The only other alternative would seem to be upgrading stat1008 in place, but I have a feeling that this would be pretty disruptive for users who are currently using it.
What do you think @elukey?
@BTullis sorry I completely forgot about the hadoop packages, let's not jump to bookworm yet, you folks have enough work on your plate, we'll see in the future :)
I have scheduled this work for Thursday May 23rd 2024 at approximately 09:15 UTC.
Reminder to self that I will have to update this page after the upgrade: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/stat1008.eqiad.wmnet
...and let users know that they can expect to see warnings of changed keys, so they will need to remove their local known_hosts entries before being able to access stat1008 again.
Change #1035346 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Add a new partman reuser recipe for stat1008
Change #1035346 merged by Btullis:
[operations/puppet@production] Add a new partman reuse recipe for stat1008
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host stat1008.eqiad.wmnet with OS bullseye
I'm not confident that the reuse recipe that I have created is correct.
Change #1035348 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Fix the stat1008 partman-reuse recipe
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host stat1008.eqiad.wmnet with OS bullseye executed with errors:
Change #1035348 merged by Btullis:
[operations/puppet@production] Fix the stat1008 partman-reuse recipe
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host stat1008.eqiad.wmnet with OS bullseye
Change #1035353 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Remove trailing slash from stat1008 partman recipe
Change #1035353 merged by Btullis:
[operations/puppet@production] Remove trailing slash from stat1008 partman recipe
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host stat1008.eqiad.wmnet with OS bullseye executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host stat1008.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host stat1008.eqiad.wmnet with OS bullseye completed: