Machine has been reimaged and is back in the cluster, closing.

Folded into T367048

A concrete approach is being tracked in T401778.

In T408538#11396458, @akosiaris wrote:

In T408538#11394844, @BWojtowicz-WMF wrote:

Thus, our setup requires connection between 2 services:

1. revise-tone-task-generator service deployed in revise-tone-task-generator namespace.
2. outlink-topic-model service deployed in articletopic-outlink namespace.

Which initiates the connection? 1 or 2? Or is it that both can initiate a connection to the other one?

In T367048#11335401, @gkyziridis wrote:

Thank you so much for working on that one @klausman!

Since the ml-lab1001 now uses the big TB filesystem, would it be possible to enable the buildkit daemon in order to use blubber, so ml-lab could be the dedicated machine for testing our production images and blubbers without translating the blubber files into Dockerfiles?

In T405647#11298600, @RobH wrote:

Please note this migration has shifted from Oct 15th start date to Nov 1 start date.

The issue on ml-lab1001 is (was, now) that docker did not use the big multi-TB filesystem as storage for images, but it does now. I copied the Dockerfile from Georgios' homedir to a subdir of mine and ran docker build. It seems to have worked fine:

 ---> 82a4c1b28e66
Successfully built 82a4c1b28e66
Successfully tagged hf:update_kserve
ml-lab1001 hf $ docker image ls
REPOSITORY                                                                                       TAG              IMAGE ID       CREATED          SIZE
hf                                                                                               update_kserve    82a4c1b28e66   15 seconds ago   14GB

ml-cache1002 can be done anytime, it just needs an Icinga/Prometheus downtime.

This has been rolled out to both eqiad and codfw GPU machines and I restarted our one prod pod that uses GPUs (editcheck). Everything looking good.

As for the discrepancy (~97% vs. ~99%), I just ran the equivalent of my query (using`increase` etc) but instead of looking at the destination namespace of edit-check, used the selectors from your query (source_workload_namespace="istio-system", app="istio-ingressgateway", destination_service_namespace=~"edit-check", destination_service_name=~"edit-check-predictor.*"), and now my query agrees with the ~97% result from your initial query:

For latency, we'd use something like this:

I think this would work:

Today we found that amd-smi is not a drop-in replacement for rocm-smi when it comes to exporting metrics to Prometheus. We use our own Python wrapper to convert the output of rocm-smi to metrics that we then dump into node-exporter. The commandline parameters etc have massivel changed between the two tools, so we will have to adapt it.

This has been deployed and confirmed working.

In T403697#11185767, @elukey wrote:

@klausman ml-serve1012 is up and running with 6.16 from backports, and nvtop seems to work without horrors in the dmesg. Also please note that rocm-smi is now /opt/rocm-6.4.3/bin/amd-smi, please use that instead of the Debian one. I haven't tried to partition the GPU yet, so the horrors may come afterwards. If you want to do some extra checks before the partitioning lemme know so we can assess if everything works beforehand :)

Seeing the partial successes above, I tried playing around a bit today, running rocm-smi and nvtop (the latter was originally nvidia-only, but current versions support AMD/ROCm as well). Unfortunately, both of them hang. Worse, the kernel is extremely unhappy about doing that: dmesg is full of breakage messages (log is attached). The only additional tool that I manage to get to work was this:

ml-lab1002 fails the same way. I haven't tried 1001, but I suspect it would fail the same way as well.

Same happening with 1010. I'll put everything back in service and try the lab machines now.

On 1009, the cookbook fails with:

When trying to run the cookbook against ml-serve1008, I got this:

Any time next week during my usual waking hours (0800-1800 UTC) should be doable. Just ping me on IRC.

In T386889#11019056, @KartikMistry wrote:

In T386889#11003047, @KartikMistry wrote:

@klausman While deploying T335491, I didn't see any timeout for eqiad. Should we close this task?

Should we go ahead?

In T396717#11060794, @Papaul wrote:

@klausman hello hope all is well. Is it possible to give us a day and time when you will be available to help us work on those servers? Thank you. it shouldn't take more then 10 minutes to fix each server.

In T393948#10985332, @Jclark-ctr wrote:

@klausman Will this be legacy or uefi? it is reachable

This has been complete in the course of assorted other work like the countainerd updates.

In T398533#10983785, @achou wrote:

@klausman Could you assist with this? I plan to test in staging first.

In T380722#10983317, @isarantopoulos wrote:

@klausman Shall we rename this task and switch to a newer version? A candidate could be the latest version 0.15.2

I've written up my thoughts, and some of the things we discussed outside of this ticket regarding making vLLM images available for use with LiftWing workloads:

Found it: the secrets were not wired up for staging because I had a brain fart when setting that up. It's been fixed in the private repo with commit 7bc13c5d2 (https://gerrit.wikimedia.org/r/c/labs/private/+/1160032 on the pseudo-private one), and staging now shows the correct diff:

In T335491#10921280, @KartikMistry wrote:

@klausman is there any reason why we can't see following in the diff in staging?

+ data:                                                                                                  
+   AWS_ACCESS_KEY_ID: '++++++++ # (18 bytes)'     
+   AWS_SECRET_ACCESS_KEY: '++++++++ # (16 bytes)'

SSDs have been enabled and 1002 is using Ceph homedirs.

With the above patch (and the private repo stuff) merged, we can diff on the deployment server (I elided some unrelated changes):

Full error/backtrace:

For this to work, Appropriate credentials need to be on ml-lab1002 (or 1001). The future proof way to do this would be to either apply the relevant Puppet role(s) to it, or, if that adds too much functionality/infrastructure, extract the relevant bits from that role or make that role modular as needed.

And the undelrying CSV data for the above graph:

The full chart for running the benchmark as described by Kevin above, on the SMC-provided MI300X test machine (using one of the 8 GPUs).

In T385173#10742305, @isarantopoulos wrote:

Nice work Kevin!
@kevinbazira @klausman could we run the same benchmark on the MI300X?

In T391465#10733690, @isarantopoulos wrote:

I've deleted 30GB from my home directory.
@klausman are there any quick wins to clean up disk space for now?
I think purging the huggingface cache (aka just deleting the files under /src/hf-cache/hub/) would be ok imo

In T381394#10661893, @VRiley-WMF wrote:

@klausman This has been completed and the drives have been added. Is there anything additional we may need to do on our end?

One additional note: we used to use our own Partman recipe (partman/custom/kubernetes-node-overlay-large-kubelet.cfg). Since the larger kubelet partition is already part of the containerd partman recipe (partman/custom/kubernetes-node-containerd.cfg), we don't need to have our own version of that.

In T369493#10595428, @elukey wrote:

@klausman we can check what inference DC takes the majority of the traffic and then depool the other one for a couple of hours, it shouldn't be a big deal, capacity wise we are able to handle all traffic from one DC.

In T369493#10590409, @elukey wrote:

@klausman @isarantopoulos @achou The only thing that I can think of is the following:

1. depool eqiad or codfw from inference.discovery.wmnet
2. manually change an isvc in the depooled DC, and verify the problem with more time (what happens, errors, etc..)
3. restored and repool once done

In T386969#10576936, @Eevans wrote:

@klausman are we OK to upgrade the ml-cache cluster?