All of the needed tweaks have been done. We may need to change things up if-when we decide on a different partitioning scheme or the like, but that should be a new Phab task.

In T420438#11958512, @elukey wrote:

@klausman first code changes out for staging, after applying them we'll be able to see if anything weird pops up. The procedure is listed in https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/IPIP, do you have time do to it?

This originated with our changes to kserve/knative-serving. It has long since stopped firing, and if it fires again, it's likely unrelated to the original cause of this one, so closing.

In T421461#11886918, @elukey wrote:

@klausman I had a chat with my team and we are ok in having it deployed on all MI300X nodes. Let me know if you still want to do it or not, I'll prep the updates for provisioning etc..

In T420438#11886144, @elukey wrote:

@klausman @DPogorzelski-WMF Hi! Do you have a timeline for this work?

So far, teh changes have not had the desired effect. I did some deeper digging and tried gating the start of amd-devplugin unit on the presence of the kfd devices via udev, i.e.

Steps 1-4 are covered in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275814

The host needed to have the amd device plugin service restarted, so all the GPUs would be visible to k8s. I've done so, and the service is now scheduling on 1012.

In T421461#11830434, @elukey wrote:

I checked for ml-serve1012, we have 'IOMMU': 'Auto',, that IIUC may not be what the kernel needs in your case. Probably setting it to Enabled may be a good test, I can set it via spicerack shell anytime if needed, so you can reboot and check. Lemme know if you want it @klausman

The parameter has been added to ml-serve-1012 and 1013, and the hosts have been rebooted. The workload we were looking at is atm scheduled on 1012. We haven't tested performance yet, but I see the following messages in the console:

Done & done.

After a clarifying chat with Luca about the intricacies of gRPC, HTTP/2 etc, I now have a better picture of what will need building (probably :) )

To clarify a basic assumption I have: gRPC only works over HTTP/2, and HTTP/2 is always TLS-encrypted, i.e. there is no way to speak gRPC over a plaintext connection, or at least not with the standard libraries for gRPC. If that is not the case, things might be simpler (or way more complex ;) ).

I think we can run this machine one a single disk until its replacement arrives. Even if it dies entirely, we have enough serving capacity in eqiad to handle our current workload (especially with 1014 and 1015 about to be added). We should put in a long-term silence for the specific alert and then decom the machine once it replacement arrives or the other disk also dies. The failed disk is also part of an array we currently don't use, so it should all be good.

The following is a write-up of what I see as possible stumbling blocks for services on LiftWing using gRPC in addition to our current HTTP(S)/REST mode of operation. Note that none of these concerns are deal-breakers or insurmountable, but rather are aspects that need addressing, and may prove to be more than just 30m of work, due to the complexities of Istio, kserve and k8s in general.

I will close this task in favor of T380279, where related/additional work for the lab/build machines will be done.

Adding @MoritzMuehlenhoff for security aspects.

One thing of note about using gRPC vs. REST/HTTP(S) based communication is that a lot of the infrastructure we have built around LiftWing assumes HTTP services (e.g. using Envoy in HTTP mode). If we set up a gRPC service that services on LW call, or a gRPC service on Liftwing, we need to make sure that "the path is clear" in this sense, also regarding pod security policies and the like.

Machine has been reimaged and is back in the cluster, closing.

Folded into T367048

A concrete approach is being tracked in T401778.

In T408538#11396458, @akosiaris wrote:

In T408538#11394844, @BWojtowicz-WMF wrote:

Thus, our setup requires connection between 2 services:

1. revise-tone-task-generator service deployed in revise-tone-task-generator namespace.
2. outlink-topic-model service deployed in articletopic-outlink namespace.

Which initiates the connection? 1 or 2? Or is it that both can initiate a connection to the other one?

In T367048#11335401, @gkyziridis wrote:

Thank you so much for working on that one @klausman!

Since the ml-lab1001 now uses the big TB filesystem, would it be possible to enable the buildkit daemon in order to use blubber, so ml-lab could be the dedicated machine for testing our production images and blubbers without translating the blubber files into Dockerfiles?

In T405647#11298600, @RobH wrote:

Please note this migration has shifted from Oct 15th start date to Nov 1 start date.

The issue on ml-lab1001 is (was, now) that docker did not use the big multi-TB filesystem as storage for images, but it does now. I copied the Dockerfile from Georgios' homedir to a subdir of mine and ran docker build. It seems to have worked fine:

 ---> 82a4c1b28e66
Successfully built 82a4c1b28e66
Successfully tagged hf:update_kserve
ml-lab1001 hf $ docker image ls
REPOSITORY                                                                                       TAG              IMAGE ID       CREATED          SIZE
hf                                                                                               update_kserve    82a4c1b28e66   15 seconds ago   14GB

ml-cache1002 can be done anytime, it just needs an Icinga/Prometheus downtime.

This has been rolled out to both eqiad and codfw GPU machines and I restarted our one prod pod that uses GPUs (editcheck). Everything looking good.

As for the discrepancy (~97% vs. ~99%), I just ran the equivalent of my query (using`increase` etc) but instead of looking at the destination namespace of edit-check, used the selectors from your query (source_workload_namespace="istio-system", app="istio-ingressgateway", destination_service_namespace=~"edit-check", destination_service_name=~"edit-check-predictor.*"), and now my query agrees with the ~97% result from your initial query:

For latency, we'd use something like this:

I think this would work:

Today we found that amd-smi is not a drop-in replacement for rocm-smi when it comes to exporting metrics to Prometheus. We use our own Python wrapper to convert the output of rocm-smi to metrics that we then dump into node-exporter. The commandline parameters etc have massivel changed between the two tools, so we will have to adapt it.

This has been deployed and confirmed working.

In T403697#11185767, @elukey wrote:

@klausman ml-serve1012 is up and running with 6.16 from backports, and nvtop seems to work without horrors in the dmesg. Also please note that rocm-smi is now /opt/rocm-6.4.3/bin/amd-smi, please use that instead of the Debian one. I haven't tried to partition the GPU yet, so the horrors may come afterwards. If you want to do some extra checks before the partitioning lemme know so we can assess if everything works beforehand :)

Seeing the partial successes above, I tried playing around a bit today, running rocm-smi and nvtop (the latter was originally nvidia-only, but current versions support AMD/ROCm as well). Unfortunately, both of them hang. Worse, the kernel is extremely unhappy about doing that: dmesg is full of breakage messages (log is attached). The only additional tool that I manage to get to work was this:

ml-lab1002 fails the same way. I haven't tried 1001, but I suspect it would fail the same way as well.

Same happening with 1010. I'll put everything back in service and try the lab machines now.

On 1009, the cookbook fails with:

When trying to run the cookbook against ml-serve1008, I got this:

Any time next week during my usual waking hours (0800-1800 UTC) should be doable. Just ping me on IRC.

In T386889#11019056, @KartikMistry wrote:

In T386889#11003047, @KartikMistry wrote:

@klausman While deploying T335491, I didn't see any timeout for eqiad. Should we close this task?

Should we go ahead?

In T396717#11060794, @Papaul wrote:

@klausman hello hope all is well. Is it possible to give us a day and time when you will be available to help us work on those servers? Thank you. it shouldn't take more then 10 minutes to fix each server.

In T393948#10985332, @Jclark-ctr wrote:

@klausman Will this be legacy or uefi? it is reachable

This has been complete in the course of assorted other work like the countainerd updates.

In T398533#10983785, @achou wrote:

@klausman Could you assist with this? I plan to test in staging first.

In T380722#10983317, @isarantopoulos wrote:

@klausman Shall we rename this task and switch to a newer version? A candidate could be the latest version 0.15.2

I've written up my thoughts, and some of the things we discussed outside of this ticket regarding making vLLM images available for use with LiftWing workloads:

Found it: the secrets were not wired up for staging because I had a brain fart when setting that up. It's been fixed in the private repo with commit 7bc13c5d2 (https://gerrit.wikimedia.org/r/c/labs/private/+/1160032 on the pseudo-private one), and staging now shows the correct diff:

In T335491#10921280, @KartikMistry wrote:

@klausman is there any reason why we can't see following in the diff in staging?

+ data:                                                                                                  
+   AWS_ACCESS_KEY_ID: '++++++++ # (18 bytes)'     
+   AWS_SECRET_ACCESS_KEY: '++++++++ # (16 bytes)'

SSDs have been enabled and 1002 is using Ceph homedirs.

With the above patch (and the private repo stuff) merged, we can diff on the deployment server (I elided some unrelated changes):

Full error/backtrace:

For this to work, Appropriate credentials need to be on ml-lab1002 (or 1001). The future proof way to do this would be to either apply the relevant Puppet role(s) to it, or, if that adds too much functionality/infrastructure, extract the relevant bits from that role or make that role modular as needed.

And the undelrying CSV data for the above graph:

The full chart for running the benchmark as described by Kevin above, on the SMC-provided MI300X test machine (using one of the 8 GPUs).

In T385173#10742305, @isarantopoulos wrote:

Nice work Kevin!
@kevinbazira @klausman could we run the same benchmark on the MI300X?

In T391465#10733690, @isarantopoulos wrote:

I've deleted 30GB from my home directory.
@klausman are there any quick wins to clean up disk space for now?
I think purging the huggingface cache (aka just deleting the files under /src/hf-cache/hub/) would be ok imo