For anyone who wants to build the above binary form the Debian sources:

Machine is imaged and running. The PXE boot was "fixed" by an ugly hack mentioned in T304483#9906962 While the firmware problem remains, at least we are unblocked on this host.

So today I wanted to instal ml-staging2003. This is a new SMC hardware type and it hits this problem.

In T357415#9905563, @Papaul wrote:

Information2
The server has only the the SFT-OOB-LIC license which is the Supermicro Out of band OOB license allowing you to only update BIOS and BMC/IPMI . This license will not allow you to upgrade the Add-on 10G and 25G network interface i mentioned in (1). This is the reason we are getting the error "ERROR:Not licensed to perform this request. The following licenses DCMS were needed Click here to return"

The DCMS which is the Supermicrp's Data Center Management Suite license will unlock some feature such as the Supermicro Update manager which will then allow you to manage firmware and the Redfish support feature which is more for @Volans and @elukey for automation.

Hope all this information was helpful please let me know if you have any questions.

It looks like the primary interface can't see the network device (the console shows "media test failure, check cable".

Machine is drained and off, so you're free to reseat memory etc. Let me know when it's back (and what we might be able to do if the memory remains problematic).

Current state:

Tuesday sounds good. I'll drain and shutdown the machine on Tuesday 17:00 CEST/15:00 UTC/10:00CDT, does that work for you?

One note: since the default OS has changed (Bullseye->Bookworm), I updated the ticket desc accordingly --- we definitely want Bookworm.

I repooled the machine just now, as I don't want to fly this close to capacity ceiling for prolonged periods.

Can't upload the ASR since it's too large. Anywhere that I should upload it to?

In T343123#9813736, @Dbrant wrote:

I'm checking out the Lift Wing API to generate article descriptions, and I see the URL path contains a colon : character. The problem is that our native network stack always urlencodes colons, but when I hit the same API path with an encoded colon (%3A) it returns 404.
Is this expected? Can we make the API accept a urlencoded path?

• we had another instance of high lat (eswiki)
• logs show fetch features being slow (extract_cache)
• we havea repor that should help with root-causing the matter

• Mistral crashlooping, startup checks usually 5m , so we bumped to 10m, but it didn't help
• Bert model works, so likely Mistral issue
• the kubelet partition increase for the install phase is in review
• ml-staging1001 is now on Bookworm, dragonfly (distributed downloading of S3 stuff) needs to be bumped
• with bookworm, there no longer are GPU drivers on the base node (besides Debian kernel support), but driver/library code lives in the Docker images

Repooled the machine:

$ sudo confctl select 'name=ml-serve2002.codfw.wmnet' set/pooled=yes
codfw/ml_serve/kubesvc/ml-serve2002.codfw.wmnet: pooled changed no => yes
WARNING:conftool.announce:conftool action : set/pooled=yes; selector: name=ml-serve2002.codfw.wmnet

Since this has not re-occurred, I am closing the task for now. If it happens again, we can always re-open.

In T287491#9803609, @jijiki wrote:

@klausman you can go head with kserve and knative-serving when you have some time, ping me for reviews etc

• Connections from isvc namespaces on staging to the Cassandra machines now work, including TLS certs and SNI
• Next step: have an actual inference service actually talk to the cache, likely with code from https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/995001
• Still need to figure out long-term maintenance of Cassandra server-side config (users, passwords, namespaces, schemas); may hand off/soft-donate the machines to Data Persistence Team

All the machinery is now in place to make connections to Cassandra from isvcs on staging (in the experimental NS):

From https://man7.org/linux/man-pages/man2/access.2.html

In T362984#9785902, @isarantopoulos wrote:

I got an error when trying llm image locally with bullseye-torch2.3.0-rocm5.7 (related patch):

Traceback (most recent call last):
  File "/srv/app/llm/model.py", line 9, in <module>
    import torch
  File "/opt/lib/python/site-packages/torch/__init__.py", line 237, in <module>
    from torch._C import *  # noqa: F403
ImportError: libamdhip64.so: cannot enable executable stack as shared object requires: Invalid argument

Haven't found any related open issue so I'm currently testing different pytorch versions to see if the issue still exists

In T363449#9773855, @elukey wrote:

@klausman leaving the decision to you :) You can file path anytime, today we'll roll out the transparent proxy changes for eqiad and then we'll be ok to proceed with the new commons host header. Before proceeding I'd suggest to check if calls to the MW API can accept commons host header and URI paths, I don't think any rewrite is happening in upper layers but better safe than sorry!

This has been implemented in change 1020194.

This is a bug in keras: it tries to open the file with mode r+b (read, append, binary), but since the file is owned by another user (nobody vs. somebody), the call fails. Why keras would need to be able to append to the file, I don't know.

In T362749#9727438, @kevinbazira wrote:

PermissionError: [Errno 13] Permission denied: '/mnt/models/logo_max_all.keras'

I've experimented a bit on Thanos, and arrived at this query:

Probably something like:

Timeline (times in UTC):

We have restarted an associated services and its logs show no more errors. It's not quite root-caused yet, but the functionality should be back to working order now. I have confirmed this for ruwiki.