People can now pip install and use models. Right now we only have a few models - the number of models should increase over time.

• Calico improvements makes the whole workflow more streamlived
• Improve our incident response procedure
• Investigate CPU spikes

• Still can't use GPU with ROCm. But we figured out what the bug is - if the control version is upgraded to Bookworm it will be fixed.
• Next step is to upgrade ml-staging to Bookworm then test.
• Working on upgrading HF with newer versions with ROCm 6.0. Tested them and they work and will be posting watch.
• Goal is to utilize GPU so we can deploy models from HuggingFace.

• Trying to fix up a Calico networking issue in Kubernetes
  • After credentials, will send patched revert risk server to ml-staging

• Narrowed down cause of symptoms of spike in CPU usage to feature extraction in revscoring isvc. Might be caused by some specific revids.

• Wait for vendor (Supermicro) to finalize order of 2x for ml-staging.
  • Chris's guess is ml-staging installed at end of quarter

• Working on plumbing on staging, should be done within week
  • Feeling good about it

Logging queries and logging when things are slow is the short term goal. Knowing WHY a query takes a long time is a future question

We have a theory that the ROCm drivers on the debian package is not required.

Decision point: Do we upgrade ROCm drivers?

Update: No update

• Rebased code after prototype.
  • Waiting for istio change for making a new service, which is imminent
  • Need to add new visual service that is tcp

• GPU order for the first GPU 2x chassis is close to complete. There are some supply issues with the chassis, so the question is going to be if we want to use an upgraded chassis for the ml-staging server.

• Merged puppet machinery to allow network policies to be generated for assorted cluster. So we can automatically generated the network policy without the 60 lines of istio config.
• Will merge change to network policy to allow Istio to talk to Cassandra.

At risk because we don't have a GPU in the data centers yet.