Page MenuHomePhabricator
Create Task
Maniphest T335756

Create a staging ingress configuration for ml-staging-codfw
Closed, ResolvedPublic
Actions

Description

We have now to create ores-legacy-staging.svc.codfw.wmnet for the ml-staging-codfw cluster, but instead of creating a specialized VIP + config we could follow what Service Ops did to keep things in sync between the teams, namely a single endpoint.

The overall idea is to:

  • create ml-staging.svc.codfw.wmnet as CNAME for one of the ml-staging200[12] nodes.
  • create a cergen certificate like the one for wikikube staging to deploy on tls-proxy containers (mesh).
  • modify the ingress module in deployment-charts to generate the correct values for this new endpoint.

On the knative / istio side:

  • Allow the definition of multiple Istio ingresses in our helmfile Istio config.
  • Allow to use a different ingress for knative-serving.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.May 2 2023, 10:45 AM
gerritbot added a comment.May 2 2023, 11:10 AM

Change 914306 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] modules: duplicate the istio ingress template for 1.0.2

https://gerrit.wikimedia.org/r/914306

gerritbot added a project: Patch-For-Review.May 2 2023, 11:10 AM

Change 914307 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] modules: add ml-staging cfg to the istio template

https://gerrit.wikimedia.org/r/914307

elukey moved this task from Unsorted to In Progress on the Machine-Learning-Team board.May 2 2023, 2:28 PM
gerritbot added a comment.May 3 2023, 9:58 AM

Change 914728 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] conftool-data: add config for the k8s ingress for ml-staging

https://gerrit.wikimedia.org/r/914728

gerritbot added a comment.May 3 2023, 10:27 AM

Change 914735 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add conftool and service config for k8s-ingress-ml-staging

https://gerrit.wikimedia.org/r/914735

gerritbot added a comment.May 3 2023, 10:27 AM

Change 914730 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Add the VIP settings for the K8s ingress for ml-staging

https://gerrit.wikimedia.org/r/914730

gerritbot added a comment.May 3 2023, 2:33 PM

Change 914306 merged by Elukey:

[operations/deployment-charts@master] modules: duplicate the istio ingress template for 1.0.2

https://gerrit.wikimedia.org/r/914306

gerritbot added a comment.May 3 2023, 2:34 PM

Change 914307 merged by Elukey:

[operations/deployment-charts@master] modules: add ml-staging cfg to the istio template

https://gerrit.wikimedia.org/r/914307

gerritbot added a comment.May 3 2023, 2:37 PM

Change 914793 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] fast-api: update ingress.istio module version to 1.0.2

https://gerrit.wikimedia.org/r/914793

gerritbot added a comment.May 3 2023, 2:39 PM

Change 914795 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] ml-services: enable the 'mlstaging' ingress flag for ores-legacy

https://gerrit.wikimedia.org/r/914795

gerritbot added a comment.May 3 2023, 2:41 PM

Change 914793 merged by Elukey:

[operations/deployment-charts@master] fast-api: update ingress.istio module version to 1.0.2

https://gerrit.wikimedia.org/r/914793

gerritbot added a comment.May 3 2023, 2:41 PM

Change 914795 merged by Elukey:

[operations/deployment-charts@master] ml-services: enable the 'mlstaging' ingress flag for ores-legacy

https://gerrit.wikimedia.org/r/914795

gerritbot added a comment.May 3 2023, 3:42 PM

Change 914859 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: add ml-staging among helmfile_namespace_certs's options

https://gerrit.wikimedia.org/r/914859

gerritbot added a comment.May 4 2023, 8:01 AM

Change 914859 merged by Elukey:

[operations/deployment-charts@master] admin_ng: add ml-staging among helmfile_namespace_certs's options

https://gerrit.wikimedia.org/r/914859

gerritbot added a comment.May 4 2023, 8:11 AM

Change 915416 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: remove tls hostname override for ores-legacy-staging

https://gerrit.wikimedia.org/r/915416

gerritbot added a comment.May 4 2023, 8:47 AM

Change 915416 merged by Elukey:

[operations/deployment-charts@master] admin_ng: remove tls hostname override for ores-legacy-staging

https://gerrit.wikimedia.org/r/915416

gerritbot added a comment.May 4 2023, 10:55 AM

Change 914728 merged by Elukey:

[operations/puppet@production] conftool-data: add config for the k8s ingress for ml-staging

https://gerrit.wikimedia.org/r/914728

gerritbot added a comment.May 4 2023, 10:59 AM

Change 914735 merged by Elukey:

[operations/puppet@production] Add conftool and service config for k8s-ingress-ml-staging

https://gerrit.wikimedia.org/r/914735

gerritbot added a comment.May 4 2023, 2:24 PM

Change 915685 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] custom_deploy.d: fix istio config for ml-staging-codfw

https://gerrit.wikimedia.org/r/915685

elukey added a comment.EditedMay 4 2023, 2:27 PM

First result! from any node, like stat100x, one could use:

curl "https://ores-legacy.k8s-ml-staging.discovery.wmnet:31443/v3/scores" -i --http1.1 --resolve  ores-legacy.k8s-ml-staging.discovery.wmnet:31443:10.192.0.201

Scoring a single element doesn't work yet, since the envoy proxies that we use to contact lift wing don't support POSTs in their default config.

The approach of using two istio ingresses seems to work fine, so now the next steps are:

  • Create the new VIP for k8s-ml-staging.discovery.wmnet
  • Create the DNS CNAME *k8s-ml-staging.discovery.wmnet
gerritbot added a comment.May 4 2023, 2:28 PM

Change 915685 merged by Elukey:

[operations/deployment-charts@master] custom_deploy.d: fix istio config for ml-staging-codfw

https://gerrit.wikimedia.org/r/915685

elukey mentioned this in T330414: Create ORES migration endpoint (ORES/Liftwing translation).May 10 2023, 8:46 AM
gerritbot added a comment.May 10 2023, 9:05 AM

Change 914730 merged by Elukey:

[operations/dns@master] Add the VIP settings for the K8s ingress for ml-staging

https://gerrit.wikimedia.org/r/914730

gerritbot added a comment.May 10 2023, 9:10 AM

Change 918409 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] service::catalog: set lvs_setup for k8s-ingress-ml-staging

https://gerrit.wikimedia.org/r/918409

gerritbot added a comment.May 15 2023, 7:39 AM

Change 919785 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Add discovery settings for k8s-ingress-ml-staging

https://gerrit.wikimedia.org/r/919785

gerritbot added a comment.May 15 2023, 8:26 AM

Change 918409 merged by Elukey:

[operations/puppet@production] service::catalog: set lvs_setup for k8s-ingress-ml-staging

https://gerrit.wikimedia.org/r/918409

Stashbot added a comment.May 15 2023, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2023-05-15T08:26:44Z] <elukey> restart pybal on lvs2010 and lvs2009 to pick up new LVS VIP for ml-staging k8s ingress - T335756

gerritbot added a comment.May 15 2023, 8:42 AM

Change 919795 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] service::catalog: switch k8s-ingress-ml-staging to production

https://gerrit.wikimedia.org/r/919795

gerritbot added a comment.May 15 2023, 9:06 AM

Change 919795 merged by Elukey:

[operations/puppet@production] service::catalog: switch k8s-ingress-ml-staging to production

https://gerrit.wikimedia.org/r/919795

gerritbot added a comment.May 15 2023, 1:10 PM

Change 919785 merged by Elukey:

[operations/dns@master] Add discovery settings for k8s-ingress-ml-staging

https://gerrit.wikimedia.org/r/919785

Maintenance_bot removed a project: Patch-For-Review.May 15 2023, 1:10 PM
elukey added a comment.May 15 2023, 1:27 PM

And we are finally done!

elukey@stat1004:~$ time curl "https://ores-legacy.k8s-ml-staging.discovery.wmnet:31443/v3/scores/enwiki/123433/damaging" -i --http1.1
HTTP/1.1 200 OK
date: Mon, 15 May 2023 13:26:50 GMT
server: istio-envoy
content-length: 377
content-type: application/json
x-envoy-upstream-service-time: 134

{
  "enwiki": {
    "models": {
      "damaging": {
        "version": "0.5.1"
      }
    }, 
    "scores": {
      "123433": {
        "damaging": {
          "score": {
            "prediction": false, 
            "probability": {
              "false": 0.9899875154315122, 
              "true": 0.010012484568487832
            }
          }
        }
      }
    }
  }
}
real	0m0.260s
user	0m0.023s
sys	0m0.003s
elukey moved this task from In Progress to Complete Q3 2022/23 on the Machine-Learning-Team board.May 15 2023, 1:27 PM
elukey closed this task as Resolved.May 15 2023, 3:14 PM
elukey mentioned this in T336726: Create k8s ingress config and VIP for ores-legacy.May 16 2023, 6:48 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits