In T229871#5414444, @Phamhi wrote:

I managed to bypass that issue by running

sudo wmf-auto-reimage-host --no-verify -p T229871 cloudvirt1023.mgmt.eqiad.wmnet

but it looks like manual intervention is required at the step below

ack!

@Bstorm I'm assigning this task to you since its mostly you working on this right now (the maintain-kubeusers).

In T228500#5387903, @Bstorm wrote:

To put my mind at ease that we aren't going to end up limited to under 3000 tools, and because I want to understand a bit better, which one of these are you currently testing @aborrero? https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/nginx-ingress-controllers.md
It seems the answers to many of these questions change depending on which nginx controller we are using.

That makes sense:

This may be a good starting task for @Phamhi, apart from the ones we have already.

ok, ACK, thanks.

@bd808 the admin module changes have been merged. I don't see the changes in the LDAP directory. Let me know if I should do them myself.

That plan sounds good. Remember that you may need to manually restart ferm in some places because we pass FQDNs directly to the ferm config and thus the backing IP change for m5-master.eqiad.wmnet won't be detected as a puppet change (so puppet agent won't restart ferm)

We are using the username from one account and the uid from another one. This is a bit confusing. I'm not sure if its even possible/desirable to drop accounts (or we just deactivate them or whatever).

Yes, the puppet information in horizon is extremely slow, specially the Prefix Puppet pages. That in concrete is a known issue with no short term fix :-(
35 seconds for me:

In T149589#5399356, @ema wrote:

Horizon really is unbearably slow, to the point of being almost unusable.

We had a really high IO usage on this server the other day, along with very high load avg.

This should be done. Anybody please reopen if there are any related issues.

For the record, I just created a basic documentation page: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts

Here you go @Vgutierrez. Ping me on IRC if you would like to have some assistance when creating the Wikitech or Striker account.

For the WMCS team meeting, needs discussion: how to better handle this. I'm not aware of the current workflow for creating service account in openstack.

I could normally operate both grid webservices and k8s webservices. There is no apparent reason for this issue.
Toolschecker didn't like that I managed the webservices on my own, so I had to stop them, and restart all the webservices again using toolscheckerctl restart.

At first sight both k8s and gridengine webservices look fine. Indeed, there was an issue with toolschecker-related webservices. I'm restarting them by hand to see what happens.

This should be done now:

For the record: https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ#Can_I_create_a_new_DNS_domain/zone_for_my_project,_or_records_under_the_wmflabs.org_domain?

note to self, for next time I get into this again (hopefully next week):

• I will need to re-deploy my testing tool into its own namespace to better test all this stuff
• the ingress object should be namespaced: $tool-ingress
• the nginx-ingress pod should not listen in 80/tcp, but other unprivileged port
• the nginx-ingress-svc object should be added to modules/toolforge/files/k8s/kubeadm-nginx-ingress.yaml

Ok, 2019-10-03, work for us. Will let my team know, since I won't be around.

Ok, so I'm proposing two dates:

I think we could either do this next week or wait until september because the WMCS team we will be traveling for Wikimania + offsite.

In T228500#5384584, @Bstorm wrote:

I do have a question on it: What namespace does it run in, and do we need to whitelist the namespace in the docker registry restrictions or to construct a container in our internal registry? The latter might be the better option. What do you think @aborrero?

Ok, I think I have a working setup that may (or may not) be headed in the right direction. First iteration anyway.

The dates you mention the WMCS team will be barely available because travel/wikimania/offsites, etc. Since the racks are "easy" for us, this shouldn't be a blocker though. Our servers are mostly ready for the operations, and will re-review them a day before to ensure nothing new (important VM) were scheduled to run there.
So, ACK, good to go.

couple of question for @Bstorm and @bd808:

Just noticed a thing about this setup. We don't preserve the source address of the original client, which may complicate things in case we need debugging.
I wonder if we should consider other proxy approach, like using a L3/L4 load balancer instead (NAT based).

For the record, this is what we do for the new k8s in Toolforge: profile::toolforge::k8s::kubeadm::preflight_checks in file modules/profile/manifests/toolforge/k8s/kubeadm/preflight_checks.pp.

ok, so nginx-ingress requires TLS to be present by default even if we don't use it. Using the default self-signed certs from the upstream example.

That was it. Now at least the pod starts with errors :-)

Seviceaccounts are wrong?

Ok this is my lastest attempt to deploy nginx-ingress: