Related T177393: Implement authentication/authorization in Kubernetes clusters

In T224188#5278081, @ayounsi wrote:

That sounds reasonable for the PoC, depending on rack space. @faidon for the last word.

Note that we don't have visibility in the cross virtual chassis links. Adding it to LibreNMS is possible but would require dev time.

Ok, now that I have a working etcd setup for k8s, I will follow up with kubernetes itself.

Worth noting that even though we will be using 10G links, we don't expect them to be fully used in any case in the short term.

I'm pretty happy now with how etcd looks in the puppet tree and the resulting state with fresh installed VMs. I will probably leave it as is and move on to kubernetes itself and see if anything else is required once k8s is actually using etcd.

I just created this project.

OK, after some help from @Joe and @Vgutierrez I got the cluster working:

I still don't find why etcd would do that.

Using puppet certs for etcd has some other tricky parts, like:

We totally forgot about this.

That was it. I was missing in the toolsbeta.admin group.

Ok, this works for us as well. For the record, the new project was approved yesterday in the WMCS team meeting. Will close the task now, as I understand you will be using a VM in the discourse project [0].
Feel free to reopen if required.

I just read the puppet code at modules/openstack/manifests/nova/compute/service.pp. I wonder if we could switch to automatically generated certs instead.

Just confirmed using this command that toolsbeta is the only CloudVPS project with that special config. We may just drop the special case and move on.

If I compare the config in toolforge servers, I see:

Hey, I'm just catching up. I wasn't aware of the existence of this phab task, and followed all your same steps until I got here with your very same conclusions :-)

I will be doing some stuff related to this task, so I'm claiming it.

Just setting expectations: new project requests are discussed in the WMCS team meeting every week on Tue. Next week, however, we won't have the team meeting due to the SRE offsite in Dublin. The next WMCS team meeting is expected to happen on 2019-06-18.

In T224743#5233694, @JHedden wrote:

@aborrero digging into the OpenStack configuration a bit more and realized that we're not using active/active services. I'd like to chat about the HA architecture and previous decisions made when you have some time.

I just checked, and the errors are gone. Closing task now.

In T224743#5230197, @JHedden wrote:

@aborrero Do we need to merge the roles or can we remove the primary secondary roles and profiles?

role::wmcs::openstack::eqiad1::services_primary
role::wmcs::openstack::eqiad1::services_secondary
profile::openstack::eqiad1::pdns::recursor::primary
profile::openstack::eqiad1::pdns::recursor::secondary

Apparently, the last missing bit to be able to merge both roles is:

In T224877#5230017, @MoritzMuehlenhoff wrote:

prometheus-pdns-rec-exporter should be available for Stretch, it's used on the production recursors, which are on Stretch:
https://debmonitor.wikimedia.org/packages/prometheus-pdns-rec-exporter

We will be upgrading cloudservices1003 to stretch today.

This should be fixed AFAIK. Closing task now. Hopefully, we will upgrade openstack soon and we will drop this hack.

Most of the unused toolforge code has been cleaned up already. We still have plenty of code in the toollabs namespace, but that code is in use, mostly by k8s and other pieces that weren't upgraded as part of the GE->SGE migration.

Sure! Adding some context for @JHedden:

current status by the time of this comment: The only toolforge Debian Jessie server that is running sssd is tools-worker-1029, that was created explicitly for testing it.

I had to cordon/drain all the k8s worker nodes that were running sssd, because your pod was scheduled in them :-P

More tests:

However, I just did:

This pod is running in a node I'm working right now for T224558: sssd: support for Debian Jessie

The pam issue may require a pam-auth-update --force --package run in the server, because there are stale entries in the pam config pointing to pam_ldap.so, which we don't use anymore after switching to sssd.

Issues continue despite the patch https://gerrit.wikimedia.org/r/513091.

On IRC:

On second thoughts, we would like to change the public VLAN for a private one, from .wikimedia.org to .wmnet.

This should be working now:

Well. Closing this task now, since we do have sudo support now. Will open other tasks for the other issues we have.

Rack proposal: anywhere in codfw, each server in a different rack, a rack with 10G support
Wiring configuration: single 10G connection each server if possible. The mgmt interface connected as in any other server (standard).
Name proposal: Per T210666#4916941, we agreed on calling them cloudbackup. They can be called cloudbackup2001.wikimedia.org and cloudbackup2002.wikimedia.org. I added the entry to https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions#Servers

+1 approved in the WMCS team meeting.

+1 approved in the WMCS team meeting.

Related: T224354: backport pdns-server version 3.x to Stretch apparently we need pdns 3x for openstack mitaka.

The logs contains plenty of messages like these:

Searching the logs for the concrete REQ, I see this:

Should be OK now.

In T216753#5211711, @Bstorm wrote:

For now, the doc is pretty good. Got help from the DBAs--also a reminder that we should be doing regular failover testing rather than things being quite so hard to do.

In T221770#5213978, @Volans wrote:

cloudcontrol1003 is flapping its systemd degraded alert since 2019-05-25 21:46. The unit that fails is:

● designate_floating_ip_ptr_records_updater.service               loaded failed failed    Designate Floating IP PTR records updater

Honestly, wikimediacloudservices.org seems overly long. Just reading that makes me feel lazy :-( If possible, I would use it only for redirecting www. to wikitech or whatever our landing page is :-P

How many floating IPs are you requesting? I guess is just 1?

In T169287#5208609, @Bstorm wrote:

But puppet doesn't run the agent like normal when it modifies a cert. It waits for signature, etc.?
However, now that you mention it, I think I was thinking about it wrong. Why not just subscribe to: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/etcd/manifests/ssl.pp#32 ?

Certs for puppet may be core to puppet and affect evaluation, but a File resource that happens to use the puppet cert as the source is something that can be watched and acted on when it changes (the puppet cert itself may be as well...but it somehow makes my head hurt).

In T223902#5209309, @BBlack wrote:

Do these belong in wikimedia.org at all? It seems this has already been discussed, but I guess I lack some context.

I've been testing generating puppet certs with SANs with no issues so far. Is just a hiera override and re-generate the certs. Steps, for the record:

I wonder if puppet subscribe => and/or notify => mechanisms would work in this case.