Ok, new proposal: 2020-10:-29

thanks for working on this!

New proposed date: 2020-11-03,

That's fair. I will try proposing a new date tomorrow.

No problems on my side.

Looking at our calendars, I think Tuesday, October 27 might work.

This change has impacts to Toolforge (NFS, databases, etc). We want to reduce the downtime, i.e, failover things etc. For this it would be good if we can do this operation during EU/US overlapping time.
Also, we would like to announce the operation window to the community 1 week prior.

Moreover, if I workaround the first validation, when I try to assign the address to the virtual router, I get:

In T263622#6508686, @ayounsi wrote:

About vlan 2107, how many IPs do you need? If similar to the diagram a /31 should be enough.

ok thanks! It works. I'm now able to reimage labtestvirt2003 (cloudgw).

New update: in order to workaround the constraints we found when working with the bonding+trunking setup, and per suggestion by @ayounsi, I think we should:

The setup above was declined by @ayounsi

When I try to reimage the server, the DHCP boot wont work. I believe the vlan configuration is right, and therefore suspect there is something going on in the link aggregation config.

I wonder if there are CloudVPS VMs in which the admin hand-edited the file. The moment we enable this, the changes would be lost, right?

yes /31

The 1.17 packages are in the repo.

thanks!

Sent a report upstream: https://marc.info/?l=netfilter-devel&m=160145994003146&w=2

We are experiencing more issues, reopening.

I updated the task description to include info on the native vlan we need in order to install the server.

Additional context: there were some ongoing ops in the eqiad datacenter when this issue happened. I didn't have time yet to investigate what exactly happened, but a prod DNS server might have been down.

With all that being said, I'm closing the task. Please reopen if required!

I suspect what's happening here is:

With the patches I merged related to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) I suspect this issue wont happen again.

it seems we don't have any buster cloudvirt in codfw1dev. It would have been awesome to be able to test this there first.

I think T263205: Strange NFS client outage on VMs running on cloudvirt1036 is related to this issue. I discovered the same error on cloudvirt1036, which banished as soon as I installed the newer iptables package.

for the record:

this can be related to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff), I'm investigating.

I guess the reboot resulted in the server running a newer kernel.

Not sure if @Andrew is aware of this issue, but posting here for the record anyway.

done!

mmm will do toolsbeta first just in case.

moving this again to doing. Will try to take a final look this week.

Raising priority, @Andrew mentioned that with the new domains for VMs we should try creating new k8s nodes and see how that works. This seems like the right test.

This should be fixed now. Thanks @Vgutierrez for the assistance on IRC.

The acme-chief backend gets OCSPResponseStatus.UNAUTHORIZED and can't generate the new certs apparently.

For the record, what happened in august is mostly related to us moving workload to Ceph, read more here:

• https://techblog.wikimedia.org/2020/08/24/ceph-distributed-vm-storage-coming-to-cloud-services/
• https://techblog.wikimedia.org/2020/08/24/ceph-at-wmcs-the-numbers-and-the-details/

I will discuss this with the team and get back to you in a couple of days.

Before start:

aborrero@ceph-restore-practice:~$ cat /proofoflife.txt 
You did it!  This is the VM that you were trying to restore.
aborrero@ceph-restore-practice:~$ sudo rm /proofoflife.txt 
aborrero@ceph-restore-practice:~$ cat /proofoflife.txt 
cat: /proofoflife.txt: No such file or directory
aborrero@ceph-restore-practice:~$ sudo poweroff

The natural followup of this is in the 2020 network refresh project (clousw/cloudgw):

In T260890#6415421, @earthgecko wrote:

However that said, we pull 55 wikimedia metrics for testing external metric retrieval and analysis and do anomaly detection on them. I just selected 55 random ones that have continuous data, however seeing as they are being analysed, would the the Wikimedia Cloud Services Team have a list of metrics that are important, critical metrics that would be better to analyse?

This test retrieval and analysis has been going on for months on these metrics and I have never opened an issue because without any domain specific information, I could not say if any of the issues were truly significantly anomalous or not, in this case I am pretty certain the events are anomalous, albeit probably low priority :)

it works now!

we may need to add to our repo all the individual lang packages. Will do soon.

Thanks for the hint @Legoktm . Please @Andrew try again!

I would say unless someone is experiencing actual difficulties with the servers, the spikes themselves don't represent an issue.

In T195217#6331352, @Bstorm wrote:

I believe this is done but for the docs.

In T256361#6330849, @Bstorm wrote:

I did a little research and confirmed that Prometheus not only does not support setting the Host header, the development team is somewhat hostile to the idea of adding arbitrary headers to scrapes outside of auth headers. So we will not have jupyterhub stats until you can introspect a Kubernetes ingress-ed service from inside the cloud.

In T211096#6331379, @Bstorm wrote:

Timeline! https://wikitech.wikimedia.org/wiki/User:Bstorm/sandbox/paws_timeline
I think that should get this done.

In T211096#6331251, @Bstorm wrote:

Updated the command used to deploy and upgrade here https://wikitech.wikimedia.org/wiki/PAWS/Admin

After syncing with @ayounsi I started to document stuff as we know it here https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph#Network

In T256881#6328603, @MoritzMuehlenhoff wrote:

In T256881#6282184, @aborrero wrote:

There are two classes (openstack::serverpackages::queens::stretch and openstack::serverpackages::rocky::stretch which are pulling 'librados2', 'librgw2', 'librbd1', 'python-rados', 'python-rbd', 'ceph-common', 'python-cephfs', 'libradosstriper1' from stretch-backports. These will need to be mirrored to a local component.

I will investigate a bit more about these. They are dependencies of the openstack suite, so it would make sense to have them in the osbpo repository.

Did you hear back from the osbpo people? Otherwise let's copy these to something like component/ceph and include it in the openstack::serverpackages::queens::stretch and openstack::serverpackages::rocky::stretch classes? These are the last remaining packages, otherwise https://gerrit.wikimedia.org/r/c/operations/puppet/+/613611 would be good to merge

hey @Bstorm could you please confirm the prometheus thing you were trying works no?

Heads up, I'm reverting the changes introduced in this ticket, see T257534: CloudVPS: a VM is unable to contact floating IPs of other VMs for reference. I'm pretty sure the changes weren't working as expected anyway, and nobody noticed.

So, after a bit more investigation I'm confident I understand what's happening here.

I confirm the behavior is different in codfw1dev. Upon research, it turns out the dmz_cidr setting is different in both deployments.

I'm also investigating codfw1dev, because at quick glance it may behave differently and I don't know why yet.