Closing as invalid for now, see T276449#6883228

Closing as invalid for now, see T276449#6883228

Closing as invalid for now, see T276449#6883228

In T276327#6878924, @Andrew wrote:

Does this imply multiattach support? I've investigated that a bit but haven't seen it work yet.

In T268393#6867249, @Ladsgroup wrote:

[..]
I put the full error in P14517 (I redacted as much as possible, let me know if you want the original data). There's no error or warning beforehand or afterwards, it's just healthchecks (and all okay) but suddenly it starts dropping packets.

Do we have some sort of ratelimit in the cloud infra that might have affected this? Or is it behind CloudFlare and it's being aggressive?

ok, updating the firmware-bnx2x + upgrading the kernel did the trick apparently.

Rebooting cloudnet1003 into the new kernel failed to bring interfaces up:

From my experience with the wmcs-drain-hypervisor.py script today, I think we can improve our workflows a bit:

VMs with the VMX cpu flag, barring some unreachable VMs:

We can safely ignore these errors, the VM is short lived, for testing purposes.

I don't see errors anymore on cloudnet1004 after the kernel upgrade. I think we should do the same in the other server (cloudnet1003).

Looking good so far. I'll wait another couple days before drawing more conclusions.

This is done, we merged the puppet change, plus:

• we don't reload neutron-l3-agent on puppet changes (on purpose, to avoid failover noise)
• in cloudnet servers manually delete iptables rules that were previously implementing the NAT exception

Ok, now running the new kernel, will leave it running at least a couple of days and see what happens:

BTW, this is just a cosmetic thing, so I'm changing priority to 'lowest' :-)

This is solved by now:

Unfortunately the server still shows the same problems, and even self-rebooted over night.

My expectation would be something like:

Rebooted the server for a clean start, I see the driver and firmware being loaded. I'll leave here the output for reference:

aborrero@cloudnet1004:~ $ sudo dmesg -T | grep bnx2x
[Thu Feb 18 14:52:35 2021] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.712.30-0 (2014/02/10)
[Thu Feb 18 14:52:35 2021] bnx2x 0000:04:00.0: msix capability found
[Thu Feb 18 14:52:35 2021] bnx2x 0000:04:00.0: part number 394D4342-31383735-31543030-47303030
[Thu Feb 18 14:52:36 2021] bnx2x 0000:04:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[Thu Feb 18 14:52:36 2021] bnx2x 0000:04:00.1: msix capability found
[Thu Feb 18 14:52:36 2021] bnx2x 0000:04:00.1: part number 394D4342-31383735-31543030-47303030
[Thu Feb 18 14:52:37 2021] bnx2x 0000:04:00.1: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[Thu Feb 18 14:52:37 2021] bnx2x 0000:04:00.0 eno49: renamed from eth0
[Thu Feb 18 14:52:38 2021] bnx2x 0000:04:00.1 eno50: renamed from eth1
[Thu Feb 18 14:52:49 2021] bnx2x 0000:04:00.0: firmware: direct-loading firmware bnx2x/bnx2x-e2-7.13.1.0.fw
[Thu Feb 18 14:52:50 2021] bnx2x 0000:04:00.0 eno49: using MSI-X  IRQs: sp 53  fp[0] 55 ... fp[7] 62
[Thu Feb 18 14:52:50 2021] bnx2x 0000:04:00.0 eno49: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
[Thu Feb 18 14:52:57 2021] bnx2x 0000:04:00.1 eno50: using MSI-X  IRQs: sp 64  fp[0] 66 ... fp[7] 75
[Thu Feb 18 14:52:58 2021] bnx2x 0000:04:00.1 eno50: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit

Again, I'm not familiar with the codebase, but what about simple try: when calling the cookbook backend, catch that particular error and print something more friendly.

We decided to:

• send a brief heads up message to community mailing lists about this change
• schedule an operation window next week (probably 2020-02-23)

I don't know the codebase, but perhaps showing the help message before configuring the log is an "easy" fix for this particular case.

HINT: activate conntrack_tcp_be_liberal in the neutron netns.

This is in very good shape. I tested several failover scenarios:

• manually stop keepalived in the primary VRRP node
• reboot of the primary VRRP node
• flapping (backup -> primary -> backup -> primary)

The change is ready. I will coordinate with @Bstorm for an operation window soon.

We decided to decline this task in the backlog grooming meeting.

In T143639#6298096, @Bstorm wrote:

At this point, we have a PoC over in the PAWS project. If we can demonstrate good behavior in a quick failover test, we could start to deploy the functionality more widely. There's a small routing issue to handle first brought up in T257534

No longer valid.

Solved by https://gerrit.wikimedia.org/r/c/operations/puppet/+/663027

We believe this is done.

In T274386#6819605, @bd808 wrote:

Untested idea for multi-hop:

--- i/modules/dynamicproxy/templates/domainproxy.conf
+++ w/modules/dynamicproxy/templates/domainproxy.conf
@@ -148,6 +148,11 @@ server {
         proxy_pass $backend;
         proxy_set_header Host $vhost;

+        # T274386: allow unverified TLS to upstream when $backend starts with
+        # `https://` to trigger TLS usage by `proxy_pass`.
+        proxy_ssl_verify off;
+        proxy_ssl_session_reuse on;
+
         proxy_set_header X-Forwarded-Proto $scheme;

         proxy_http_version 1.1;

This would also need changes in the data retrieved via domainproxy.lua to allow it to return an https:// prefixed upstream route when appropriate. We may also want to add proxy_ssl_protocols and proxy_ssl_ciphers settings in this block to tune the "internal" TLS connection configuration.

I have no idea, perhaps we should ask @RobH who created this task. This sever wasn't in my radar before this task.

It seems we send the client address to the NFS server?