Page MenuHomePhabricator

aborrero (arturo)
SRE at Wikimedia Cloud Services Team

Projects (8)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 23 2017, 12:19 PM (156 w, 2 d)
Availability
Available
IRC Nick
arturo
LDAP User
Arturo Borrero Gonzalez
MediaWiki User
ABorrero (WMF) [ Global Accounts ]

I'm Arturo Borrero Gonzalez from Spain (Seville). I'm Site Reliability Engineer (SRE) in the Wikimedia Cloud Services Team, a Wikimedia Foundation staff.

You may find me in some FLOSS projects, like Netfilter and Debian.

Recent Activity

Today

aborrero moved T151158: Support queries against Quarry's own database and ToolsDB from Needs discussion to Doing on the cloud-services-team (Kanban) board.
Wed, Oct 21, 3:44 PM · cloud-services-team (Kanban), Quarry
aborrero moved T265135: wikireplicas: Define MW sections per host from Needs discussion to Watching on the cloud-services-team (Kanban) board.
Wed, Oct 21, 3:44 PM · Data-Services, cloud-services-team (Kanban)
aborrero added a comment to T265288: Enable L3 routing on cloudsw nodes.

Ok, new proposal: 2020-10:-29

Wed, Oct 21, 9:25 AM · Operations, netops, cloud-services-team (Kanban)

Yesterday

aborrero added a comment to T260698: Update wikitech docs to prefer wmcloud.org and eqiad1.wikimedia.cloud domains.

thanks for working on this!

Tue, Oct 20, 4:41 PM · good first task, Documentation, cloud-services-team (Kanban)
aborrero awarded T260698: Update wikitech docs to prefer wmcloud.org and eqiad1.wikimedia.cloud domains a Pirate Logo token.
Tue, Oct 20, 4:40 PM · good first task, Documentation, cloud-services-team (Kanban)
aborrero added a comment to T265288: Enable L3 routing on cloudsw nodes.

New proposed date: 2020-11-03,

Tue, Oct 20, 4:34 PM · Operations, netops, cloud-services-team (Kanban)
aborrero added a comment to T265288: Enable L3 routing on cloudsw nodes.

That's fair. I will try proposing a new date tomorrow.

Tue, Oct 20, 4:31 PM · Operations, netops, cloud-services-team (Kanban)
aborrero added a comment to T265864: Remove 185.15.56.0/24 from network::external.

No problems on my side.

Tue, Oct 20, 9:51 AM · cloud-services-team (Kanban), Operations, netops
aborrero moved T265864: Remove 185.15.56.0/24 from network::external from Inbox to Watching on the cloud-services-team (Kanban) board.
Tue, Oct 20, 8:14 AM · cloud-services-team (Kanban), Operations, netops

Fri, Oct 16

aborrero updated the task description for T261724: cloudgw: evaluate / validate setup in codfw1dev.
Fri, Oct 16, 11:51 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a parent task for T265288: Enable L3 routing on cloudsw nodes: T244727: CloudVPS: networking improvements.
Fri, Oct 16, 11:50 AM · Operations, netops, cloud-services-team (Kanban)
aborrero added a subtask for T244727: CloudVPS: networking improvements: T265288: Enable L3 routing on cloudsw nodes.
Fri, Oct 16, 11:50 AM · cloud-services-team (Kanban), Epic

Thu, Oct 15

aborrero updated the task description for T261724: cloudgw: evaluate / validate setup in codfw1dev.
Thu, Oct 15, 1:56 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a comment to T265288: Enable L3 routing on cloudsw nodes.

Looking at our calendars, I think Tuesday, October 27 might work.

Thu, Oct 15, 10:38 AM · Operations, netops, cloud-services-team (Kanban)

Wed, Oct 14

aborrero moved T265288: Enable L3 routing on cloudsw nodes from Needs discussion to Watching on the cloud-services-team (Kanban) board.
Wed, Oct 14, 4:05 PM · Operations, netops, cloud-services-team (Kanban)
aborrero updated subscribers of T265288: Enable L3 routing on cloudsw nodes.

This change has impacts to Toolforge (NFS, databases, etc). We want to reduce the downtime, i.e, failover things etc. For this it would be good if we can do this operation during EU/US overlapping time.
Also, we would like to announce the operation window to the community 1 week prior.

Wed, Oct 14, 3:48 PM · Operations, netops, cloud-services-team (Kanban)
aborrero moved T265288: Enable L3 routing on cloudsw nodes from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
Wed, Oct 14, 3:40 PM · Operations, netops, cloud-services-team (Kanban)

Fri, Oct 9

aborrero awarded T264888: Review default ferm INPUT policy a The World Burns token.
Fri, Oct 9, 10:09 AM · Patch-For-Review, Security, Operations, netops, User-jbond

Thu, Oct 8

aborrero added a comment to T263622: codfw: more vlans setup changes in the cloudgw PoC.

Moreover, if I workaround the first validation, when I try to assign the address to the virtual router, I get:

Thu, Oct 8, 4:15 PM · cloud-services-team (Kanban)
aborrero added a comment to T263622: codfw: more vlans setup changes in the cloudgw PoC.

About vlan 2107, how many IPs do you need? If similar to the diagram a /31 should be enough.

Thu, Oct 8, 3:37 PM · cloud-services-team (Kanban)
aborrero added a comment to T263622: codfw: more vlans setup changes in the cloudgw PoC.

ok thanks! It works. I'm now able to reimage labtestvirt2003 (cloudgw).

Thu, Oct 8, 10:52 AM · cloud-services-team (Kanban)
aborrero updated the task description for T263622: codfw: more vlans setup changes in the cloudgw PoC.
Thu, Oct 8, 10:50 AM · cloud-services-team (Kanban)

Wed, Oct 7

aborrero updated the task description for T263622: codfw: more vlans setup changes in the cloudgw PoC.
Wed, Oct 7, 11:47 AM · cloud-services-team (Kanban)
aborrero triaged T263622: codfw: more vlans setup changes in the cloudgw PoC as High priority.
Wed, Oct 7, 11:45 AM · cloud-services-team (Kanban)
aborrero added a comment to T263622: codfw: more vlans setup changes in the cloudgw PoC.

New update: in order to workaround the constraints we found when working with the bonding+trunking setup, and per suggestion by @ayounsi, I think we should:

Wed, Oct 7, 9:17 AM · cloud-services-team (Kanban)

Fri, Oct 2

aborrero updated subscribers of T263622: codfw: more vlans setup changes in the cloudgw PoC.

The setup above was declined by @ayounsi

Fri, Oct 2, 10:40 AM · cloud-services-team (Kanban)
aborrero added a comment to T263622: codfw: more vlans setup changes in the cloudgw PoC.

When I try to reimage the server, the DHCP boot wont work. I believe the vlan configuration is right, and therefore suspect there is something going on in the link aggregation config.

Fri, Oct 2, 9:46 AM · cloud-services-team (Kanban)

Thu, Oct 1

aborrero added a comment to T264311: Prepare for puppetizing /etc/apt/sources.list.

I wonder if there are CloudVPS VMs in which the admin hand-edited the file. The moment we enable this, the changes would be lost, right?

Thu, Oct 1, 3:28 PM · cloud-services-team (Kanban)
aborrero updated the task description for T263622: codfw: more vlans setup changes in the cloudgw PoC.
Thu, Oct 1, 2:01 PM · cloud-services-team (Kanban)
aborrero added a comment to T263622: codfw: more vlans setup changes in the cloudgw PoC.

yes /31

Thu, Oct 1, 2:00 PM · cloud-services-team (Kanban)
aborrero added a comment to T263284: Upgrade Toolforge K8s to 1.17.

The 1.17 packages are in the repo.

Thu, Oct 1, 11:25 AM · Kubernetes, Toolforge, cloud-services-team (Kanban)
aborrero closed T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) as Resolved.

thanks!

Thu, Oct 1, 9:34 AM · cloud-services-team (Kanban)

Wed, Sep 30

aborrero added a comment to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff).

Sent a report upstream: https://marc.info/?l=netfilter-devel&m=160145994003146&w=2

Wed, Sep 30, 10:03 AM · cloud-services-team (Kanban)
aborrero reopened T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) as "Open".

We are experiencing more issues, reopening.

Wed, Sep 30, 10:02 AM · cloud-services-team (Kanban)
aborrero awarded T263480: Create second revision of Phab tutorial videos (smaller improvements etc based on feedback) a Love token.
Wed, Sep 30, 8:36 AM · Developer-Advocacy (Jan-Mar 2021), Documentation

Tue, Sep 29

aborrero added a comment to T263622: codfw: more vlans setup changes in the cloudgw PoC.

I updated the task description to include info on the native vlan we need in order to install the server.

Tue, Sep 29, 11:39 AM · cloud-services-team (Kanban)
aborrero renamed T263622: codfw: more vlans setup changes in the cloudgw PoC from cofdw: enable more vlans in the cloudgw PoC to cofdw: more vlans setup changes in the cloudgw PoC.
Tue, Sep 29, 11:35 AM · cloud-services-team (Kanban)

Fri, Sep 25

aborrero added a comment to T263774: Cloudcontrol split brain issues.

Additional context: there were some ongoing ops in the eqiad datacenter when this issue happened. I didn't have time yet to investigate what exactly happened, but a prod DNS server might have been down.

Fri, Sep 25, 11:58 AM · Cloud-VPS, cloud-services-team (Kanban)

Wed, Sep 23

aborrero moved T263622: codfw: more vlans setup changes in the cloudgw PoC from Inbox to Watching on the cloud-services-team (Kanban) board.
Wed, Sep 23, 9:38 AM · cloud-services-team (Kanban)
aborrero created T263622: codfw: more vlans setup changes in the cloudgw PoC.
Wed, Sep 23, 9:24 AM · cloud-services-team (Kanban)

Tue, Sep 22

brennen awarded T221212: spicerack/cookbook: add additional arguments IRC/SAL logging a Like token.
Tue, Sep 22, 5:31 PM · Patch-For-Review, SRE-tools, Operations

Sep 18 2020

Bstorm awarded T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) a Yellow Medal token.
Sep 18 2020, 3:30 PM · cloud-services-team (Kanban)
aborrero renamed T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) from Buster cloudvirts unable to launch new VMs to cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff).
Sep 18 2020, 10:13 AM · cloud-services-team (Kanban)
aborrero closed T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) as Resolved.

With all that being said, I'm closing the task. Please reopen if required!

Sep 18 2020, 10:12 AM · cloud-services-team (Kanban)
aborrero added a comment to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff).

I suspect what's happening here is:

Sep 18 2020, 10:06 AM · cloud-services-team (Kanban)
aborrero closed T263205: Strange NFS client outage on VMs running on cloudvirt1036 as Resolved.

With the patches I merged related to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) I suspect this issue wont happen again.

Sep 18 2020, 9:59 AM · cloud-services-team (Kanban), Cloud-VPS, Data-Services
aborrero added a comment to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff).

it seems we don't have any buster cloudvirt in codfw1dev. It would have been awesome to be able to test this there first.

Sep 18 2020, 9:03 AM · cloud-services-team (Kanban)
aborrero added a comment to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff).

I think T263205: Strange NFS client outage on VMs running on cloudvirt1036 is related to this issue. I discovered the same error on cloudvirt1036, which banished as soon as I installed the newer iptables package.

Sep 18 2020, 8:54 AM · cloud-services-team (Kanban)
aborrero added a comment to T263205: Strange NFS client outage on VMs running on cloudvirt1036.

for the record:

Sep 18 2020, 8:48 AM · cloud-services-team (Kanban), Cloud-VPS, Data-Services
aborrero added a comment to T263205: Strange NFS client outage on VMs running on cloudvirt1036.

this can be related to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff), I'm investigating.

Sep 18 2020, 8:48 AM · cloud-services-team (Kanban), Cloud-VPS, Data-Services

Sep 16 2020

aborrero updated the task description for T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff).
Sep 16 2020, 10:07 AM · cloud-services-team (Kanban)
aborrero triaged T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) as Medium priority.
Sep 16 2020, 10:07 AM · cloud-services-team (Kanban)
aborrero added a comment to T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff).

I guess the reboot resulted in the server running a newer kernel.

Sep 16 2020, 10:06 AM · cloud-services-team (Kanban)

Sep 10 2020

aborrero added a comment to T260614: Phase out use of .wmflabs tld.

Not sure if @Andrew is aware of this issue, but posting here for the record anyway.

Sep 10 2020, 10:40 AM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
aborrero closed T250172: Toolforge: k8s: ingress: consider creating ingress-specific nodes as Resolved.

done!

Sep 10 2020, 10:34 AM · cloud-services-team (Kanban), Tools

Sep 9 2020

aborrero added a comment to T250172: Toolforge: k8s: ingress: consider creating ingress-specific nodes.

mmm will do toolsbeta first just in case.

Sep 9 2020, 11:22 AM · cloud-services-team (Kanban), Tools

Sep 8 2020

aborrero added a comment to T256881: wmcs: evaluate impact of stretch-backports being archived.

moving this again to doing. Will try to take a final look this week.

Sep 8 2020, 4:40 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero moved T256881: wmcs: evaluate impact of stretch-backports being archived from Soon! to Doing on the cloud-services-team (Kanban) board.
Sep 8 2020, 4:39 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero claimed T250172: Toolforge: k8s: ingress: consider creating ingress-specific nodes.

Raising priority, @Andrew mentioned that with the new domains for VMs we should try creating new k8s nodes and see how that works. This seems like the right test.

Sep 8 2020, 4:34 PM · cloud-services-team (Kanban), Tools
aborrero closed T262237: The TLS certificate for https://wma.wmflabs.org is expired as Resolved.

This should be fixed now. Thanks @Vgutierrez for the assistance on IRC.

Sep 8 2020, 10:14 AM · cloud-services-team (Kanban), Cloud-VPS
aborrero claimed T262237: The TLS certificate for https://wma.wmflabs.org is expired.

The acme-chief backend gets OCSPResponseStatus.UNAUTHORIZED and can't generate the new certs apparently.

Sep 8 2020, 9:54 AM · cloud-services-team (Kanban), Cloud-VPS
aborrero triaged T262237: The TLS certificate for https://wma.wmflabs.org is expired as High priority.
Sep 8 2020, 8:56 AM · cloud-services-team (Kanban), Cloud-VPS
aborrero added a project to T262237: The TLS certificate for https://wma.wmflabs.org is expired: cloud-services-team (Kanban).
Sep 8 2020, 8:55 AM · cloud-services-team (Kanban), Cloud-VPS
aborrero added a comment to T260890: Possible issue on cloudinfra.cloud-cumin-01.

For the record, what happened in august is mostly related to us moving workload to Ceph, read more here:

Sep 8 2020, 8:46 AM · cloud-services-team (Kanban), Cloud-VPS
aborrero reopened T260890: Possible issue on cloudinfra.cloud-cumin-01 as "Open".

I will discuss this with the team and get back to you in a couple of days.

Sep 8 2020, 8:39 AM · cloud-services-team (Kanban), Cloud-VPS

Sep 7 2020

aborrero closed T260941: Practice restoring ceph backups as Resolved.

Before start:

aborrero@ceph-restore-practice:~$ cat /proofoflife.txt 
You did it!  This is the VM that you were trying to restore.
aborrero@ceph-restore-practice:~$ sudo rm /proofoflife.txt 
aborrero@ceph-restore-practice:~$ cat /proofoflife.txt 
cat: /proofoflife.txt: No such file or directory
aborrero@ceph-restore-practice:~$ sudo poweroff
Sep 7 2020, 3:36 PM · Goal, cloud-services-team (Kanban)
aborrero closed T260941: Practice restoring ceph backups, a subtask of T260692: Ceph VM image backups, as Resolved.
Sep 7 2020, 3:36 PM · Patch-For-Review, Goal, cloud-services-team (Kanban)

Sep 4 2020

aborrero updated the task description for T261724: cloudgw: evaluate / validate setup in codfw1dev.
Sep 4 2020, 8:41 AM · Patch-For-Review, cloud-services-team (Kanban)

Sep 2 2020

aborrero updated the task description for T261724: cloudgw: evaluate / validate setup in codfw1dev.
Sep 2 2020, 9:31 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero updated the task description for T261724: cloudgw: evaluate / validate setup in codfw1dev.
Sep 2 2020, 8:39 AM · Patch-For-Review, cloud-services-team (Kanban)

Sep 1 2020

aborrero updated the task description for T261724: cloudgw: evaluate / validate setup in codfw1dev.
Sep 1 2020, 10:34 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero triaged T261724: cloudgw: evaluate / validate setup in codfw1dev as Medium priority.
Sep 1 2020, 9:59 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero created T261724: cloudgw: evaluate / validate setup in codfw1dev.
Sep 1 2020, 9:59 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a comment to T209460: CloudVPS: our ideal future model.

The natural followup of this is in the 2020 network refresh project (clousw/cloudgw):

Sep 1 2020, 9:56 AM · Operations, cloud-services-team (Kanban), Epic

Aug 27 2020

aborrero added a comment to T260890: Possible issue on cloudinfra.cloud-cumin-01.

However that said, we pull 55 wikimedia metrics for testing external metric retrieval and analysis and do anomaly detection on them. I just selected 55 random ones that have continuous data, however seeing as they are being analysed, would the the Wikimedia Cloud Services Team have a list of metrics that are important, critical metrics that would be better to analyse?

This test retrieval and analysis has been going on for months on these metrics and I have never opened an issue because without any domain specific information, I could not say if any of the issues were truly significantly anomalous or not, in this case I am pretty certain the events are anomalous, albeit probably low priority :)

Aug 27 2020, 2:54 PM · cloud-services-team (Kanban), Cloud-VPS

Aug 25 2020

aborrero closed T247422: Update Tesseract on Toolforge to v4.1.0 as Resolved.

it works now!

Aug 25 2020, 10:46 AM · cloud-services-team (Kanban), Toolforge (Software install/update)
aborrero closed T247422: Update Tesseract on Toolforge to v4.1.0, a subtask of T244100: Spike: New/Improved OCR tool [8 hours], as Resolved.
Aug 25 2020, 10:45 AM · Community-Tech (Kanban-2019-20-Q4), Internet-Archive

Aug 24 2020

aborrero added a comment to T259399: Upgrade cloudvirts to Debian Buster.
NOTE: I tested ebtables-nft 1.8.3 from buster-bpo and it doesn't work. We would need 1.8.5 per iptables upstream changelog (among support).
Aug 24 2020, 4:07 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
aborrero added a comment to T247422: Update Tesseract on Toolforge to v4.1.0.

we may need to add to our repo all the individual lang packages. Will do soon.

Aug 24 2020, 3:47 PM · cloud-services-team (Kanban), Toolforge (Software install/update)
aborrero added a comment to T247422: Update Tesseract on Toolforge to v4.1.0.

Thanks for the hint @Legoktm . Please @Andrew try again!

Aug 24 2020, 11:35 AM · cloud-services-team (Kanban), Toolforge (Software install/update)
aborrero closed T260890: Possible issue on cloudinfra.cloud-cumin-01 as Declined.

I would say unless someone is experiencing actual difficulties with the servers, the spikes themselves don't represent an issue.

Aug 24 2020, 11:22 AM · cloud-services-team (Kanban), Cloud-VPS

Aug 12 2020

Krinkle awarded T221212: spicerack/cookbook: add additional arguments IRC/SAL logging a Orange Medal token.
Aug 12 2020, 4:06 PM · Patch-For-Review, SRE-tools, Operations

Jul 24 2020

aborrero reassigned T253836: Update quotas for MWoffliner VPS from aborrero to Andrew.
Jul 24 2020, 10:32 AM · Cloud-VPS (Quota-requests), affects-Kiwix-and-openZIM
aborrero updated the task description for T195217: Simplify ingress methods for PAWS.
Jul 24 2020, 10:18 AM · Patch-For-Review, cloud-services-team (Kanban), PAWS (zero-to-jupyterhub-k8s 0.8.0)
aborrero closed T195217: Simplify ingress methods for PAWS, a subtask of T211096: PAWS: Rebuild and upgrade Kubernetes, as Resolved.
Jul 24 2020, 10:17 AM · Patch-For-Review, Toolforge, Epic, Goal, cloud-services-team (Kanban), PAWS
aborrero closed T195217: Simplify ingress methods for PAWS, a subtask of T218150: Update PAWS with Zero to JupyterHub k8s 0.8.0 chart, as Resolved.
Jul 24 2020, 10:17 AM · PAWS (zero-to-jupyterhub-k8s 0.8.0)
aborrero closed T195217: Simplify ingress methods for PAWS as Resolved.

I believe this is done but for the docs.

Jul 24 2020, 10:17 AM · Patch-For-Review, cloud-services-team (Kanban), PAWS (zero-to-jupyterhub-k8s 0.8.0)
aborrero added a comment to T256361: PAWS: get new service and cluster metrics into prometheus.

I did a little research and confirmed that Prometheus not only does not support setting the Host header, the development team is somewhat hostile to the idea of adding arbitrary headers to scrapes outside of auth headers. So we will not have jupyterhub stats until you can introspect a Kubernetes ingress-ed service from inside the cloud.

Jul 24 2020, 9:58 AM · Patch-For-Review, cloud-services-team (Kanban), PAWS
aborrero added a comment to T211096: PAWS: Rebuild and upgrade Kubernetes.
Jul 24 2020, 9:43 AM · Patch-For-Review, Toolforge, Epic, Goal, cloud-services-team (Kanban), PAWS
aborrero added a comment to T211096: PAWS: Rebuild and upgrade Kubernetes.

Updated the command used to deploy and upgrade here https://wikitech.wikimedia.org/wiki/PAWS/Admin

Jul 24 2020, 9:37 AM · Patch-For-Review, Toolforge, Epic, Goal, cloud-services-team (Kanban), PAWS

Jul 23 2020

aborrero updated subscribers of T258619: Decide on network setup for WMCS ceph cluster.

After syncing with @ayounsi I started to document stuff as we know it here https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph#Network

Jul 23 2020, 11:52 AM · Goal, cloud-services-team (Kanban)
aborrero added a comment to T256881: wmcs: evaluate impact of stretch-backports being archived.

There are two classes (openstack::serverpackages::queens::stretch and openstack::serverpackages::rocky::stretch which are pulling 'librados2', 'librgw2', 'librbd1', 'python-rados', 'python-rbd', 'ceph-common', 'python-cephfs', 'libradosstriper1' from stretch-backports. These will need to be mirrored to a local component.

I will investigate a bit more about these. They are dependencies of the openstack suite, so it would make sense to have them in the osbpo repository.

Did you hear back from the osbpo people? Otherwise let's copy these to something like component/ceph and include it in the openstack::serverpackages::queens::stretch and openstack::serverpackages::rocky::stretch classes? These are the last remaining packages, otherwise https://gerrit.wikimedia.org/r/c/operations/puppet/+/613611 would be good to merge

Jul 23 2020, 11:22 AM · Patch-For-Review, cloud-services-team (Kanban)

Jul 20 2020

aborrero awarded T102367: Migrate tools.wmflabs.org to https only (and set HSTS) a Love token.
Jul 20 2020, 1:01 PM · User-bd808, User-QEDK, Patch-For-Review, cloud-services-team (Kanban), Operations, Traffic, HTTPS, Toolforge

Jul 16 2020

aborrero added a comment to T257534: CloudVPS: a VM is unable to contact floating IPs of other VMs.

hey @Bstorm could you please confirm the prometheus thing you were trying works no?

Jul 16 2020, 11:57 AM · cloud-services-team (Kanban), PAWS
aborrero added a comment to T206261: Routing RFC1918 private IP addresses to/from WMCS floating IPs.

Heads up, I'm reverting the changes introduced in this ticket, see T257534: CloudVPS: a VM is unable to contact floating IPs of other VMs for reference. I'm pretty sure the changes weren't working as expected anyway, and nobody noticed.

Jul 16 2020, 10:16 AM · cloud-services-team (Kanban), User-herron, Operations, Cloud-Services, Mail
aborrero added a comment to T257534: CloudVPS: a VM is unable to contact floating IPs of other VMs.

So, after a bit more investigation I'm confident I understand what's happening here.

Jul 16 2020, 10:15 AM · cloud-services-team (Kanban), PAWS

Jul 15 2020

aborrero added a comment to T257534: CloudVPS: a VM is unable to contact floating IPs of other VMs.

I confirm the behavior is different in codfw1dev. Upon research, it turns out the dmz_cidr setting is different in both deployments.

Jul 15 2020, 4:55 PM · cloud-services-team (Kanban), PAWS
aborrero added a comment to T257534: CloudVPS: a VM is unable to contact floating IPs of other VMs.

I'm also investigating codfw1dev, because at quick glance it may behave differently and I don't know why yet.

Jul 15 2020, 12:00 PM · cloud-services-team (Kanban), PAWS
aborrero renamed T257534: CloudVPS: a VM is unable to contact floating IPs of other VMs from CloudVPS: issues when routing to static internal IP to CloudVPS: a VM is unable to contact floating IPs of other VMs.
Jul 15 2020, 11:03 AM · cloud-services-team (Kanban), PAWS

Jul 14 2020

Nintendofan885 awarded T234617: Toolforge. introduce new domain toolforge.org a Party Time token.
Jul 14 2020, 8:58 PM · Patch-For-Review, Goal, Toolforge, cloud-services-team (Kanban), Kubernetes