Page MenuHomePhabricator

aborrero (arturo)
Operations Engineer at Wikimedia Cloud Services Team

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 23 2017, 12:19 PM (104 w, 20 m)
Availability
Available
IRC Nick
arturo
LDAP User
Arturo Borrero Gonzalez
MediaWiki User
ABorrero (WMF) [ Global Accounts ]

I'm Arturo Borrero Gonzalez from Spain (Seville). I'm Operations Engineer as part of the Wikimedia Cloud Services Team, a Wikimedia Foundation staff.

You may found me in some FLOSS projects, like Netfilter and Debian.

Recent Activity

Today

aborrero added a comment to T235627: Toolforge: upgrade main proxy servers to Debian Buster.

Proposed operation steps:

Mon, Oct 21, 10:35 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero updated the task description for T235627: Toolforge: upgrade main proxy servers to Debian Buster.
Mon, Oct 21, 10:19 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T235627: Toolforge: upgrade main proxy servers to Debian Buster.

Operation announcement sent: https://lists.wikimedia.org/pipermail/cloud-announce/2019-October/000226.html

Mon, Oct 21, 10:19 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T235846: wikimedia.cloud: setup new domain.

Copying from the patch for visibility: Let's please not put anything in the admin project that doesn't strictly need to be there - IIRC it has some special meaning/purpose, novaobserver's rights don't apply there, and it may cause difficulty splitting up permissions later on if we wanted to.
We can just make an empty project (in terms of instances) to hold a DNS zone or two.

Mon, Oct 21, 9:16 AM · Patch-For-Review, cloud-services-team (Kanban)

Fri, Oct 18

aborrero added a comment to T235863: Openstack: designate API issues.

The patch seems simple enough that perhaps we can do the patching ourselves? What do you think?

--- a/designateclient/v2/client.py
+++ b/designateclient/v2/client.py
@@ -58,7 +58,7 @@ class DesignateAdapter(adapter.LegacyJsonAdapter):
         if self.all_projects:
             kwargs['headers'].setdefault(
                 'X-Auth-All-Projects',
-                self.all_projects
+                str(self.all_projects)
             )
         if self.edit_managed:

I think that's a good idea. What's our best approach to adding a patch to the python-designateclient=2.3.0-2 package? I was looking at using dpkg-source --commit but I'm not sure how we should increment the version.

Fri, Oct 18, 4:37 PM · cloud-services-team (Kanban)
aborrero added a comment to T235863: Openstack: designate API issues.

@aborrero can you reproduce this now? I just ran some tests (in the 'testlabs' project) and things seemed ok.

Fri, Oct 18, 4:00 PM · cloud-services-team (Kanban)
aborrero added a comment to T235846: wikimedia.cloud: setup new domain.

+1 for the admin project! Will try creating it now.

Fri, Oct 18, 3:57 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero triaged T235863: Openstack: designate API issues as High priority.
Fri, Oct 18, 12:45 PM · cloud-services-team (Kanban)
aborrero created T235863: Openstack: designate API issues.
Fri, Oct 18, 12:45 PM · cloud-services-team (Kanban)
aborrero added a comment to T235846: wikimedia.cloud: setup new domain.

It is not clear to me which project should own the eqiad1.wikimedia.cloud subdomain. The admin project? The wmflabsdotorg project? a new one?
I remember there was a special case for domains with no project association, I guess that's the case of the current eqiad.wmflabs:

Fri, Oct 18, 11:47 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero moved T235846: wikimedia.cloud: setup new domain from Inbox to Important on the cloud-services-team (Kanban) board.
Fri, Oct 18, 11:04 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero triaged T235846: wikimedia.cloud: setup new domain as Normal priority.
Fri, Oct 18, 11:03 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero created T235846: wikimedia.cloud: setup new domain.
Fri, Oct 18, 11:03 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero renamed T235630: Update authoratiative nameservers for the wmcloud.org domain to point to Designate from Update authoratiative nameservers for the wmcloud.org and wikimedia.cloud domains to point to Designate to Update authoratiative nameservers for the wmcloud.org domain to point to Designate.
Fri, Oct 18, 10:59 AM · Traffic, DNS, Operations, cloud-services-team (Kanban)
aborrero added a comment to T234232: Hosts in puppet with $cluster missing from wikimedia_clusters.

Much better now, thanks!

Fri, Oct 18, 10:23 AM · Operations, observability
aborrero added a comment to T234232: Hosts in puppet with $cluster missing from wikimedia_clusters.

In CloudVPS every VM I could check have this puppet agent error now:

Fri, Oct 18, 9:54 AM · Operations, observability
aborrero triaged T235825: wikitech SAL: handle '{' and '}' in log entries as Low priority.
Fri, Oct 18, 9:13 AM · cloud-services-team (Kanban), Toolforge, wikitech.wikimedia.org
aborrero created T235825: wikitech SAL: handle '{' and '}' in log entries.
Fri, Oct 18, 9:12 AM · cloud-services-team (Kanban), Toolforge, wikitech.wikimedia.org
aborrero added a comment to T235303: Update authoratiative nameservers for the toolforge.org domain to point to Designate.

Any news on this one? CC @Andrew @RobH

Fri, Oct 18, 9:01 AM · Traffic, Operations, DNS, Toolforge, cloud-services-team (Kanban)

Thu, Oct 17

aborrero triaged T235756: Toolforge: webservice utility: add support for thew new k8s setup as Normal priority.
Thu, Oct 17, 12:25 PM · Goal, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero created T235756: Toolforge: webservice utility: add support for thew new k8s setup.
Thu, Oct 17, 12:25 PM · Goal, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T223458: mgmt outages for cloud* systems seem to page everyone.

Apparently with the current setup, nobody in SRE is paged (by SMS) because mgmt interfaces, but we WMCS do get paged:

Thu, Oct 17, 12:03 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a comment to T235627: Toolforge: upgrade main proxy servers to Debian Buster.

cool thanks! I will work on the operation steps soon for you to review.

Thu, Oct 17, 8:50 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero awarded T235708: Redesign for wmcs custom puppet settings a Love token.
Thu, Oct 17, 8:42 AM · cloud-services-team (Kanban), Cloud-VPS

Wed, Oct 16

aborrero triaged T235630: Update authoratiative nameservers for the wmcloud.org domain to point to Designate as Normal priority.
Wed, Oct 16, 12:07 PM · Traffic, DNS, Operations, cloud-services-team (Kanban)
aborrero created T235630: Update authoratiative nameservers for the wmcloud.org domain to point to Designate.
Wed, Oct 16, 12:07 PM · Traffic, DNS, Operations, cloud-services-team (Kanban)
aborrero added a comment to T234037: Toolforge ingress: decide on final layout of north-south proxy setup.

Drafting docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_ingress

Wed, Oct 16, 11:48 AM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T235627: Toolforge: upgrade main proxy servers to Debian Buster.

Please @Bstorm and @JHedden confirm the scheduled operation window works for you both, thanks!

Wed, Oct 16, 11:33 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero triaged T235627: Toolforge: upgrade main proxy servers to Debian Buster as Normal priority.
Wed, Oct 16, 11:31 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero updated the task description for T235627: Toolforge: upgrade main proxy servers to Debian Buster.
Wed, Oct 16, 11:30 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero created T235627: Toolforge: upgrade main proxy servers to Debian Buster.
Wed, Oct 16, 11:29 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero closed T235059: Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster, a subtask of T234037: Toolforge ingress: decide on final layout of north-south proxy setup, as Resolved.
Wed, Oct 16, 11:24 AM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero closed T235059: Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster as Resolved.

This is done.

Wed, Oct 16, 11:24 AM · Toolforge, cloud-services-team (Kanban), Kubernetes

Mon, Oct 14

aborrero added a comment to T234037: Toolforge ingress: decide on final layout of north-south proxy setup.

Ok, I've been playing with the dynamicproxy nginx+lua components and I have a working setup. I disabled SSL/https in my tests until we handle T235252: Toolforge: SSL support for new domain toolforge.org.

Mon, Oct 14, 4:13 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T235252: Toolforge: SSL support for new domain toolforge.org.

Yep.

toolforge:
  CN: toolforge.org
  SNI:
  - toolforge.org
  - '*.toolforge.org'
  - tools.wmflabs.org
  - '*.tools.wmflabs.org'
  authorized_regexes:
  - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$
  challenge: dns-01
Mon, Oct 14, 10:31 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero assigned T235303: Update authoratiative nameservers for the toolforge.org domain to point to Designate to Andrew.

I believe the last person doing this kind of movement was @Andrew, so assigning to him to see if he can provide further info. I couldn't find any docs on wikitech.

Mon, Oct 14, 9:47 AM · Traffic, Operations, DNS, Toolforge, cloud-services-team (Kanban)

Fri, Oct 11

aborrero added a comment to T235252: Toolforge: SSL support for new domain toolforge.org.

ping @Krenair would you like to help me in this project? also T234617: Toolforge. introduce new domain toolforge.org

Fri, Oct 11, 12:55 PM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T235059: Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster.

I think the patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/508560 is mostly ready.

Fri, Oct 11, 12:48 PM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero created T235257: Toolforge: updatetools service is failing.
Fri, Oct 11, 10:21 AM · Toolforge, cloud-services-team (Kanban)
aborrero added a comment to T234231: Toolforge ingress: decide on how ingress configuration objects will be managed.

We had a team meeting on 2019-10-10 and we decided to try the custom admission controller to enforce correctness in ingress objects.

Fri, Oct 11, 9:05 AM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T234037: Toolforge ingress: decide on final layout of north-south proxy setup.

We had a meeting yesterday 2019-10-10 and we decided to try option 3 first, with fallback to option 2.

Fri, Oct 11, 9:03 AM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero triaged T235252: Toolforge: SSL support for new domain toolforge.org as Normal priority.
Fri, Oct 11, 8:57 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T235252: Toolforge: SSL support for new domain toolforge.org.

@Vgutierrez could you please advice on how to proceed with this?

Fri, Oct 11, 8:57 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero created T235252: Toolforge: SSL support for new domain toolforge.org.
Fri, Oct 11, 8:56 AM · Toolforge, cloud-services-team (Kanban), Kubernetes

Thu, Oct 10

aborrero updated the task description for T227536: b1-eqiad pdu refresh (Thursday 10/10 @11am UTC).
Thu, Oct 10, 10:45 AM · DC-Ops, Operations, ops-eqiad
aborrero closed T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress , a subtask of T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07, as Resolved.
Thu, Oct 10, 10:35 AM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
aborrero closed T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress as Resolved.
Thu, Oct 10, 10:35 AM · cloud-services-team (Kanban)
aborrero added a comment to T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress .

Thanks! Done already:

Thu, Oct 10, 9:26 AM · cloud-services-team (Kanban)

Wed, Oct 9

aborrero triaged T235070: Resolve keystone schema errors as High priority.
Wed, Oct 9, 2:14 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
aborrero created T235061: CloudVPS: create a wrapper script to ease virsh console to VMs.
Wed, Oct 9, 12:12 PM · cloud-services-team (Kanban)
aborrero triaged T235059: Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster as Normal priority.
Wed, Oct 9, 11:59 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero created T235059: Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster.
Wed, Oct 9, 11:59 AM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero assigned T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress to Andrew.

We agreed on doing this in the WMCS team meeting. Assigning to @Andrew for final approval before doing the changes.

Wed, Oct 9, 9:45 AM · cloud-services-team (Kanban)

Tue, Oct 8

aborrero renamed T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress from CloudVPS: update DNS record for eqiad1 routing_source_ip to CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress .
Tue, Oct 8, 2:07 PM · cloud-services-team (Kanban)
aborrero moved T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress from Important to Needs discussion on the cloud-services-team (Kanban) board.
Tue, Oct 8, 2:07 PM · cloud-services-team (Kanban)
aborrero added a subtask for T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07: T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress .
Tue, Oct 8, 12:36 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
aborrero added a parent task for T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress : T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07.
Tue, Oct 8, 12:36 PM · cloud-services-team (Kanban)
aborrero added a comment to T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress .

I'm proposing 2 changes:

Tue, Oct 8, 11:26 AM · cloud-services-team (Kanban)
aborrero added a comment to T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress .

More things to adjust:

Tue, Oct 8, 10:42 AM · cloud-services-team (Kanban)
aborrero added a subtask for T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07: T234876: nova-conductor running out of mysql connections.
Tue, Oct 8, 9:32 AM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
aborrero added a parent task for T234876: nova-conductor running out of mysql connections: T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07.
Tue, Oct 8, 9:32 AM · cloud-services-team (Kanban)
aborrero removed a subtask for T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton: T234830: CloudVPS: m5-master databases for openstack may require re-encoding.
Tue, Oct 8, 9:31 AM · Cloud-VPS, Patch-For-Review, cloud-services-team (Kanban)
aborrero added a subtask for T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07: T234830: CloudVPS: m5-master databases for openstack may require re-encoding.
Tue, Oct 8, 9:31 AM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
aborrero edited parent tasks for T234830: CloudVPS: m5-master databases for openstack may require re-encoding, added: T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07; removed: T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton.
Tue, Oct 8, 9:31 AM · cloud-services-team (Kanban), Cloud-VPS, DBA
aborrero added a comment to T234876: nova-conductor running out of mysql connections.

I checked nova-fullstack this morning. Everything looks good. No leaks so far.

Tue, Oct 8, 9:27 AM · cloud-services-team (Kanban)

Mon, Oct 7

aborrero lowered the priority of T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07 from Unbreak Now! to High.
Mon, Oct 7, 4:46 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
aborrero triaged T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress as Normal priority.
Mon, Oct 7, 4:44 PM · cloud-services-team (Kanban)
aborrero created T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress .
Mon, Oct 7, 4:43 PM · cloud-services-team (Kanban)
aborrero triaged T234830: CloudVPS: m5-master databases for openstack may require re-encoding as High priority.
Mon, Oct 7, 3:36 PM · cloud-services-team (Kanban), Cloud-VPS, DBA
aborrero created T234830: CloudVPS: m5-master databases for openstack may require re-encoding.
Mon, Oct 7, 3:36 PM · cloud-services-team (Kanban), Cloud-VPS, DBA
aborrero added a comment to P9249 (An Untitled Masterwork).
diff --git a/neutron/db/migration/__init__.py b/neutron/db/migration/__init__.py
index c5b1d773b8..ed55ff5f13 100644
--- a/neutron/db/migration/__init__.py
+++ b/neutron/db/migration/__init__.py
@@ -203,7 +203,7 @@ def create_table_if_not_exist_psql(table_name, values):
     op.execute("CREATE OR REPLACE FUNCTION table_exist(TEXT) RETURNS bool as "
                "$$ SELECT exists(select 1 from pg_class where relname=$1);"
                "$$ language sql STRICT;")
-    op.execute("SELECT execute($$CREATE TABLE %(name)s %(columns)s $$) "
+    op.execute("SELECT execute($$CREATE TABLE IF NOT EXISTS %(name)s %(columns)s $$) "
                "WHERE NOT table_exist(%(name)r);" %
                {'name': table_name,
                 'columns': values})
Mon, Oct 7, 3:16 PM
aborrero added a comment to T234683: Build, package bdsync for Buster.

I can do the buster build. Perhaps we should consider uploading this package to Debian, it should be interesting for other people too.

Mon, Oct 7, 10:23 AM · Cloud-Services, ops-codfw, Operations
aborrero added a comment to T228500: Toolforge: evaluate ingress mechanism.

Awesome! I think I get the idea. Perhaps we should continue conversation about this at T234231: Toolforge ingress: decide on how ingress configuration objects will be managed.
BTW I believe the FQDN scheme is <$toolname>.toolforge.org and not <$toolname>.tools.toolforge.org.

Mon, Oct 7, 9:31 AM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes

Fri, Oct 4

aborrero triaged T234617: Toolforge. introduce new domain toolforge.org as Normal priority.
Fri, Oct 4, 1:07 PM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero created T234617: Toolforge. introduce new domain toolforge.org.
Fri, Oct 4, 1:06 PM · Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero updated the task description for T234037: Toolforge ingress: decide on final layout of north-south proxy setup.
Fri, Oct 4, 11:06 AM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T234037: Toolforge ingress: decide on final layout of north-south proxy setup.

I've been playing with option 2, and here my tests:

Fri, Oct 4, 10:57 AM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero awarded T224033: Fix operations/puppet.git "rebase hell" a The World Burns token.
Fri, Oct 4, 9:53 AM · Release-Engineering-Team (Development services), Gerrit, Release-Engineering-Team-TODO, Continuous-Integration-Config, Operations
aborrero triaged T224188: rack/setup/install (3) new osd ceph nodes as High priority.

Raising priority of this ticket, since the ceph project is part of our Q2 goals.

Fri, Oct 4, 8:49 AM · ops-eqiad, Operations, cloud-services-team (Kanban), Cloud-Services
aborrero raised the priority of T228102: rack/setup/install cloudcephmon100[123] from Normal to High.

Raising priority of this ticket, since the ceph project is part of our Q2 goals.

Fri, Oct 4, 8:48 AM · cloud-services-team (Kanban), Operations, Cloud-Services, ops-eqiad

Thu, Oct 3

aborrero closed T233656: Request creation of finding-glams VPS project as Resolved.

All is done!

Thu, Oct 3, 11:19 AM · User-LokalProfil, User-Alicia_Fagerving_WMSE, WMSE-FindingGLAMs-2018 (Campaign), Cloud-VPS (Project-requests)
aborrero closed T234269: Request for temporarily increased quota for dwl Cloud VPS project to rebuild and test deprecated instances as Resolved.

This should be done now:

Thu, Oct 3, 11:10 AM · Cloud-VPS (Quota-requests)
aborrero added a subtask for T229441: CloudVPS: codfw1dev: missing bits: T234518: codfw1dev: glance_rsync_images is failing.
Thu, Oct 3, 10:34 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a parent task for T234518: codfw1dev: glance_rsync_images is failing: T229441: CloudVPS: codfw1dev: missing bits.
Thu, Oct 3, 10:34 AM · cloud-services-team (Kanban)
aborrero triaged T234518: codfw1dev: glance_rsync_images is failing as Low priority.
Thu, Oct 3, 10:34 AM · cloud-services-team (Kanban)
aborrero created T234518: codfw1dev: glance_rsync_images is failing.
Thu, Oct 3, 10:33 AM · cloud-services-team (Kanban)

Wed, Oct 2

aborrero closed T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton as Resolved.

I created a VM and tried both dmz_cidr and routing_source_ip:

Wed, Oct 2, 4:03 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
aborrero closed T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton, a subtask of T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton, as Resolved.
Wed, Oct 2, 4:03 PM · Cloud-VPS, Patch-For-Review, cloud-services-team (Kanban)
aborrero added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

Thanks @JHedden. In this case it helped me review every bit of the setup. I cleaned a lot of stuff and documented a bunch of missing bits about the network setup for codfw1dev in https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron

Wed, Oct 2, 2:58 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
aborrero added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

I put the l3-agent in debug mode and it reports creating the rule correctly, no mention to the wrong rule in the logs:

Wed, Oct 2, 11:50 AM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
aborrero added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

I'm reviewing the neutron setup in codfw1dev following this checklist: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployment_sanity_checklist#General_networking_&_neutron
I noticed something weird (possibly a bug somewhere) when checking the dmz_cidr setting and how that translates to iptables rules.

Wed, Oct 2, 11:08 AM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Tue, Oct 1

aborrero closed T234324: CloudVPS: wmcs-cold-migrate: AttributeError: 'NoneType' object has no attribute 'find' as Invalid.

This may be due to lack of credentials, thus a false positive. I didn't load the env vars before running the script.

Tue, Oct 1, 12:24 PM · cloud-services-team (Kanban)
aborrero created T234324: CloudVPS: wmcs-cold-migrate: AttributeError: 'NoneType' object has no attribute 'find'.
Tue, Oct 1, 12:13 PM · cloud-services-team (Kanban)

Mon, Sep 30

aborrero added a comment to T234037: Toolforge ingress: decide on final layout of north-south proxy setup.

For the record, Bryan mentioned another option: having dynamicproxy understand how to forward to the new k8s cluster.

Mon, Sep 30, 3:59 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero triaged T234231: Toolforge ingress: decide on how ingress configuration objects will be managed as High priority.
Mon, Sep 30, 3:33 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero created T234231: Toolforge ingress: decide on how ingress configuration objects will be managed.
Mon, Sep 30, 3:33 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
aborrero added a comment to T234225: Wikimedia Space fails to start with "iptables: No chain/target/match by that name.".

This is related to T234218: Can't login into Wikimedia Space . I already replied there. Please restart the docker service.

Mon, Sep 30, 2:58 PM · Space
aborrero added a comment to T234218: Can't login into Wikimedia Space.

I see some issues with docker in the discuss-space server:

Mon, Sep 30, 2:53 PM · Cloud-Services, Space (Jul-Sep-2019)
aborrero triaged T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton as High priority.
Mon, Sep 30, 12:47 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
aborrero triaged T228500: Toolforge: evaluate ingress mechanism as High priority.
Mon, Sep 30, 12:30 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes