aborrero (arturo)
Operations Engineer at Wikimedia Cloud Services Team

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 23 2017, 12:19 PM (59 w, 4 d)
Availability
Available
IRC Nick
arturo
LDAP User
Arturo Borrero Gonzalez
MediaWiki User
ABorrero (WMF) [ Global Accounts ]

I'm Arturo Borrero Gonzalez from Spain (Seville). I'm Operations Engineer as part of the Wikimedia Cloud Services Team, a Wikimedia Foundation staff.

You may found me in some FLOSS projects, like Netfilter and Debian.

Recent Activity

Today

aborrero moved T211977: toolforge: webservicemonitor for stretch/sge from Inbox to Doing on the cloud-services-team (Kanban) board.
Fri, Dec 14, 2:13 PM · Cloud-VPS (Ubuntu Trusty Deprecation), cloud-services-team (Kanban)
aborrero updated the task description for T211977: toolforge: webservicemonitor for stretch/sge.
Fri, Dec 14, 2:12 PM · Cloud-VPS (Ubuntu Trusty Deprecation), cloud-services-team (Kanban)
aborrero triaged T211977: toolforge: webservicemonitor for stretch/sge as Normal priority.
Fri, Dec 14, 2:11 PM · Cloud-VPS (Ubuntu Trusty Deprecation), cloud-services-team (Kanban)
aborrero added a parent task for T211684: Toolforge: Port sge.py stats to Prometheus: T207591: tools-services: Migrate to Stretch.
Fri, Dec 14, 1:54 PM · monitoring, cloud-services-team (Kanban), Goal, Operations
aborrero added a subtask for T207591: tools-services: Migrate to Stretch: T211684: Toolforge: Port sge.py stats to Prometheus.
Fri, Dec 14, 1:54 PM · Patch-For-Review, Cloud-VPS (Ubuntu Trusty Deprecation), cloud-services-team (Kanban)
aborrero added a comment to T199003: Develop timeline for Cloud VPS wide deprecation of Trusty.

Needs discussion: the final deadline is approaching: 2018-12-18. How to handle remaining Trusty VMs?

Fri, Dec 14, 1:27 PM · Cloud-VPS (Ubuntu Trusty Deprecation), cloud-services-team (Kanban), Goal
aborrero closed T211451: 'Tool Labs instance distribution' check failing on cloudvirt1003 as Resolved.

I think this is solved. Feel free to reopen if you find any additional issue.

Fri, Dec 14, 12:56 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero moved T37947: Enable IPv6 on CloudVPS from Needs discussion to Important on the cloud-services-team (Kanban) board.
Fri, Dec 14, 10:44 AM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS
aborrero updated subscribers of T37947: Enable IPv6 on CloudVPS.

Persisting here some notes from @chasemp for future reference:

Fri, Dec 14, 10:43 AM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS

Yesterday

aborrero added a comment to T168967: Upload shiny-server .deb to our Stretch apt repository.

Please @mpopov try now.

Thu, Dec 13, 5:32 PM · Patch-For-Review, Product-Analytics, Operations
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.

For the D day:

Thu, Dec 13, 11:59 AM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.

I love diagrams, they help me better understand topology and architectures. Please, confirm the following are right.

Thu, Dec 13, 11:04 AM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations

Wed, Dec 12

aborrero added a comment to T209460: CloudVPS: our ideal future model.

Can someone point me to the current network layout? Vlans, ip space in use, what's used to route/filter traffic, etc.? Knowing the current situation is usually a good first step when designing a to be situation. Does Wikimedia have an overall architecture or architecture principles? That would be good input too.

Wed, Dec 12, 2:32 PM · Operations, cloud-services-team (Kanban), Epic
aborrero closed T211391: Neutron API not properly exposed? as Declined.

I don't think we can allow the GET in the API policy because doing so will also allow PUT and DELETE:

Wed, Dec 12, 2:27 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero closed T211391: Neutron API not properly exposed?, a subtask of T211393: openstack-browser and horizon: Security group and floating IP quota information being pulled from Nova instead of Neutron for eqiad1-r, as Declined.
Wed, Dec 12, 2:27 PM · Horizon, Tools
aborrero added a comment to T211391: Neutron API not properly exposed?.

The original issue has been solved, however:

Wed, Dec 12, 2:17 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero moved T211391: Neutron API not properly exposed? from Inbox to Doing on the cloud-services-team (Kanban) board.
Wed, Dec 12, 2:00 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero claimed T211391: Neutron API not properly exposed?.
Wed, Dec 12, 2:00 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS
aborrero moved T199003: Develop timeline for Cloud VPS wide deprecation of Trusty from Important to Needs discussion on the cloud-services-team (Kanban) board.
Wed, Dec 12, 1:57 PM · Cloud-VPS (Ubuntu Trusty Deprecation), cloud-services-team (Kanban), Goal
aborrero closed T209818: Mount dumps NFS share to instances in the soweego VPS project as Resolved.
root@soweego-1:~# ls -l /public/dumps/
total 16
lrwxrwxrwx 1 root root 59 Dec 12 13:52 incr -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/incr
lrwxrwxrwx 1 root root 88 Dec 12 13:52 pagecounts-all-sites -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pagecounts-all-sites
lrwxrwxrwx 1 root root 82 Dec 12 13:52 pagecounts-raw -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pagecounts-raw
lrwxrwxrwx 1 root root 77 Dec 12 13:52 pageviews -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pageviews
lrwxrwxrwx 1 root root 61 Dec 12 13:52 public -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public

I think you are all set. Feel free to reopen the ticket if you find anything wrong.

Wed, Dec 12, 1:54 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Wikidata
aborrero raised the priority of T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Low to Normal.

My team agreed on following up with eqiad1. The only requirement is we have a clear rollback plan in case something goes wrong.

Wed, Dec 12, 12:19 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero awarded T179463: Create a single application to provision and manage developer (LDAP) accounts a Like token.
Wed, Dec 12, 12:17 PM · LDAP, Operations, Developer-Advocacy, Cloud-Services

Tue, Dec 11

aborrero added a comment to T177855: Difficulty applying profile class parameters in Horizon interface.

Any possibility of getting this fixed? :D It'd be nice to use the more flexible Horizon hiera to configure the new cloud-analytics cluster in T204951 instead of having to merge puppet commits for every change.

Tue, Dec 11, 5:44 PM · Patch-For-Review, cloud-services-team (Kanban), Horizon
aborrero added a comment to T168967: Upload shiny-server .deb to our Stretch apt repository.

I don't know if Guillaume uploaded https://download3.rstudio.org/ubuntu-14.04/x86_64/shiny-server-1.5.9.923-amd64.deb to the internal Debian repository or if it's still only in the Ubuntu one (idk how to check). @aborrero can you or someone else on WMCS team please check? Or is it just a single apt repository? (Again, I'm not ops so I don't know.) From shiny_server::init.pp#L93-L94:

# Assuming shiny-server-1.5.3.838-amd64.deb exists in the WMF apt repo...
require_package('shiny-server')
Tue, Dec 11, 5:37 PM · Patch-For-Review, Product-Analytics, Operations
aborrero moved T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Graveyard to Doing on the cloud-services-team (Kanban) board.
Tue, Dec 11, 4:53 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero moved T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Needs discussion to Graveyard on the cloud-services-team (Kanban) board.
Tue, Dec 11, 4:52 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero moved T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Doing to Needs discussion on the cloud-services-team (Kanban) board.
Tue, Dec 11, 4:26 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero moved T37947: Enable IPv6 on CloudVPS from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
Tue, Dec 11, 4:23 PM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS
aborrero claimed T209818: Mount dumps NFS share to instances in the soweego VPS project.
Tue, Dec 11, 2:24 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Wikidata
aborrero moved T209818: Mount dumps NFS share to instances in the soweego VPS project from Inbox to Doing on the cloud-services-team (Kanban) board.
Tue, Dec 11, 1:54 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Wikidata
aborrero triaged T209818: Mount dumps NFS share to instances in the soweego VPS project as Normal priority.

Looking into this.

Tue, Dec 11, 1:54 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Wikidata
aborrero added a comment to T211644: PROBLEM - ensure kvm processes are running on cloudvirt1023 is CRITICAL: PROCS CRITICAL: 96 processes with regex args qemu-system-x86_64.

BTW cloudvirt1023 is full while others are almost empty. Nova scheduling issues? I would expect most hypervisors to be filled more or less at the same time?
Could this be related to new HW being added to the pool?

Tue, Dec 11, 1:49 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a comment to T211644: PROBLEM - ensure kvm processes are running on cloudvirt1023 is CRITICAL: PROCS CRITICAL: 96 processes with regex args qemu-system-x86_64.

I just checked the server. Is actually running 95 VMs.

Tue, Dec 11, 1:23 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero awarded T211643: Update and Improve: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin a Love token.
Tue, Dec 11, 11:49 AM · User-srodlund, Toolforge, Documentation
aborrero added a comment to T37947: Enable IPv6 on CloudVPS.

Before we can move forward with this, there are several things to sort out:

Tue, Dec 11, 11:01 AM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS
aborrero added a subtask for T37947: Enable IPv6 on CloudVPS: T211575: Enable IPv6 on tools.wmflabs.org.
Tue, Dec 11, 10:57 AM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS
aborrero added a parent task for T211575: Enable IPv6 on tools.wmflabs.org: T37947: Enable IPv6 on CloudVPS.
Tue, Dec 11, 10:57 AM · cloud-services-team (Kanban), Toolforge, IPv6
aborrero removed a parent task for T37947: Enable IPv6 on CloudVPS: T211575: Enable IPv6 on tools.wmflabs.org.
Tue, Dec 11, 10:57 AM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS
aborrero removed a subtask for T211575: Enable IPv6 on tools.wmflabs.org: T37947: Enable IPv6 on CloudVPS.
Tue, Dec 11, 10:57 AM · cloud-services-team (Kanban), Toolforge, IPv6
aborrero added a parent task for T37947: Enable IPv6 on CloudVPS: T209460: CloudVPS: our ideal future model.
Tue, Dec 11, 10:54 AM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS
aborrero added a subtask for T209460: CloudVPS: our ideal future model: T37947: Enable IPv6 on CloudVPS.
Tue, Dec 11, 10:54 AM · Operations, cloud-services-team (Kanban), Epic
aborrero added a project to T211575: Enable IPv6 on tools.wmflabs.org: cloud-services-team (Kanban).
Tue, Dec 11, 10:53 AM · cloud-services-team (Kanban), Toolforge, IPv6
aborrero renamed T37947: Enable IPv6 on CloudVPS from Enable ipv6 on labs to Enable IPv6 on CloudVPS.
Tue, Dec 11, 10:53 AM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS
aborrero added a comment to T37947: Enable IPv6 on CloudVPS.

Note for myself: https://docs.openstack.org/mitaka/networking-guide/config-ipv6.html

Tue, Dec 11, 10:52 AM · cloud-services-team (Kanban), Operations, IPv6, Cloud-VPS

Mon, Dec 10

aborrero closed T207377: Reboot WMCS servers for L1TF as Resolved.

Thanks @Bstorm and @GTirloni you both did most of the heavy work :-)

Mon, Dec 10, 10:22 AM · Patch-For-Review, cloud-services-team (Kanban), Operations
aborrero claimed T211451: 'Tool Labs instance distribution' check failing on cloudvirt1003.

I think adding a region selector could help. I will take a look.

Mon, Dec 10, 10:20 AM · Patch-For-Review, cloud-services-team (Kanban)

Wed, Dec 5

aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.

@ayounsi and I did in real time both:

Wed, Dec 5, 5:55 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.

Trying now with only adding a new subnet object:

Wed, Dec 5, 5:00 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.

I will try reusing the same network object to try to make this even cleaner.

Wed, Dec 5, 1:41 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
root@labtestcontrol2003:~# neutron router-gateway-set --fixed-ip subnet_id=cloud-instances-transport1-b-codfw,ip_address=208.80.153.190  cloudinstances2b-gw wan-transport-codfw
Set gateway for router cloudinstances2b-gw
root@labtestcontrol2003:~# neutron router-list
+--------------------------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+------+
| id                                   | name                | external_gateway_info                                                                                                                                                                      | distributed | ha   |
+--------------------------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+------+
| 5712e22e-134a-40d3-a75a-1c9b441717ad | cloudinstances2b-gw | {"network_id": "07d9efe1-bed6-4b44-85af-4a37d8e3c766", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "eb4db443-2184-4456-b414-6e53fa878bee", "ip_address": "208.80.153.190"}]} | False       | True |
+--------------------------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+------+
root@labtestcontrol2003:~# neutron router-show cloudinstances2b-gw
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                                      |
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | True                                                                                                                                                                                       |
| availability_zone_hints |                                                                                                                                                                                            |
| availability_zones      | nova                                                                                                                                                                                       |
| description             |                                                                                                                                                                                            |
| distributed             | False                                                                                                                                                                                      |
| external_gateway_info   | {"network_id": "07d9efe1-bed6-4b44-85af-4a37d8e3c766", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "eb4db443-2184-4456-b414-6e53fa878bee", "ip_address": "208.80.153.190"}]} |
| ha                      | True                                                                                                                                                                                       |
| id                      | 5712e22e-134a-40d3-a75a-1c9b441717ad                                                                                                                                                       |
| name                    | cloudinstances2b-gw                                                                                                                                                                        |
| routes                  |                                                                                                                                                                                            |
| status                  | ACTIVE                                                                                                                                                                                     |
| tenant_id               | admin                                                                                                                                                                                      |
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Wed, Dec 5, 1:17 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
root@labtestcontrol2003:~# neutron subnet-create --gateway 208.80.153.185 --name cloud-instances-transport1-b-codfw --ip-version 4 --disable-dhcp wan-transport-codfw 208.80.153.184/29
Created a new subnet:
+-------------------+------------------------------------------------------+
| Field             | Value                                                |
+-------------------+------------------------------------------------------+
| allocation_pools  | {"start": "208.80.153.186", "end": "208.80.153.190"} |
| cidr              | 208.80.153.184/29                                    |
| created_at        | 2018-12-05T12:57:55                                  |
| description       |                                                      |
| dns_nameservers   |                                                      |
| enable_dhcp       | False                                                |
| gateway_ip        | 208.80.153.185                                       |
| host_routes       |                                                      |
| id                | eb4db443-2184-4456-b414-6e53fa878bee                 |
| ip_version        | 4                                                    |
| ipv6_address_mode |                                                      |
| ipv6_ra_mode      |                                                      |
| name              | cloud-instances-transport1-b-codfw                   |
| network_id        | 07d9efe1-bed6-4b44-85af-4a37d8e3c766                 |
| subnetpool_id     |                                                      |
| tenant_id         | admin                                                |
| updated_at        | 2018-12-05T12:57:55                                  |
+-------------------+------------------------------------------------------+
Wed, Dec 5, 12:58 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
root@labtestcontrol2003:~# neutron net-create 'wan-transport-codfw' --router:external=true --provider:network_type=flat --provider:physical_network=br-transport --shared
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-12-05T12:54:25                  |
| description               |                                      |
| id                        | 07d9efe1-bed6-4b44-85af-4a37d8e3c766 |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| is_default                | False                                |
| mtu                       | 1500                                 |
| name                      | wan-transport-codfw                  |
| port_security_enabled     | True                                 |
| provider:network_type     | flat                                 |
| provider:physical_network | br-transport                         |
| provider:segmentation_id  |                                      |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| tenant_id                 | admin                                |
| updated_at                | 2018-12-05T12:54:25                  |
+---------------------------+--------------------------------------+
Wed, Dec 5, 12:55 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero claimed T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
Wed, Dec 5, 12:34 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero placed T204840: wikitech-static: not synced up for grabs.
Wed, Dec 5, 12:05 PM · cloud-services-team (Kanban), Patch-For-Review, wikitech.wikimedia.org
aborrero closed T211055: Fix the init script for gridmaster in sonofgridengine, a subtask of T200557: Create a stretch and Son of Grid Engine grid in toolsbeta, as Resolved.
Wed, Dec 5, 11:40 AM · Cloud-VPS (Ubuntu Trusty Deprecation), Patch-For-Review, Toolforge, Epic, cloud-services-team (Kanban)
aborrero closed T211055: Fix the init script for gridmaster in sonofgridengine as Resolved.
Wed, Dec 5, 11:40 AM · Patch-For-Review, Cloud-VPS (Ubuntu Trusty Deprecation), Toolforge, cloud-services-team (Kanban)
aborrero added a comment to T211055: Fix the init script for gridmaster in sonofgridengine.

From the systemd point of view, the daemon seems to be working just fine. I think we can close the task and reopen if we see anything else.

Wed, Dec 5, 11:39 AM · Patch-For-Review, Cloud-VPS (Ubuntu Trusty Deprecation), Toolforge, cloud-services-team (Kanban)
aborrero placed T206013: importDump.php --uploads crashes on wikitech-static up for grabs.

I won't be working on this task in the short/mid term, cleaning assignation.

Wed, Dec 5, 11:34 AM · MW-1.31-release-notes, MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), MW-1.31-release, MediaWiki-Export-or-Import, Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org
aborrero added a comment to T205969: labstore1007: high load avg issue.

This week we got several pages related to this. I just downtimed the server for 1 week.

Wed, Dec 5, 10:18 AM · cloud-services-team (Kanban)

Tue, Dec 4

aborrero moved T202889: cloudvps: dedicated openstack database from Needs discussion to Graveyard on the cloud-services-team (Kanban) board.
Tue, Dec 4, 4:56 PM · cloud-services-team (Kanban), Patch-For-Review, DBA
aborrero moved T211055: Fix the init script for gridmaster in sonofgridengine from Inbox to Doing on the cloud-services-team (Kanban) board.
Tue, Dec 4, 10:37 AM · Patch-For-Review, Cloud-VPS (Ubuntu Trusty Deprecation), Toolforge, cloud-services-team (Kanban)
aborrero claimed T211055: Fix the init script for gridmaster in sonofgridengine.

I can handle this if you want :-)

Tue, Dec 4, 10:37 AM · Patch-For-Review, Cloud-VPS (Ubuntu Trusty Deprecation), Toolforge, cloud-services-team (Kanban)

Mon, Dec 3

aborrero moved T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Inbox to Doing on the cloud-services-team (Kanban) board.
Mon, Dec 3, 1:45 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero lowered the priority of T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Normal to Low.
Mon, Dec 3, 1:45 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero edited projects for T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs, added: cloud-services-team (Kanban); removed Cloud-Services.
Mon, Dec 3, 1:44 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero added a parent task for T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs: T209460: CloudVPS: our ideal future model.
Mon, Dec 3, 1:44 PM · cloud-services-team (Kanban), Patch-For-Review, netops, Operations
aborrero added a subtask for T209460: CloudVPS: our ideal future model: T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
Mon, Dec 3, 1:44 PM · Operations, cloud-services-team (Kanban), Epic
aborrero updated the task description for T210995: cloudvps: rabbitmq metrics.
Mon, Dec 3, 10:36 AM · cloud-services-team (Kanban)
aborrero triaged T210995: cloudvps: rabbitmq metrics as Normal priority.
Mon, Dec 3, 10:32 AM · cloud-services-team (Kanban)
aborrero created T210995: cloudvps: rabbitmq metrics.
Mon, Dec 3, 10:32 AM · cloud-services-team (Kanban)
aborrero lowered the priority of T205524: cloudvps: neutron: agents failed to communicate with server from Normal to Low.

I just created a grafana dashboard: https://grafana.wikimedia.org/dashboard/db/cloudvps-rabbitmq?orgId=1&from=1514764800000&to=1543832014906

Mon, Dec 3, 10:27 AM · cloud-services-team (Kanban), Cloud-Services

Fri, Nov 30

aborrero closed T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses as Resolved.

Using the designate CLI I've created the following records:

Fri, Nov 30, 2:11 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero closed T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
Fri, Nov 30, 2:11 PM · Patch-For-Review, Epic, Cloud-Services
aborrero closed T210596: cloudvps: puppet code & hiera keys per datacenter as Declined.

For example, I would like to have all profile::openstack::eqiad1::xxxx hiera keys under hieradata/eqiad/

It's not obvious to be that that's better, although maybe I don't understand the context. That hiera path already specifies the deployment ('eqiad1') and we're not going to duplicate /that/ in another datacenter. And, as you see, both data centers sometimes need to know about the eqiad1 deployment.

Fri, Nov 30, 12:45 PM · cloud-services-team (Kanban)
aborrero closed T210754: cloudvps: convert profile::openstack::eqiad1::neutron::dmz_cidr to a list/array as Resolved.
Fri, Nov 30, 10:48 AM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a comment to T203529: Set up a packagist mirror for Wikimedia.

In fact, I just created a separate security group, called HTTP/HTTPS, instead of adding rules to the default one.

Fri, Nov 30, 9:40 AM · Composer, Wikimedia-General-or-Unknown
aborrero added a comment to T203529: Set up a packagist mirror for Wikimedia.

I added a rule allowing HTTP (80/tcp) traffic to the instance from anywhere to the security group.

Fri, Nov 30, 9:34 AM · Composer, Wikimedia-General-or-Unknown

Thu, Nov 29

aborrero triaged T210754: cloudvps: convert profile::openstack::eqiad1::neutron::dmz_cidr to a list/array as Normal priority.
Thu, Nov 29, 5:05 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero created T210754: cloudvps: convert profile::openstack::eqiad1::neutron::dmz_cidr to a list/array.
Thu, Nov 29, 5:04 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero moved T194855: Degraded RAID on cloudvirt1020 from Inbox to Blocked on the cloud-services-team (Kanban) board.
Thu, Nov 29, 1:17 PM · Patch-For-Review, cloud-services-team (Kanban), ops-eqiad, Operations
aborrero added a project to T194855: Degraded RAID on cloudvirt1020: cloud-services-team (Kanban).
Thu, Nov 29, 1:17 PM · Patch-For-Review, cloud-services-team (Kanban), ops-eqiad, Operations
aborrero moved T208754: rename cloudvirt1019 and cloudvirt1020 to cloudvirtdb1001 and cloudvirtdb1002 from Important to Blocked on the cloud-services-team (Kanban) board.
Thu, Nov 29, 1:17 PM · cloud-services-team (Kanban)
aborrero moved T196507: Degraded RAID on cloudvirt1019 from Inbox to Blocked on the cloud-services-team (Kanban) board.
Thu, Nov 29, 1:17 PM · Patch-For-Review, cloud-services-team (Kanban), ops-eqiad, Operations
aborrero added a project to T196507: Degraded RAID on cloudvirt1019: cloud-services-team (Kanban).
Thu, Nov 29, 1:16 PM · Patch-For-Review, cloud-services-team (Kanban), ops-eqiad, Operations
aborrero added a comment to T196507: Degraded RAID on cloudvirt1019.

Question: what is the warranty status of this server? would it make sense to get a more complete replacement by HP? (not just some spare pieces like disk and raid controllers)

Thu, Nov 29, 1:15 PM · Patch-For-Review, cloud-services-team (Kanban), ops-eqiad, Operations
aborrero closed T209517: Upgrade/reboot labsdb* servers as Resolved.

Thanks @Banyek, @Bstorm and @Marostegui

Thu, Nov 29, 12:36 PM · User-Banyek, Patch-For-Review, Data-Services, cloud-services-team (Kanban), DBA
aborrero closed T209517: Upgrade/reboot labsdb* servers, a subtask of T207377: Reboot WMCS servers for L1TF, as Resolved.
Thu, Nov 29, 12:35 PM · Patch-For-Review, cloud-services-team (Kanban), Operations
aborrero added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.

mmmm creating a loose record without an associated instance in the eqiad.wmflabs domain might be flushed by the dnsleaks.py script.

Thu, Nov 29, 12:34 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.

I realize now that the records have been deleted from the PDNS database after a AXFR (from designate?) I might need to add the records in the designate DB in m5-master instead.

Thu, Nov 29, 12:25 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.

Question is, what will happen if somebody creates a VM with these names?

Thu, Nov 29, 12:17 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.

I was thinking on creating these records in the PDNS database living in cloudservices1003.wikimedia.org:

Thu, Nov 29, 11:34 AM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
aborrero updated the task description for T210715: cloudvps: PDNS 3.x vs 4.x.
Thu, Nov 29, 10:57 AM · cloud-services-team (Kanban)
aborrero triaged T210715: cloudvps: PDNS 3.x vs 4.x as Normal priority.
Thu, Nov 29, 10:54 AM · cloud-services-team (Kanban)
aborrero created T210715: cloudvps: PDNS 3.x vs 4.x.
Thu, Nov 29, 10:54 AM · cloud-services-team (Kanban)
aborrero moved T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses from Important to Doing on the cloud-services-team (Kanban) board.
Thu, Nov 29, 10:36 AM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services

Wed, Nov 28

aborrero closed T210595: cloudvps: keystone extra services as Resolved.
Wed, Nov 28, 5:28 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero closed T210595: cloudvps: keystone extra services, a subtask of T201504: cloudvps: main/eqiad1 keystone merge, as Resolved.
Wed, Nov 28, 5:28 PM · Patch-For-Review, cloud-services-team, Epic, Cloud-Services
aborrero moved T210595: cloudvps: keystone extra services from Inbox to Doing on the cloud-services-team (Kanban) board.
Wed, Nov 28, 12:13 PM · Patch-For-Review, cloud-services-team (Kanban)
aborrero added a comment to T210596: cloudvps: puppet code & hiera keys per datacenter.

AFAIK they are not using the wrong datacenter-

# Firewall rules for the misc db host used by wmcs

Those are firewall rules that should be common to all datacenters, no matter if wmcs is on eqiad only. If m5 database is, for some reason, failed over to codfw, it will have to conect to codfw, and thus have the appropiate firewall configured.

Wed, Nov 28, 11:53 AM · cloud-services-team (Kanban)
aborrero claimed T210596: cloudvps: puppet code & hiera keys per datacenter.
Wed, Nov 28, 11:44 AM · cloud-services-team (Kanban)