I'm Arturo Borrero Gonzalez from Spain (Seville). I'm Operations Engineer as part of the Wikimedia Cloud Services Team, a Wikimedia Foundation staff.
You may found me in some FLOSS projects, like Netfilter and Debian.
User Details
User Details
- User Since
- Oct 23 2017, 12:19 PM (59 w, 4 d)
- Availability
- Available
- IRC Nick
- arturo
- LDAP User
- Arturo Borrero Gonzalez
- MediaWiki User
- ABorrero (WMF) [ Global Accounts ]
Today
Today
aborrero moved T211977: toolforge: webservicemonitor for stretch/sge from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero updated the task description for T211977: toolforge: webservicemonitor for stretch/sge.
aborrero added a parent task for T211684: Toolforge: Port sge.py stats to Prometheus: T207591: tools-services: Migrate to Stretch.
aborrero added a comment to T199003: Develop timeline for Cloud VPS wide deprecation of Trusty.
Needs discussion: the final deadline is approaching: 2018-12-18. How to handle remaining Trusty VMs?
aborrero closed T211451: 'Tool Labs instance distribution' check failing on cloudvirt1003 as Resolved.
I think this is solved. Feel free to reopen if you find any additional issue.
aborrero moved T37947: Enable IPv6 on CloudVPS from Needs discussion to Important on the cloud-services-team (Kanban) board.
aborrero updated subscribers of T37947: Enable IPv6 on CloudVPS.
Persisting here some notes from @chasemp for future reference:
Yesterday
Yesterday
aborrero added a comment to T168967: Upload shiny-server .deb to our Stretch apt repository.
Please @mpopov try now.
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
For the D day:
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
I love diagrams, they help me better understand topology and architectures. Please, confirm the following are right.
Wed, Dec 12
Wed, Dec 12
aborrero added a comment to T209460: CloudVPS: our ideal future model.
I don't think we can allow the GET in the API policy because doing so will also allow PUT and DELETE:
aborrero added a comment to T211391: Neutron API not properly exposed?.
The original issue has been solved, however:
aborrero moved T211391: Neutron API not properly exposed? from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero moved T199003: Develop timeline for Cloud VPS wide deprecation of Trusty from Important to Needs discussion on the cloud-services-team (Kanban) board.
root@soweego-1:~# ls -l /public/dumps/ total 16 lrwxrwxrwx 1 root root 59 Dec 12 13:52 incr -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/incr lrwxrwxrwx 1 root root 88 Dec 12 13:52 pagecounts-all-sites -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pagecounts-all-sites lrwxrwxrwx 1 root root 82 Dec 12 13:52 pagecounts-raw -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pagecounts-raw lrwxrwxrwx 1 root root 77 Dec 12 13:52 pageviews -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/other/pageviews lrwxrwxrwx 1 root root 61 Dec 12 13:52 public -> /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public
I think you are all set. Feel free to reopen the ticket if you find anything wrong.
aborrero raised the priority of T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Low to Normal.
My team agreed on following up with eqiad1. The only requirement is we have a clear rollback plan in case something goes wrong.
Tue, Dec 11
Tue, Dec 11
aborrero added a comment to T177855: Difficulty applying profile class parameters in Horizon interface.
aborrero added a comment to T168967: Upload shiny-server .deb to our Stretch apt repository.
aborrero moved T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Graveyard to Doing on the cloud-services-team (Kanban) board.
aborrero moved T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Needs discussion to Graveyard on the cloud-services-team (Kanban) board.
aborrero moved T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Doing to Needs discussion on the cloud-services-team (Kanban) board.
aborrero moved T37947: Enable IPv6 on CloudVPS from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
aborrero moved T209818: Mount dumps NFS share to instances in the soweego VPS project from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero triaged T209818: Mount dumps NFS share to instances in the soweego VPS project as Normal priority.
Looking into this.
aborrero added a comment to T211644: PROBLEM - ensure kvm processes are running on cloudvirt1023 is CRITICAL: PROCS CRITICAL: 96 processes with regex args qemu-system-x86_64.
BTW cloudvirt1023 is full while others are almost empty. Nova scheduling issues? I would expect most hypervisors to be filled more or less at the same time?
Could this be related to new HW being added to the pool?
aborrero added a comment to T211644: PROBLEM - ensure kvm processes are running on cloudvirt1023 is CRITICAL: PROCS CRITICAL: 96 processes with regex args qemu-system-x86_64.
I just checked the server. Is actually running 95 VMs.
aborrero added a comment to T37947: Enable IPv6 on CloudVPS.
Before we can move forward with this, there are several things to sort out:
aborrero added a subtask for T37947: Enable IPv6 on CloudVPS: T211575: Enable IPv6 on tools.wmflabs.org.
aborrero added a parent task for T211575: Enable IPv6 on tools.wmflabs.org: T37947: Enable IPv6 on CloudVPS.
aborrero removed a parent task for T37947: Enable IPv6 on CloudVPS: T211575: Enable IPv6 on tools.wmflabs.org.
aborrero removed a subtask for T211575: Enable IPv6 on tools.wmflabs.org: T37947: Enable IPv6 on CloudVPS.
aborrero added a parent task for T37947: Enable IPv6 on CloudVPS: T209460: CloudVPS: our ideal future model.
aborrero added a subtask for T209460: CloudVPS: our ideal future model: T37947: Enable IPv6 on CloudVPS.
aborrero added a project to T211575: Enable IPv6 on tools.wmflabs.org: cloud-services-team (Kanban).
aborrero renamed T37947: Enable IPv6 on CloudVPS from Enable ipv6 on labs to Enable IPv6 on CloudVPS.
aborrero added a comment to T37947: Enable IPv6 on CloudVPS.
Note for myself: https://docs.openstack.org/mitaka/networking-guide/config-ipv6.html
Mon, Dec 10
Mon, Dec 10
I think adding a region selector could help. I will take a look.
Wed, Dec 5
Wed, Dec 5
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
@ayounsi and I did in real time both:
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
Trying now with only adding a new subnet object:
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
I will try reusing the same network object to try to make this even cleaner.
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
root@labtestcontrol2003:~# neutron router-gateway-set --fixed-ip subnet_id=cloud-instances-transport1-b-codfw,ip_address=208.80.153.190 cloudinstances2b-gw wan-transport-codfw Set gateway for router cloudinstances2b-gw root@labtestcontrol2003:~# neutron router-list +--------------------------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+------+ | id | name | external_gateway_info | distributed | ha | +--------------------------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+------+ | 5712e22e-134a-40d3-a75a-1c9b441717ad | cloudinstances2b-gw | {"network_id": "07d9efe1-bed6-4b44-85af-4a37d8e3c766", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "eb4db443-2184-4456-b414-6e53fa878bee", "ip_address": "208.80.153.190"}]} | False | True | +--------------------------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+------+ root@labtestcontrol2003:~# neutron router-show cloudinstances2b-gw +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | nova | | description | | | distributed | False | | external_gateway_info | {"network_id": "07d9efe1-bed6-4b44-85af-4a37d8e3c766", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "eb4db443-2184-4456-b414-6e53fa878bee", "ip_address": "208.80.153.190"}]} | | ha | True | | id | 5712e22e-134a-40d3-a75a-1c9b441717ad | | name | cloudinstances2b-gw | | routes | | | status | ACTIVE | | tenant_id | admin | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
root@labtestcontrol2003:~# neutron subnet-create --gateway 208.80.153.185 --name cloud-instances-transport1-b-codfw --ip-version 4 --disable-dhcp wan-transport-codfw 208.80.153.184/29 Created a new subnet: +-------------------+------------------------------------------------------+ | Field | Value | +-------------------+------------------------------------------------------+ | allocation_pools | {"start": "208.80.153.186", "end": "208.80.153.190"} | | cidr | 208.80.153.184/29 | | created_at | 2018-12-05T12:57:55 | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 208.80.153.185 | | host_routes | | | id | eb4db443-2184-4456-b414-6e53fa878bee | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | cloud-instances-transport1-b-codfw | | network_id | 07d9efe1-bed6-4b44-85af-4a37d8e3c766 | | subnetpool_id | | | tenant_id | admin | | updated_at | 2018-12-05T12:57:55 | +-------------------+------------------------------------------------------+
aborrero added a comment to T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.
root@labtestcontrol2003:~# neutron net-create 'wan-transport-codfw' --router:external=true --provider:network_type=flat --provider:physical_network=br-transport --shared Created a new network: +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | | | created_at | 2018-12-05T12:54:25 | | description | | | id | 07d9efe1-bed6-4b44-85af-4a37d8e3c766 | | ipv4_address_scope | | | ipv6_address_scope | | | is_default | False | | mtu | 1500 | | name | wan-transport-codfw | | port_security_enabled | True | | provider:network_type | flat | | provider:physical_network | br-transport | | provider:segmentation_id | | | router:external | True | | shared | True | | status | ACTIVE | | subnets | | | tags | | | tenant_id | admin | | updated_at | 2018-12-05T12:54:25 | +---------------------------+--------------------------------------+
aborrero placed T204840: wikitech-static: not synced up for grabs.
aborrero closed T211055: Fix the init script for gridmaster in sonofgridengine, a subtask of T200557: Create a stretch and Son of Grid Engine grid in toolsbeta, as Resolved.
aborrero added a comment to T211055: Fix the init script for gridmaster in sonofgridengine.
From the systemd point of view, the daemon seems to be working just fine. I think we can close the task and reopen if we see anything else.
aborrero placed T206013: importDump.php --uploads crashes on wikitech-static up for grabs.
I won't be working on this task in the short/mid term, cleaning assignation.
aborrero added a comment to T205969: labstore1007: high load avg issue.
This week we got several pages related to this. I just downtimed the server for 1 week.
Tue, Dec 4
Tue, Dec 4
aborrero moved T202889: cloudvps: dedicated openstack database from Needs discussion to Graveyard on the cloud-services-team (Kanban) board.
aborrero moved T211055: Fix the init script for gridmaster in sonofgridengine from Inbox to Doing on the cloud-services-team (Kanban) board.
I can handle this if you want :-)
Mon, Dec 3
Mon, Dec 3
aborrero moved T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero lowered the priority of T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs from Normal to Low.
aborrero edited projects for T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs, added: cloud-services-team (Kanban); removed Cloud-Services.
aborrero updated the task description for T210995: cloudvps: rabbitmq metrics.
aborrero lowered the priority of T205524: cloudvps: neutron: agents failed to communicate with server from Normal to Low.
I just created a grafana dashboard: https://grafana.wikimedia.org/dashboard/db/cloudvps-rabbitmq?orgId=1&from=1514764800000&to=1543832014906
Fri, Nov 30
Fri, Nov 30
Using the designate CLI I've created the following records:
aborrero closed T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses, a subtask of T167293: Nova-network to Neutron migration, as Resolved.
aborrero added a comment to T203529: Set up a packagist mirror for Wikimedia.
In fact, I just created a separate security group, called HTTP/HTTPS, instead of adding rules to the default one.
aborrero added a comment to T203529: Set up a packagist mirror for Wikimedia.
I added a rule allowing HTTP (80/tcp) traffic to the instance from anywhere to the security group.
Thu, Nov 29
Thu, Nov 29
aborrero triaged T210754: cloudvps: convert profile::openstack::eqiad1::neutron::dmz_cidr to a list/array as Normal priority.
aborrero moved T194855: Degraded RAID on cloudvirt1020 from Inbox to Blocked on the cloud-services-team (Kanban) board.
aborrero moved T196507: Degraded RAID on cloudvirt1019 from Inbox to Blocked on the cloud-services-team (Kanban) board.
aborrero added a comment to T196507: Degraded RAID on cloudvirt1019.
Question: what is the warranty status of this server? would it make sense to get a more complete replacement by HP? (not just some spare pieces like disk and raid controllers)
Thanks @Banyek, @Bstorm and @Marostegui
aborrero closed T209517: Upgrade/reboot labsdb* servers, a subtask of T207377: Reboot WMCS servers for L1TF, as Resolved.
aborrero added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.
mmmm creating a loose record without an associated instance in the eqiad.wmflabs domain might be flushed by the dnsleaks.py script.
aborrero added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.
I realize now that the records have been deleted from the PDNS database after a AXFR (from designate?) I might need to add the records in the designate DB in m5-master instead.
aborrero added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.
aborrero added a comment to T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses.
I was thinking on creating these records in the PDNS database living in cloudservices1003.wikimedia.org:
aborrero updated the task description for T210715: cloudvps: PDNS 3.x vs 4.x.
aborrero moved T202886: cloudvps: eqiad1: create DNS PTR records for cloud addresses from Important to Doing on the cloud-services-team (Kanban) board.
Wed, Nov 28
Wed, Nov 28
aborrero closed T210595: cloudvps: keystone extra services, a subtask of T201504: cloudvps: main/eqiad1 keystone merge, as Resolved.
aborrero moved T210595: cloudvps: keystone extra services from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added a comment to T210596: cloudvps: puppet code & hiera keys per datacenter.