I'm Arturo Borrero Gonzalez from Spain (Seville). I'm Operations Engineer as part of the Wikimedia Cloud Services Team, a Wikimedia Foundation staff.
You may found me in some FLOSS projects, like Netfilter and Debian.
User Details
User Details
- User Since
- Oct 23 2017, 12:19 PM (69 w, 6 h)
- Availability
- Available
- IRC Nick
- arturo
- LDAP User
- Arturo Borrero Gonzalez
- MediaWiki User
- ABorrero (WMF) [ Global Accounts ]
Today
Today
aborrero changed the status of T216373: CloudVPS: run maintain-dbusers inside Toolforge from Open to Stalled.
Since we aren't doing this right now I'm marking the task as such to avoid confusion.
aborrero changed the status of T216373: CloudVPS: run maintain-dbusers inside Toolforge, a subtask of T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020, from Open to Stalled.
aborrero added a comment to T216373: CloudVPS: run maintain-dbusers inside Toolforge.
aborrero added a comment to T216353: toolsdb: firewalling changes for new setup (temporal mysql replication).
It works!
aborrero added a comment to T216353: toolsdb: firewalling changes for new setup (temporal mysql replication).
Verifying that CR is indeed blocking the connections:
Yesterday
Yesterday
aborrero updated the task description for T216373: CloudVPS: run maintain-dbusers inside Toolforge.
aborrero closed T216353: toolsdb: firewalling changes for new setup (temporal mysql replication) as Resolved.
It works!
aborrero updated the task description for T216353: toolsdb: firewalling changes for new setup (temporal mysql replication).
aborrero updated the task description for T216353: toolsdb: firewalling changes for new setup (temporal mysql replication).
aborrero updated the task description for T216353: toolsdb: firewalling changes for new setup (temporal mysql replication).
aborrero added a project to T216353: toolsdb: firewalling changes for new setup (temporal mysql replication): DBA.
aborrero moved T216353: toolsdb: firewalling changes for new setup (temporal mysql replication) from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero triaged T216353: toolsdb: firewalling changes for new setup (temporal mysql replication) as Unbreak Now! priority.
Sat, Feb 16
Sat, Feb 16
aborrero added a comment to T216208: ToolsDB overload and cleanup.
aborrero moved T216324: cloudvirt1009: evaluate upgrading to 10G from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added a comment to T194855: Degraded RAID on cloudvirt1020.
Checking cloudvirt1020:
aborrero added a comment to T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020.
Fri, Feb 15
Fri, Feb 15
To make it clear that we are now putting this HW into service, I'm closing this task. Please re-open if you have any objection.
aborrero closed T196507: Degraded RAID on cloudvirt1019, a subtask of T216208: ToolsDB overload and cleanup, as Resolved.
aborrero closed T196507: Degraded RAID on cloudvirt1019, a subtask of T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020, as Resolved.
aborrero updated the task description for T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009.
aborrero updated the task description for T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009.
aborrero added a comment to T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009.
For the record, before any operation in the RAID:
aborrero updated the task description for T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009.
aborrero added a comment to T216190: Rebuild labvirt1012 as cloudvirt1012.
We should rebuild the RAID and leave some spare disks before adding actual workload to this server.
aborrero added a comment to T210735: decommission labvirt101[01].eqiad.wmnet (Dec 2018 lease return).
Could you please refresh netbox status for these 2 servers:
- https://netbox.wikimedia.org/dcim/devices/1356/ (labvirt1010)
- https://netbox.wikimedia.org/dcim/devices/1355/ (labvirt1011)
aborrero renamed T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009 from CloudVPS: drain and rebuild labvirt1009 as cloudvirt10 to CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009.
aborrero moved T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009 from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero renamed T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009 from CloudVPS: Drain and rebuild labvirt1009 as cloudvirt10 to CloudVPS: drain and rebuild labvirt1009 as cloudvirt10.
aborrero renamed T216190: Rebuild labvirt1012 as cloudvirt1012 from Rebuild labvirt1012 and cloudvirt1012 to Rebuild labvirt1012 as cloudvirt1012.
aborrero moved T216190: Rebuild labvirt1012 as cloudvirt1012 from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added a project to T216190: Rebuild labvirt1012 as cloudvirt1012: cloud-services-team (Kanban).
aborrero closed T208754: rename cloudvirt1019 and cloudvirt1020 to cloudvirtdb1001 and cloudvirtdb1002 as Declined.
Today I had a chat with @faidon and the following concerns were raised about the rename:
Thu, Feb 14
Thu, Feb 14
aborrero added a comment to T216076: PHP does not find mysqli on tools-sgebastion-07.
Package: php-common Source: php-defaults (69) Version: 2:69 [...]
aborrero updated the task description for T215812: reprepro: automate incoming processing.
aborrero reopened T215154: Toolforge: sgebastion: systemd resource control not working as "Open".
Reopening because we moved from tools-sgebastion06 to tools-sgebastion07 and the systemd version isn't right, so we don't have resource control:
aborrero updated the task description for T216041: Order spare cloudvirt SSDs for eqiad.
aborrero updated the task description for T90364: Test Ceph for instance storage.
aborrero triaged T216133: Increase visibility of auto-generated tasks for RAID errors as High priority.
aborrero moved T90364: Test Ceph for instance storage from Inbox to Important on the cloud-services-team (Kanban) board.
aborrero moved T90364: Test Ceph for instance storage from To Triage to Follow-up/Actionables on the Wikimedia-Incident board.
aborrero moved T216041: Order spare cloudvirt SSDs for eqiad from To Triage to Follow-up/Actionables on the Wikimedia-Incident board.
aborrero renamed T216132: CloudVPS: create wmcs-vm-fsck script from CloudVPS: crete wmcs-vm-fsck script to CloudVPS: create wmcs-vm-fsck script.
aborrero moved T216132: CloudVPS: create wmcs-vm-fsck script from Inbox to Important on the cloud-services-team (Kanban) board.
Wed, Feb 13
Wed, Feb 13
aborrero added a comment to T216004: Degraded RAID on cloudvirt1018.
The server is back online:
14:43 <gtirloni> slot 2: failed / slot 3: offline. I forced slot 2 offline and then online. I then forced slot 3 online. Reboot server and RAID reported the VD was fine.
aborrero updated the task description for T216030: cloudvps: evaluate draining cloudvirt1018.
aborrero added a comment to T216004: Degraded RAID on cloudvirt1018.
There are apparently 2 failed disks?
aborrero added a comment to T216004: Degraded RAID on cloudvirt1018.
After a reboot, I got this in the console:
aborrero moved T216030: cloudvps: evaluate draining cloudvirt1018 from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added a project to T216030: cloudvps: evaluate draining cloudvirt1018: cloud-services-team (Kanban).
aborrero closed T215968: profile::grafana - Could not find data item profile::grafana::secret_key as Resolved.
Solved by this patch: https://gerrit.wikimedia.org/r/#/c/labs/private/+/490298/
Tue, Feb 12
Tue, Feb 12
aborrero added a comment to T215888: openstack-browser: include deployment view support.
BTW all this info can be seen using Horizon in the admin project.
Krenair awarded T215888: openstack-browser: include deployment view support a Like token.
Mon, Feb 11
Mon, Feb 11
aborrero closed T215578: Weird dependency issue on the stretch grid around libmariadb-dev-compat and friends as Resolved.
Closing task. Feel free to reopen if required.
aborrero added a comment to T206706: Establish documentation review process for WMCS.
Fri, Feb 8
Fri, Feb 8
Bstorm awarded T215578: Weird dependency issue on the stretch grid around libmariadb-dev-compat and friends a Yellow Medal token.
Thanks @Cmjohnson and @MoritzMuehlenhoff, the server seems fine now:
aborrero moved T209029: cloudelastic1004: SMART/disk error from Needs discussion to Inbox on the cloud-services-team (Kanban) board.
I'm having troubles reimaging the server:
aborrero triaged T215605: cloudvps: missing packages in stretch for cloudcontrol servers as Normal priority.
aborrero moved T209029: cloudelastic1004: SMART/disk error from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
aborrero added a comment to T215578: Weird dependency issue on the stretch grid around libmariadb-dev-compat and friends.
TL;DR: I think we can have this combo installed: libgdal-dev libmariadbclient-dev-compat libmariadbclient-dev (even though I have doubts we need the -compat one). Will create a patch proposal for this.