Page MenuHomePhabricator

Andrew (Andrew Bogott)
User

Projects (10)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (258 w, 6 h)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott [ Global Accounts ]

Recent Activity

Thu, Oct 10

Andrew merged T208884: Puppet errors on automation-framework project into T234452: Puppet breakage in automation-feedback VMs.
Thu, Oct 10, 9:38 PM · Operations
Andrew merged task T208884: Puppet errors on automation-framework project into T234452: Puppet breakage in automation-feedback VMs.
Thu, Oct 10, 9:38 PM · VPS-Projects
Andrew added a comment to T232676: af-nb-db-2.automation-framework.eqiad.wmflabs has broken network.

This is an error that sometimes happens during VM creation -- I think it's something like...

Thu, Oct 10, 9:37 PM · Cloud-VPS
Andrew added a parent task for T234452: Puppet breakage in automation-feedback VMs: T228866: CloudVPS: VMs with broken puppet 2019-07-14.
Thu, Oct 10, 9:30 PM · Operations
Andrew added a subtask for T228866: CloudVPS: VMs with broken puppet 2019-07-14: T234452: Puppet breakage in automation-feedback VMs.
Thu, Oct 10, 9:30 PM · cloud-services-team (Kanban)
Andrew updated subscribers of T235218: Catch cloud-puppetmasters up with production puppetmaster versions.
Thu, Oct 10, 8:51 PM · cloud-services-team (Kanban)
Andrew created T235218: Catch cloud-puppetmasters up with production puppetmaster versions.
Thu, Oct 10, 8:51 PM · cloud-services-team (Kanban)
Andrew committed rLPRIe2106a72719c: Added snakeoil certs for cloud puppetmasters (authored by Andrew).
Added snakeoil certs for cloud puppetmasters
Thu, Oct 10, 7:46 PM
Andrew committed rLPRI703e2b1e2cbe: Added dummy password for cloud puppetmasters (authored by Andrew).
Added dummy password for cloud puppetmasters
Thu, Oct 10, 7:43 PM
Andrew renamed T234683: Build, package bdsync for Buster from Build bdsync for Buster, or update block_sync.py script to use rsync --copy-devices to Build, package bdsync for Buster.
Thu, Oct 10, 1:56 PM · Cloud-Services, ops-codfw, Operations
Andrew closed T234876: nova-conductor running out of mysql connections, a subtask of T234834: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07, as Resolved.
Thu, Oct 10, 1:48 PM · Wikimedia-Incident, Release-Engineering-Team (CI & Testing services), cloud-services-team (Kanban), Cloud-VPS, Tools
Andrew closed T234876: nova-conductor running out of mysql connections as Resolved.
Thu, Oct 10, 1:48 PM · cloud-services-team (Kanban)
Andrew closed T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker, a subtask of T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton, as Resolved.
Thu, Oct 10, 1:48 PM · Cloud-VPS, Patch-For-Review, cloud-services-team (Kanban)
Andrew closed T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker as Resolved.
Thu, Oct 10, 1:47 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Wed, Oct 9

Andrew created T235129: nova-fullstack: add cleanup checking.
Wed, Oct 9, 9:04 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a comment to T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress .

This all sounds good to me. lmk if you need me to make the designate changes.

Wed, Oct 9, 5:56 PM · cloud-services-team (Kanban)

Tue, Oct 8

Andrew added a comment to T234876: nova-conductor running out of mysql connections.

I'm merging an experimental patch to reduce the number of connections needed. It's possible that this issue was caused by Newton upgrade (and some changein behavior) but it could also be a result of us switching to an HA setup (if the connection limit on the db side is per user/database and not per host/user/database).

Tue, Oct 8, 3:18 AM · cloud-services-team (Kanban)
Andrew added a comment to T234876: nova-conductor running out of mysql connections.

heh, the first forum post I found about this topic suggests raising the connection limit to 2000

Tue, Oct 8, 3:08 AM · cloud-services-team (Kanban)
Andrew triaged T234876: nova-conductor running out of mysql connections as High priority.
Tue, Oct 8, 3:02 AM · cloud-services-team (Kanban)
Andrew created T234876: nova-conductor running out of mysql connections.
Tue, Oct 8, 3:02 AM · cloud-services-team (Kanban)

Mon, Oct 7

Andrew added a comment to T234683: Build, package bdsync for Buster.

@arturo if you wanted to submit the package upstream there's at least one other person who would appreciate it.

Mon, Oct 7, 1:06 PM · Cloud-Services, ops-codfw, Operations

Fri, Oct 4

Andrew added a comment to T234683: Build, package bdsync for Buster.

@aborrero, do you have intuition about whether packaging bdsync for Buster is hard or easy?

Fri, Oct 4, 6:51 PM · Cloud-Services, ops-codfw, Operations
Andrew renamed T234683: Build, package bdsync for Buster from Update block_sync.py script to use rsync --copy-devices to Build bdsync for Buster, or update block_sync.py script to use rsync --copy-devices.
Fri, Oct 4, 6:33 PM · Cloud-Services, ops-codfw, Operations
Andrew created T234683: Build, package bdsync for Buster.
Fri, Oct 4, 6:23 PM · Cloud-Services, ops-codfw, Operations
Andrew added a comment to T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.

(nevermind, I think I see what's happening)

Fri, Oct 4, 3:14 PM · Cloud-Services, Operations, ops-codfw

Thu, Oct 3

Andrew added a comment to T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.

I am finally back looking at this! I'm not sure quite what I should expect regarding the raids here -- I reimaged (for buster) and saw the partitioner offer to create two volumes, but in the OS I only see one:

Thu, Oct 3, 8:44 PM · Cloud-Services, Operations, ops-codfw
Andrew added a comment to T233978: Drop 'designate_pool_manager' database from m5 and remove associated grants.
Thu, Oct 3, 2:06 PM · Patch-For-Review, DBA

Wed, Oct 2

Andrew closed T104620: Automatically sync puppet certs from primary labs controller to backup as Invalid.

I'm closing this because everything is different now

Wed, Oct 2, 9:01 PM · cloud-services-team (Kanban), Cloud-VPS
Andrew reassigned T233236: Move labtestwikitech database to clouddb2001-dev from Andrew to Joe.

The system for assigning a particular wiki to a particular db host in mediawiki-config has changed a lot since I last touched this code. @Joe, if you could write me a sample patch of how to break out labtestwiki into its own group and direct it to a different db server, I should be able to take it from there.

Wed, Oct 2, 8:17 PM · cloud-services-team (Kanban)
Andrew updated the task description for T229441: CloudVPS: codfw1dev: missing bits.
Wed, Oct 2, 8:08 PM · cloud-services-team (Kanban)
Andrew added a comment to T229441: CloudVPS: codfw1dev: missing bits.

regarding ldap: I just created a new project in codfw1dev and added a member. Ldap config looks correct to me, for example:

Wed, Oct 2, 8:08 PM · cloud-services-team (Kanban)
Andrew created T234462: reclaim/decom/whatever labpuppetmaster1001 and 1002.
Wed, Oct 2, 6:00 PM · decommission, DC-Ops
Andrew closed T227918: labpuppetmaster1001/1002 complaining about unmerged patches as Resolved.

this is moot now that labpuppetmaster1001/1002 aren't puppetmasters anymore :)

Wed, Oct 2, 5:58 PM · observability, cloud-services-team (Kanban)
Andrew added a comment to T234452: Puppet breakage in automation-feedback VMs.

I dug a little deeper, and the primary issue is local diffs in /var/lib/git/operations/puppet on af-puppetmaster02.automation-framework.eqiad.wmflabs. If you commit those and are able to get a sensible rebase with modern upstream puppet, then you'll need to update a couple of other things:

Wed, Oct 2, 4:16 PM · Operations
Andrew created T234452: Puppet breakage in automation-feedback VMs.
Wed, Oct 2, 4:15 PM · Operations

Tue, Oct 1

Andrew assigned T234269: Request for temporarily increased quota for dwl Cloud VPS project to rebuild and test deprecated instances to aborrero.
Tue, Oct 1, 5:01 PM · Cloud-VPS (Quota-requests)

Mon, Sep 30

Andrew added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

Things look better after that last patch

Mon, Sep 30, 8:55 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539960/

Mon, Sep 30, 8:37 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

I noticed this issue because of the source IP that was detected by the dns recursor on cloudservices2002-dev. After this change, things are slightly worse:

Mon, Sep 30, 7:15 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew created T234258: Delete readers-web-stephen.reading-webstaging.eqiad.wmflabs and marvin-staging.reading-webstaging.eqiad.wmflabs?.
Mon, Sep 30, 6:56 PM · Readers-Web-Backlog (Tracking)
Andrew added a comment to T210008: upgrade krypton (webserver_misc_apps) to stretch.

Is T210008.wikistats.eqiad.wmflabs associated with this bug? It has had broken puppet for many weeks -- perhaps it can be deleted?

Mon, Sep 30, 6:44 PM · serviceops, Operations
Andrew created T234256: Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs.
Mon, Sep 30, 6:43 PM · Operations, Traffic

Fri, Sep 27

Andrew edited projects for T225713: CPU scaling governor audit, added: cloud-services-team (Kanban); removed cloud-services-team.
Fri, Sep 27, 8:25 PM · cloud-services-team (Kanban), Cloud-VPS, User-fgiunchedi, Operations
Andrew placed T233978: Drop 'designate_pool_manager' database from m5 and remove associated grants up for grabs.
Fri, Sep 27, 2:03 PM · Patch-For-Review, DBA

Thu, Sep 26

Andrew created T233978: Drop 'designate_pool_manager' database from m5 and remove associated grants.
Thu, Sep 26, 6:20 PM · Patch-For-Review, DBA

Wed, Sep 25

Andrew added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

(btw, I bet that exposing these IPs to production hosts breaks a lot of our 'future ideal model' rules, so if we can move towards total-outside-world-natting it might be considered forward progress in some circles)

Wed, Sep 25, 11:26 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

The other question I have about this hack is... do we need it? The issue I ran into that caused me to notice it was the dns-recursors not recognizing the source IPs, but that's quite easy for me to work around.

Wed, Sep 25, 11:24 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew closed T233440: Raise quota for integration project as Resolved.

I put a new cloudvirt online yesterday, and boosted your quotas. If things get scheduled on hdd systems and you need them moved just let me know.

Wed, Sep 25, 11:15 PM · Continuous-Integration-Infrastructure, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (201909), Cloud-VPS (Quota-requests)
Andrew moved T233347: Remove access.log generation from default lighttpd.conf generated by `webservice` from Needs discussion to Doing on the cloud-services-team (Kanban) board.
Wed, Sep 25, 3:30 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
Andrew assigned T233347: Remove access.log generation from default lighttpd.conf generated by `webservice` to Phamhi.
Wed, Sep 25, 3:30 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
Andrew moved T227029: Prevent catalog breakage on cloud instances by decoupling core cloud puppetmaster from custom puppetmasters from Doing to Important on the cloud-services-team (Kanban) board.
Wed, Sep 25, 3:25 PM · Puppet, cloud-services-team (Kanban)
Andrew moved T226270: Reduce the effects of puppet breakage on VPS from Doing to Important on the cloud-services-team (Kanban) board.
Wed, Sep 25, 3:25 PM · Puppet, cloud-services-team (Kanban)
Andrew moved T221631: Dedicated servers on WMCS to test WDQS scalability strategy from Doing to Blocked on the cloud-services-team (Kanban) board.
Wed, Sep 25, 3:24 PM · cloud-services-team (Kanban), Wikidata, Wikidata-Query-Service, Discovery-Search
Andrew placed T187504: integrate proxy handling with the designate recordset UI up for grabs.
Wed, Sep 25, 3:19 PM · Horizon, cloud-services-team (Kanban)
Andrew moved T187504: integrate proxy handling with the designate recordset UI from Doing to Graveyard on the cloud-services-team (Kanban) board.
Wed, Sep 25, 3:19 PM · Horizon, cloud-services-team (Kanban)
Andrew closed T216132: CloudVPS: create wmcs-vm-fsck script, a subtask of T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure, as Declined.
Wed, Sep 25, 3:17 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew closed T216132: CloudVPS: create wmcs-vm-fsck script as Declined.

Brooke took a stab at this but writing the script turns out to be non-trivial; this happens infrequently and we have good docs now so we're going to try to avoid writing this.

Wed, Sep 25, 3:17 PM · Wikimedia-Incident, cloud-services-team (Kanban)
Andrew moved T233236: Move labtestwikitech database to clouddb2001-dev from Inbox to Doing on the cloud-services-team (Kanban) board.
Wed, Sep 25, 3:06 PM · cloud-services-team (Kanban)
Tgr awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Barnstar token.
Wed, Sep 25, 10:58 AM

Tue, Sep 24

Andrew closed T205232: Request increased quota for xtools Cloud VPS project, a subtask of T232787: XTools: Move Debian Jessie instances to Buster, as Resolved.
Tue, Sep 24, 5:45 PM · XTools
Andrew closed T205232: Request increased quota for xtools Cloud VPS project as Resolved.

@MusikAnimal nice work! I've revered the quota boost.

Tue, Sep 24, 5:45 PM · XTools, Cloud-VPS (Quota-requests)
Andrew added a comment to T233347: Remove access.log generation from default lighttpd.conf generated by `webservice`.

Approved during WMCS meeting

Tue, Sep 24, 4:17 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
Andrew added a comment to T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.

If someone (arturo?) knows how to reliably forward the patch, I'm inclined to go with that for now and then refactor to other mechanisms post-upgrade, just in the interest of changing fewer things at a time. I don't know if forwarding the patch is something we can do without introducing unknowns though.

Tue, Sep 24, 2:13 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T231793: Remove systemd from openstack-mitaka.

We're precariously close to upgrading to Newton, so maybe this is moot?

Tue, Sep 24, 2:03 PM · cloud-services-team (Kanban), Operations

Mon, Sep 23

Andrew created T233665: Forward our neutron-l3-agent routing hacks to Openstack Newton.
Mon, Sep 23, 9:19 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Sat, Sep 21

Andrew closed T229372: Remove swap partitions from VPS base images as Resolved.

I just uploaded a new Stretch image (9.11); now neither buster nor stretch will have swap partitions.

Sat, Sep 21, 10:09 PM · cloud-services-team (Kanban), Cloud-VPS

Thu, Sep 19

Wechat2 awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Manufacturing Defect? token.
Thu, Sep 19, 9:11 PM
Andrew awarded T232644: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 a Orange Medal token.
Thu, Sep 19, 8:54 PM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, cloud-services-team
Andrew added a comment to T210715: cloudvps: PDNS 3.x vs 4.x.

Now that we're running designate/newton this is unblocked. Switching will probably involve downtime, though, since we need to swap in a different pdns version at the same time as a different designate backend.

Thu, Sep 19, 8:30 PM · cloud-services-team (Kanban)
bd808 awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Barnstar token.
Thu, Sep 19, 8:21 PM
Andrew added a comment to T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker.

Other than the database cleanup this is now done.

Thu, Sep 19, 8:21 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Quiddity awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Love token.
Thu, Sep 19, 7:00 PM
Andrew added a comment to T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker.

after this is done we can clean up the db indicated by mysql://designate:<password>@clouddb2001-dev.codfw.wmnet/designate_pool_manager and the equivalent in prod

Thu, Sep 19, 6:51 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton.

Cloudservices1003 and 1004 are now running Designate version Newton. There are a few more steps that we should take before we're ready for Ocata there, though -- we need to move to the worker/producer model and also (probably) to pdns4.

Thu, Sep 19, 2:22 PM · Cloud-VPS, Patch-For-Review, cloud-services-team (Kanban)
Andrew closed T232555: Grant "Tool root" rights to Krenair as Resolved.

done!

Thu, Sep 19, 12:28 PM · cloud-services-team (Kanban), Toolforge

Wed, Sep 18

Krenair awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Love token.
Wed, Sep 18, 9:58 PM
Andrew published Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair.
Wed, Sep 18, 9:54 PM
Andrew created P9128 designate-mdns upset during record update.
Wed, Sep 18, 8:51 PM
Andrew created T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker.
Wed, Sep 18, 7:58 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a parent task for T233236: Move labtestwikitech database to clouddb2001-dev: T201082: labtestweb2001 is sending updates to a read-only db host: db2037.
Wed, Sep 18, 4:24 PM · cloud-services-team (Kanban)
Andrew added a subtask for T201082: labtestweb2001 is sending updates to a read-only db host: db2037: T233236: Move labtestwikitech database to clouddb2001-dev.
Wed, Sep 18, 4:24 PM · cloud-services-team (Kanban), wikitech.wikimedia.org, Wikimedia-production-error
Andrew created T233236: Move labtestwikitech database to clouddb2001-dev.
Wed, Sep 18, 4:24 PM · cloud-services-team (Kanban)
Andrew added a comment to T232644: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002.

I'm sorry I didn't get to this! It sounds like you are (probably) all set.

Wed, Sep 18, 1:42 AM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, cloud-services-team

Tue, Sep 17

Andrew added a comment to T205232: Request increased quota for xtools Cloud VPS project.

@MusikAnimal, I'm temporarily doubled the ram and CPU quotas in this project. Once you've created the new VMs and deleted the old ones let me know and I'll revert the change.

Tue, Sep 17, 7:07 PM · XTools, Cloud-VPS (Quota-requests)
Andrew added a comment to T205232: Request increased quota for xtools Cloud VPS project.

Approved, I'll help with this shortly.

Tue, Sep 17, 5:17 PM · XTools, Cloud-VPS (Quota-requests)

Mon, Sep 16

Andrew added a comment to T227476: Labtestwiki returns 503 error.

Sorry, this shouldn't have alerted -- the downtime expired. This will be talking to a test database server (clouddb2001-dev).

Mon, Sep 16, 1:51 PM · Wikimedia-production-error, wikitech.wikimedia.org, cloud-services-team (Kanban)

Sep 13 2019

Andrew closed T232772: Audit tools project puppet CA certs to ensure that they are all consistent, a subtask of T232536: Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail, as Resolved.
Sep 13 2019, 7:26 PM · Wikimedia-Incident, cloud-services-team (Kanban), Toolforge
Andrew closed T232772: Audit tools project puppet CA certs to ensure that they are all consistent as Resolved.

I fixed the file that @Krenair mentioned and confirmed that /var/lib/puppet/ssl/certs/ca.pem == /etc/ssl/certs/Puppet_Internal_CA.pem ==/var/lib/puppet/client/ssl/certs/ca.pem on all hosts in tools.

Sep 13 2019, 7:26 PM · Wikimedia-Incident, cloud-services-team (Kanban), Toolforge
Andrew closed T232428: Resolve local commits on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs and cloud-puppetmaster-02.cloudinfra.eqiad.wmflabs, a subtask of T171188: Move the main WMCS puppetmaster into the Labs realm, as Resolved.
Sep 13 2019, 3:42 PM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew closed T232428: Resolve local commits on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs and cloud-puppetmaster-02.cloudinfra.eqiad.wmflabs as Resolved.

The attached patch resolves the issue without need for revert.

Sep 13 2019, 3:42 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations

Sep 12 2019

Andrew added a comment to T232428: Resolve local commits on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs and cloud-puppetmaster-02.cloudinfra.eqiad.wmflabs.

Here is what happens without those three reverts:

Sep 12 2019, 4:13 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew created P9095 New VM cert problems.
Sep 12 2019, 4:12 PM

Sep 11 2019

Andrew added a subtask for T221631: Dedicated servers on WMCS to test WDQS scalability strategy: T232654: eqiad: three clouvirt-wdqs servers for WDQS testing.
Sep 11 2019, 6:57 PM · cloud-services-team (Kanban), Wikidata, Wikidata-Query-Service, Discovery-Search
Andrew added a parent task for T232654: eqiad: three clouvirt-wdqs servers for WDQS testing: T221631: Dedicated servers on WMCS to test WDQS scalability strategy.
Sep 11 2019, 6:57 PM · Operations, hardware-requests
Andrew created T232654: eqiad: three clouvirt-wdqs servers for WDQS testing.
Sep 11 2019, 6:57 PM · Operations, hardware-requests
Andrew reassigned T232429: Create in-cloud, cloud-vps-wide cumin masters from Andrew to Krenair.

I think this task is done but I'll let @Krenair comment and close :)

Sep 11 2019, 3:57 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew added a comment to T232429: Create in-cloud, cloud-vps-wide cumin masters.

I built a second cumin host, cloud-cumin-02.cloudinfra.eqiad.wmflabs. It's partly for backup, and partly because I wanted to confirm that the existing puppetization is sufficient. It turns out that it is! The new host just required a reboot to get keyholder on board.

Sep 11 2019, 3:43 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew closed T157415: Puppet fails on tools-docker-builder-03 as Resolved.

Closing as the VM doesn't exist anymore

Sep 11 2019, 3:40 PM · cloud-services-team (Kanban), Toolforge
Andrew closed T119660: Set up LVS for labs dns recursors as Declined.

I'm no longer clear that this is a good idea/necessary

Sep 11 2019, 3:39 PM · cloud-services-team (Kanban), Cloud-VPS, Operations, Patch-For-Review
Andrew closed T216041: Order spare cloudvirt SSDs for eqiad, a subtask of T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure, as Declined.
Sep 11 2019, 3:31 PM · Cloud-VPS, cloud-services-team (Kanban)