Page MenuHomePhabricator

Andrew (Andrew Bogott)
User

Projects (10)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (254 w, 6 d)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott [ Global Accounts ]

Recent Activity

Yesterday

Andrew closed T229372: Remove swap partitions from VPS base images as Resolved.

I just uploaded a new Stretch image (9.11); now neither buster nor stretch will have swap partitions.

Sat, Sep 21, 10:09 PM · cloud-services-team (Kanban), Cloud-VPS

Thu, Sep 19

Wechat2 awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Manufacturing Defect? token.
Thu, Sep 19, 9:11 PM
Andrew awarded T232644: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 a Orange Medal token.
Thu, Sep 19, 8:54 PM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, cloud-services-team
Andrew added a comment to T210715: cloudvps: PDNS 3.x vs 4.x.

Now that we're running designate/newton this is unblocked. Switching will probably involve downtime, though, since we need to swap in a different pdns version at the same time as a different designate backend.

Thu, Sep 19, 8:30 PM · cloud-services-team (Kanban)
bd808 awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Barnstar token.
Thu, Sep 19, 8:21 PM
Andrew added a comment to T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker.

Other than the database cleanup this is now done.

Thu, Sep 19, 8:21 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Quiddity awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Love token.
Thu, Sep 19, 7:00 PM
Andrew added a comment to T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker.

after this is done we can clean up the db indicated by mysql://designate:<password>@clouddb2001-dev.codfw.wmnet/designate_pool_manager and the equivalent in prod

Thu, Sep 19, 6:51 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton.

Cloudservices1003 and 1004 are now running Designate version Newton. There are a few more steps that we should take before we're ready for Ocata there, though -- we need to move to the worker/producer model and also (probably) to pdns4.

Thu, Sep 19, 2:22 PM · Cloud-VPS, Patch-For-Review, cloud-services-team (Kanban)
Andrew closed T232555: Grant "Tool root" rights to Krenair as Resolved.

done!

Thu, Sep 19, 12:28 PM · cloud-services-team (Kanban), Toolforge

Wed, Sep 18

Krenair awarded Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair a Love token.
Wed, Sep 18, 9:58 PM
Andrew published Blog Post: Cloud-vps Puppetmasters Moved to VMs, thanks to Krenair.
Wed, Sep 18, 9:54 PM
Andrew created P9128 designate-mdns upset during record update.
Wed, Sep 18, 8:51 PM
Andrew created T233258: designate: switch from designate-pool-manager to designate-producer/designate-worker.
Wed, Sep 18, 7:58 PM · Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a parent task for T233236: Move labtestwikitech database to clouddb2001-dev: T201082: labtestweb2001 is sending updates to a read-only db host: db2037.
Wed, Sep 18, 4:24 PM · cloud-services-team (Kanban)
Andrew added a subtask for T201082: labtestweb2001 is sending updates to a read-only db host: db2037: T233236: Move labtestwikitech database to clouddb2001-dev.
Wed, Sep 18, 4:24 PM · cloud-services-team (Kanban), wikitech.wikimedia.org, Wikimedia-production-error
Andrew created T233236: Move labtestwikitech database to clouddb2001-dev.
Wed, Sep 18, 4:24 PM · cloud-services-team (Kanban)
Andrew added a comment to T232644: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002.

I'm sorry I didn't get to this! It sounds like you are (probably) all set.

Wed, Sep 18, 1:42 AM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure, cloud-services-team

Tue, Sep 17

Andrew added a comment to T205232: Request increased quota for xtools Cloud VPS project.

@MusikAnimal, I'm temporarily doubled the ram and CPU quotas in this project. Once you've created the new VMs and deleted the old ones let me know and I'll revert the change.

Tue, Sep 17, 7:07 PM · XTools, Cloud-VPS (Quota-requests)
Andrew added a comment to T205232: Request increased quota for xtools Cloud VPS project.

Approved, I'll help with this shortly.

Tue, Sep 17, 5:17 PM · XTools, Cloud-VPS (Quota-requests)

Mon, Sep 16

Andrew added a comment to T227476: Labtestwiki returns 503 error.

Sorry, this shouldn't have alerted -- the downtime expired. This will be talking to a test database server (clouddb2001-dev).

Mon, Sep 16, 1:51 PM · Wikimedia-production-error, wikitech.wikimedia.org, cloud-services-team (Kanban)

Fri, Sep 13

Andrew closed T232772: Audit tools project puppet CA certs to ensure that they are all consistent, a subtask of T232536: Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail, as Resolved.
Fri, Sep 13, 7:26 PM · cloud-services-team (Kanban), Toolforge
Andrew closed T232772: Audit tools project puppet CA certs to ensure that they are all consistent as Resolved.

I fixed the file that @Krenair mentioned and confirmed that /var/lib/puppet/ssl/certs/ca.pem == /etc/ssl/certs/Puppet_Internal_CA.pem ==/var/lib/puppet/client/ssl/certs/ca.pem on all hosts in tools.

Fri, Sep 13, 7:26 PM · Wikimedia-Incident, cloud-services-team (Kanban), Toolforge
Andrew closed T232428: Resolve local commits on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs and cloud-puppetmaster-02.cloudinfra.eqiad.wmflabs, a subtask of T171188: Move the main WMCS puppetmaster into the Labs realm, as Resolved.
Fri, Sep 13, 3:42 PM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew closed T232428: Resolve local commits on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs and cloud-puppetmaster-02.cloudinfra.eqiad.wmflabs as Resolved.

The attached patch resolves the issue without need for revert.

Fri, Sep 13, 3:42 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations

Thu, Sep 12

Andrew added a comment to T232428: Resolve local commits on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs and cloud-puppetmaster-02.cloudinfra.eqiad.wmflabs.

Here is what happens without those three reverts:

Thu, Sep 12, 4:13 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew created P9095 New VM cert problems.
Thu, Sep 12, 4:12 PM

Wed, Sep 11

Andrew added a subtask for T221631: Dedicated servers on WMCS to test WDQS scalability strategy: T232654: eqiad: three clouvirt-wdqs servers for WDQS testing.
Wed, Sep 11, 6:57 PM · cloud-services-team (Kanban), Wikidata, Wikidata-Query-Service, Discovery-Search
Andrew added a parent task for T232654: eqiad: three clouvirt-wdqs servers for WDQS testing: T221631: Dedicated servers on WMCS to test WDQS scalability strategy.
Wed, Sep 11, 6:57 PM · hardware-requests, Operations
Andrew created T232654: eqiad: three clouvirt-wdqs servers for WDQS testing.
Wed, Sep 11, 6:57 PM · hardware-requests, Operations
Andrew reassigned T232429: Create in-cloud, cloud-vps-wide cumin masters from Andrew to Krenair.

I think this task is done but I'll let @Krenair comment and close :)

Wed, Sep 11, 3:57 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew added a comment to T232429: Create in-cloud, cloud-vps-wide cumin masters.

I built a second cumin host, cloud-cumin-02.cloudinfra.eqiad.wmflabs. It's partly for backup, and partly because I wanted to confirm that the existing puppetization is sufficient. It turns out that it is! The new host just required a reboot to get keyholder on board.

Wed, Sep 11, 3:43 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew closed T157415: Puppet fails on tools-docker-builder-03 as Resolved.

Closing as the VM doesn't exist anymore

Wed, Sep 11, 3:40 PM · cloud-services-team (Kanban), Toolforge
Andrew closed T119660: Set up LVS for labs dns recursors as Declined.

I'm no longer clear that this is a good idea/necessary

Wed, Sep 11, 3:39 PM · cloud-services-team (Kanban), Cloud-VPS, Operations, Patch-For-Review
Andrew closed T216041: Order spare cloudvirt SSDs for eqiad, a subtask of T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure, as Declined.
Wed, Sep 11, 3:31 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew closed T216041: Order spare cloudvirt SSDs for eqiad as Declined.
Wed, Sep 11, 3:31 PM · Wikimedia-Incident, cloud-services-team (Kanban)
Andrew moved T210715: cloudvps: PDNS 3.x vs 4.x from Important to Inbox on the cloud-services-team (Kanban) board.
Wed, Sep 11, 3:30 PM · cloud-services-team (Kanban)
Andrew added a parent task for T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton: T210715: cloudvps: PDNS 3.x vs 4.x.
Wed, Sep 11, 3:30 PM · Cloud-VPS, Patch-For-Review, cloud-services-team (Kanban)
Andrew added a subtask for T210715: cloudvps: PDNS 3.x vs 4.x: T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton.
Wed, Sep 11, 3:30 PM · cloud-services-team (Kanban)
Andrew added a comment to T210715: cloudvps: PDNS 3.x vs 4.x.

Migrating to 4 requires designate support for pdns4, which is not present in Mitaka. So this needs to be delayed until at least one OpenStack version upgrade.

Wed, Sep 11, 3:29 PM · cloud-services-team (Kanban)
Andrew moved T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton from Important to Doing on the cloud-services-team (Kanban) board.
Wed, Sep 11, 3:28 PM · Cloud-VPS, Patch-For-Review, cloud-services-team (Kanban)
Andrew moved T168433: Deprecate DSA (ssh-dss) SSH keys for Cloud VPS and Toolforge users from Important to Doing on the cloud-services-team (Kanban) board.
Wed, Sep 11, 3:27 PM · Cloud-VPS, Toolforge, cloud-services-team (Kanban)
Andrew closed T228974: CloudVPS: codfw1dev: proper DNS setup as Resolved.
Wed, Sep 11, 3:26 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew closed T228974: CloudVPS: codfw1dev: proper DNS setup, a subtask of T229441: CloudVPS: codfw1dev: missing bits, as Resolved.
Wed, Sep 11, 3:26 PM · cloud-services-team (Kanban)
Andrew closed T228972: CloudVPS: codfw1dev: refresh glance images, a subtask of T229441: CloudVPS: codfw1dev: missing bits, as Resolved.
Wed, Sep 11, 3:26 PM · cloud-services-team (Kanban)
Andrew closed T228972: CloudVPS: codfw1dev: refresh glance images as Resolved.

We're using prod images in codf1dev now.

Wed, Sep 11, 3:26 PM · cloud-services-team (Kanban)
Andrew claimed T229441: CloudVPS: codfw1dev: missing bits.
Wed, Sep 11, 3:25 PM · cloud-services-team (Kanban)
Andrew moved T229441: CloudVPS: codfw1dev: missing bits from Important to Doing on the cloud-services-team (Kanban) board.
Wed, Sep 11, 3:25 PM · cloud-services-team (Kanban)
Andrew moved T177850: Page if the grid engine master is unreachable from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
Wed, Sep 11, 3:17 PM · observability, Toolforge, cloud-services-team (Kanban)
Andrew moved T164992: Add /public/dumps access via NFS to dumpgrepper instance from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
Wed, Sep 11, 3:17 PM · cloud-services-team (Kanban), Cloud-VPS
Andrew moved T218713: Clean up non-proxy entries in wmflabs.org zone from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
Wed, Sep 11, 3:16 PM · cloud-services-team (Kanban), Cloud-VPS
Andrew moved T228866: CloudVPS: VMs with broken puppet 2019-07-14 from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
Wed, Sep 11, 3:14 PM · cloud-services-team (Kanban)
Andrew added a comment to T232619: integration-agent-docker-1001.integration.eqiad.wmflabs is unreachable.

Oh, if this was created on Monday then it was probably built during the puppetmaster swap and was dead on arrival.

Wed, Sep 11, 2:25 PM · Release-Engineering-Team-TODO (201909), Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure

Tue, Sep 10

Andrew closed T232429: Create in-cloud, cloud-vps-wide cumin masters, a subtask of T171188: Move the main WMCS puppetmaster into the Labs realm, as Resolved.
Tue, Sep 10, 8:21 PM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew closed T232429: Create in-cloud, cloud-vps-wide cumin masters as Resolved.

There's now a cumin master on cloud-cumon-01.cloudinfra.eqiad.wmflabs that seems to work just fine.

Tue, Sep 10, 8:21 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew created T232509: cloud-puppetmasters: move some hiera settings from Horizon to git/gerrit.
Tue, Sep 10, 5:37 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew closed T232427: cloud-vps puppet cert cleaner not working properly as Resolved.
Tue, Sep 10, 4:14 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew closed T232427: cloud-vps puppet cert cleaner not working properly, a subtask of T171188: Move the main WMCS puppetmaster into the Labs realm, as Resolved.
Tue, Sep 10, 4:14 PM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew created T232429: Create in-cloud, cloud-vps-wide cumin masters.
Tue, Sep 10, 3:25 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew created T232428: Resolve local commits on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs and cloud-puppetmaster-02.cloudinfra.eqiad.wmflabs.
Tue, Sep 10, 3:23 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew created T232427: cloud-vps puppet cert cleaner not working properly.
Tue, Sep 10, 3:22 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew updated the task description for T171188: Move the main WMCS puppetmaster into the Labs realm.
Tue, Sep 10, 3:21 AM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations

Sun, Sep 8

Andrew added a comment to T229441: CloudVPS: codfw1dev: missing bits.

I have instance designate/dns working now on codfw1dev with the .cloud tld:

Sun, Sep 8, 3:16 AM · cloud-services-team (Kanban)
Andrew updated the task description for T229441: CloudVPS: codfw1dev: missing bits.
Sun, Sep 8, 3:14 AM · cloud-services-team (Kanban)

Fri, Sep 6

Andrew updated the task description for T229441: CloudVPS: codfw1dev: missing bits.
Fri, Sep 6, 7:44 PM · cloud-services-team (Kanban)
Andrew updated the task description for T224561: Migrate remaining cloudvirt hosts to Stretch/Mitaka.
Fri, Sep 6, 3:39 AM · cloud-services-team, Operations
Andrew updated the task description for T224549: Track remaining jessie systems in production.
Fri, Sep 6, 3:35 AM · Operations
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Fri, Sep 6, 3:34 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew closed T229871: relocate/reimage cloudvirt1023 with 10G interfaces as Resolved.
Fri, Sep 6, 3:33 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew closed T229871: relocate/reimage cloudvirt1023 with 10G interfaces, a subtask of T216195: Move cloudvirt hosts to 10Gb ethernet, as Resolved.
Fri, Sep 6, 3:33 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew closed T229872: relocate/reimage cloudvirt1022 with 10G interfaces as Resolved.
Fri, Sep 6, 3:33 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew closed T229872: relocate/reimage cloudvirt1022 with 10G interfaces, a subtask of T216195: Move cloudvirt hosts to 10Gb ethernet, as Resolved.
Fri, Sep 6, 3:33 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew closed T229873: relocate/reimage cloudvirt1021 with 10G interfaces as Resolved.
Fri, Sep 6, 3:33 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew closed T229873: relocate/reimage cloudvirt1021 with 10G interfaces, a subtask of T216195: Move cloudvirt hosts to 10Gb ethernet, as Resolved.
Fri, Sep 6, 3:33 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)

Wed, Sep 4

Andrew added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

@wiki_willy, the parent task of this task is the procurement for four identical systems: cloudvirt1015, 1016. 1017, 1018. 1018 has had some problems as well, but I don't see a lot of issues for 1016 or 1017 in phab history.

Wed, Sep 4, 9:05 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
Andrew added a parent task for T216004: Degraded RAID on cloudvirt1018: Unknown Object (Task).
Wed, Sep 4, 9:04 PM · cloud-services-team (Kanban), ops-eqiad, Operations
Andrew added a parent task for T216218: Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure: Unknown Object (Task).
Wed, Sep 4, 9:02 PM · Cloud-VPS, cloud-services-team (Kanban)
Andrew added a parent task for T230575: Degraded RAID on cloudvirt1018: Unknown Object (Task).
Wed, Sep 4, 9:01 PM · ops-eqiad, Operations
Andrew reopened T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory as "Open".

I can still make this crash -- my process is scheduling 80 VMs on the host, and then getting them all busy, like this:

Wed, Sep 4, 8:25 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
Andrew created P9043 Console output of a crashing cloudvirt1015.
Wed, Sep 4, 8:24 PM
Andrew created P9042 cloudvirt1015 crashing after a new mainboard on 2019-09-04.
Wed, Sep 4, 8:20 PM
Andrew added a comment to T231811: Check router ACLs for early install SSH access from puppet masters/cumin hosts.

install_console works just fine from cumin1001. So, no need for a special case here, we can just go ahead and decom iron.

Wed, Sep 4, 3:34 PM · netops, Operations
Andrew added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

(I just now enabled virtualization in the bios)

Wed, Sep 4, 3:11 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
Andrew added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

btw, @Cmjohnson, did you restore BIOS settings after replacing the board?

Wed, Sep 4, 3:06 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
Andrew added a comment to T231811: Check router ACLs for early install SSH access from puppet masters/cumin hosts.

On the system/ferm level there's fleet-wide Ferm rule which grants SSH access from Cumin masters.
On the router level, Arzhel needs to comment.

Wed, Sep 4, 2:41 PM · netops, Operations
Andrew claimed T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

I'll see if I can make it crash again!

Wed, Sep 4, 2:27 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
Andrew updated subscribers of T231811: Check router ACLs for early install SSH access from puppet masters/cumin hosts.

Moving the iron exception to cloudnet1003 works for me -- presuming we mean adding it to 'wmcs::openstack::eqiad1::net'.

Wed, Sep 4, 2:24 PM · netops, Operations
Andrew reassigned T229873: relocate/reimage cloudvirt1021 with 10G interfaces from Jclark-ctr to Cmjohnson.
Wed, Sep 4, 12:09 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew reassigned T229872: relocate/reimage cloudvirt1022 with 10G interfaces from Jclark-ctr to Cmjohnson.
Wed, Sep 4, 12:09 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew updated the task description for T229873: relocate/reimage cloudvirt1021 with 10G interfaces.
Wed, Sep 4, 12:08 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew added a comment to T229872: relocate/reimage cloudvirt1022 with 10G interfaces.

I can confirm that this seems to have 10g networking set up already. So all that's left dc-wise is the raid setup.

Wed, Sep 4, 12:08 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew added a comment to T229873: relocate/reimage cloudvirt1021 with 10G interfaces.

I can confirm that this seems to have 10g networking set up already. So all that's left dc-wise is the raid setup.

Wed, Sep 4, 12:08 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)

Aug 22 2019

Andrew updated subscribers of T229873: relocate/reimage cloudvirt1021 with 10G interfaces.

@Cmjohnson and @wiki_willy, I just want to clarify what's happening on this ticket.

Aug 22 2019, 9:01 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew updated subscribers of T229872: relocate/reimage cloudvirt1022 with 10G interfaces.

@Cmjohnson and @wiki_willy, I just want to clarify what's happening on this ticket.

Aug 22 2019, 9:01 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew updated subscribers of T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

@Cmjohnson and @wiki_willy, I just want to clarify what's happening on this ticket.

Aug 22 2019, 9:01 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)

Aug 21 2019

bd808 awarded T229587: Introduction to Wikimedia Cloud Services session a 100 token.
Aug 21 2019, 4:23 PM · cloud-services-team (Kanban), Wikimania-Hackathon-2019

Aug 18 2019

Andrew added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

I did a manual install-console on this host and it's doing its initial puppet run now.

Aug 18 2019, 9:53 AM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew added a comment to T230674: shinken: issue with shinkengen.

The shinken service itself also won't start up properly:

Aug 18 2019, 9:46 AM · Shinken, cloud-services-team (Kanban)

Aug 15 2019

Andrew updated the task description for T171188: Move the main WMCS puppetmaster into the Labs realm.
Aug 15 2019, 10:50 AM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Andrew updated the task description for T171188: Move the main WMCS puppetmaster into the Labs realm.
Aug 15 2019, 10:41 AM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations