Andrew (Andrew Bogott)
User

Projects (10)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (162 w, 3 d)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott

Recent Activity

Today

Andrew renamed T182893: Get role::puppetmaster::standalone working on Stretch from Get role::puppetmaster::self working on Stretch to Get role::puppetmaster::standalone working on Stretch.
Thu, Dec 14, 4:03 PM · cloud-services-team
Andrew created T182893: Get role::puppetmaster::standalone working on Stretch.
Thu, Dec 14, 4:02 PM · cloud-services-team

Yesterday

Andrew added a subtask for T182810: Remove role::puppet::self and related support code: T128740: role::puppet::self requires a puppetmaster restart during apply.
Wed, Dec 13, 6:10 PM · cloud-services-team (Kanban), Puppet
Andrew added a parent task for T128740: role::puppet::self requires a puppetmaster restart during apply: T182810: Remove role::puppet::self and related support code.
Wed, Dec 13, 6:10 PM · Cloud-Services
Andrew added a subtask for T120159: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module: T182810: Remove role::puppet::self and related support code.
Wed, Dec 13, 6:09 PM · Patch-For-Review, Cloud-Services, Operations
Andrew added a parent task for T182810: Remove role::puppet::self and related support code: T120159: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module.
Wed, Dec 13, 6:09 PM · cloud-services-team (Kanban), Puppet
Andrew created T182810: Remove role::puppet::self and related support code.
Wed, Dec 13, 6:08 PM · cloud-services-team (Kanban), Puppet
Andrew updated subscribers of T178717: Upgrade wmcs instances and masters to puppet 4.8.

All masters are upgraded, and Debian base images now include puppet 4.

Wed, Dec 13, 5:59 PM · Patch-For-Review, Cloud-Services
Andrew closed T182721: Stop using etckeeper (at least before/after puppet runs) as Resolved.
Wed, Dec 13, 5:01 AM · Patch-For-Review, cloud-services-team, Puppet, Puppet-infrastructure-modernization, Operations

Tue, Dec 12

Andrew added projects to T182721: Stop using etckeeper (at least before/after puppet runs): Operations, Puppet-infrastructure-modernization, Puppet, cloud-services-team.
Tue, Dec 12, 10:09 PM · Patch-For-Review, cloud-services-team, Puppet, Puppet-infrastructure-modernization, Operations
Andrew created T182721: Stop using etckeeper (at least before/after puppet runs).
Tue, Dec 12, 10:09 PM · Patch-For-Review, cloud-services-team, Puppet, Puppet-infrastructure-modernization, Operations
Andrew closed T181836: Delete the wikidata-build cloud VPS project as Resolved.

deleted

Tue, Dec 12, 6:16 PM · Cloud-VPS (Project-requests), cloud-services-team (Kanban), User-Addshore, Wikidata
Andrew closed T181836: Delete the wikidata-build cloud VPS project, a subtask of T181706: Kill the automated wikidata build on labs, as Resolved.
Tue, Dec 12, 6:16 PM · Patch-For-Review, User-Addshore, Wikidata
Andrew added a comment to T182663: labtestwiki memcache errors, lots of requests from parsoid to labtestweb2001.

@ssastry, labtestweb and wikitech (basically anything with 'lab' in its db name) are baby little servers that are unlikely to handle automated traffic very well. Can you just exclude them from future tests?

Tue, Dec 12, 3:14 PM · Parsoid, cloud-services-team, MediaWiki-General-or-Unknown
Andrew updated the task description for T178717: Upgrade wmcs instances and masters to puppet 4.8.
Tue, Dec 12, 1:53 PM · Patch-For-Review, Cloud-Services

Tue, Dec 5

Andrew added a comment to T182070: tools-webgrid-lighttpd have ~ 90 procs stuck at 100% CPU time (mostly tools.jembot).

Chase killed off all of jem's processes, and CPU usage is plummeting.

Tue, Dec 5, 3:42 PM · Cloud-VPS, Toolforge

Fri, Dec 1

Andrew added a comment to T181375: Revamp first boot process for new VMs.

My patch to define 'puppet' works for what it is...

Fri, Dec 1, 11:47 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a comment to T180462: puppet-compiler issue with CloudVPS instances.

Full instructions for how to set up a working puppet compiler are on https://phabricator.wikimedia.org/T97081

Fri, Dec 1, 7:01 PM · Cloud-VPS, cloud-services-team

Thu, Nov 30

Andrew claimed T178717: Upgrade wmcs instances and masters to puppet 4.8.
Thu, Nov 30, 5:54 PM · Patch-For-Review, Cloud-Services

Wed, Nov 29

Andrew added a comment to T170492: figure out if nodepool is overwhelming rabbitmq and/or nova.

Mentioned in SAL (#wikimedia-operations) [2017-11-29T22:23:14Z] <hashar> Nodepool had some troubles spawning new instances from 21:09 to 21:36, and took a while to recover. Issue similar to T170492#3581822

Wed, Nov 29, 10:53 PM · cloud-services-team (Kanban), Release-Engineering-Team (Watching / External), Nodepool, Cloud-VPS, Continuous-Integration-Infrastructure, Patch-For-Review

Tue, Nov 28

Andrew created T181551: Puppet w/environments breaks the role/profile filters on Horizon.
Tue, Nov 28, 6:59 PM · cloud-services-team (Kanban), Horizon

Mon, Nov 27

Qgil awarded T180662: Request creation of qna VPS project a Party Time token.
Mon, Nov 27, 10:52 AM · Cloud-VPS (Project-requests)
Andrew created T181375: Revamp first boot process for new VMs.
Mon, Nov 27, 2:48 AM · Patch-For-Review, cloud-services-team (Kanban)

Sun, Nov 26

Andrew added a comment to T178661: Drop wb_entity_per_page views in Wiki Replicas.

@Marostegui this is on my plate but I have no idea how to proceed... is this timeout due to some long-running job that I need to locate and kill?

Sun, Nov 26, 8:00 PM · cloud-services-team (Kanban), Data-Services, Wikidata-Former-Sprint-Board, Wikidata
Andrew closed T180662: Request creation of qna VPS project as Resolved.

@Tgr: I created this project with you as a project admin. You should be able to add new members or admins as needed via wikitech -- let me know if you need anything else.

Sun, Nov 26, 7:12 PM · Cloud-VPS (Project-requests)

Fri, Nov 24

Andrew closed T181204: delete project "gerrit-staging" as Resolved.

deleted!

Fri, Nov 24, 6:29 PM · Cloud-VPS

Thu, Nov 23

Andrew added a comment to T181204: delete project "gerrit-staging".

gerrit-staging contains one instance, 'gerrit-stretch' which is about 4 hours old. So someone thinks the project is still active :)

Thu, Nov 23, 4:44 AM · Cloud-VPS

Wed, Nov 22

Andrew closed T181196: Horizon: editing hiera clears role params as Resolved.

I think this is fixed.

Wed, Nov 22, 10:41 PM · Patch-For-Review, Horizon, cloud-services-team
Andrew created T181196: Horizon: editing hiera clears role params.
Wed, Nov 22, 9:03 PM · Patch-For-Review, Horizon, cloud-services-team
Andrew added a comment to T180662: Request creation of qna VPS project.

@Tgr is this still something you're interested in, or is it resolved via discussion in T180854?

Wed, Nov 22, 4:20 PM · Cloud-VPS (Project-requests)
Andrew closed T180347: Request creation of wikidata-primary-sources-tool VPS project as Resolved.

@Hjfocs I've created the new project; you should be able to access it and add additional users or project admins as needed.

Wed, Nov 22, 4:05 PM · Wikidata-primary-sources, Wikidata, Cloud-VPS (Project-requests)
Andrew closed T180421: Request creation of PartnerMetrics VPS project as Resolved.

@Harej we don't support CamelCase project names, but I've created the 'partnermetrics' with you as an admin. You can add other users as needed.

Wed, Nov 22, 4:02 PM · Cloud-VPS (Project-requests)

Tue, Nov 21

Andrew closed T180178: Request increased quota for traffic Cloud VPS project as Resolved.

All set.

Tue, Nov 21, 6:48 PM · netops, Traffic, Cloud-VPS (Quota-requests), Operations

Mon, Nov 20

Andrew added a comment to T168470: Setup wikitech, horizon, and striker on new labweb hardware.

I tried to set MW up on these systems but the puppet classes on Stretch were a total mess so I just downtimed them and set this aside. Probably someone fixed puppet and they sprang to life.

Mon, Nov 20, 9:24 PM · cloud-services-team (Kanban), Cloud-Services
Andrew created Blog Post: Ubuntu Trusty now deprecated for new WMCS instances.
Mon, Nov 20, 6:32 PM · cloud-services-team
Andrew closed T161899: End self-service new Trusty instance creation in Cloud VPS; standardize on Debian base images as Resolved.

I made our standard Trusty image 'private' by default, and announced here: https://lists.wikimedia.org/pipermail/cloud-announce/2017-November/000010.html

Mon, Nov 20, 6:23 PM · cloud-services-team (Kanban), Cloud-VPS, User-bd808, Operations
Andrew closed T177298: Switch Toolforge project hosts to the future parser as Resolved.

Done with https://gerrit.wikimedia.org/r/#/c/392172/

Mon, Nov 20, 5:16 PM · cloud-services-team (Kanban), Toolforge, Puppet
Andrew closed T177298: Switch Toolforge project hosts to the future parser, a subtask of T169548: Prepare for Puppet 4, as Resolved.
Mon, Nov 20, 5:16 PM · User-Joe, Puppet, Operations
Andrew closed T177298: Switch Toolforge project hosts to the future parser, a subtask of T179451: Switch Cloud VPS puppet default to future parser, as Resolved.
Mon, Nov 20, 5:16 PM · cloud-services-team (Kanban), Cloud-VPS, Puppet
Andrew updated subscribers of T180837: Take down livingstyleguide.wmflabs.org.
Mon, Nov 20, 12:59 AM · Design
Andrew edited projects for T180837: Take down livingstyleguide.wmflabs.org, added: Design; removed Cloud-Services.
Mon, Nov 20, 12:57 AM · Design
Andrew added a comment to T180837: Take down livingstyleguide.wmflabs.org.

@MarcoAurelio this appears to be a project of the design team, so I'm adding them and removing the cloud team. The site is hosted by WMCS but we don't have anything to do with decisions within the project.

Mon, Nov 20, 12:57 AM · Design

Nov 8 2017

phuedx awarded Blog Post: Automated OpenStack Testing, now with charts and graphs a Like token.
Nov 8 2017, 12:28 PM · cloud-services-team

Oct 30 2017

Andrew added a comment to T178508: Update VPS puppetmasters to 4.8 or newer.

Update: I don't think our custom hiera lookup is working properly with the 4.8 master. That's the next thing to understand/fix.

Oct 30 2017, 9:28 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a comment to T178508: Update VPS puppetmasters to 4.8 or newer.

I have a 4.8 puppetmaster running on andrew-test-master-upgrade.puppet.eqiad.wmflabs. Things I've tested with it are:

Oct 30 2017, 3:19 PM · Patch-For-Review, cloud-services-team (Kanban)

Oct 27 2017

Andrew closed T177944: k8s nodes sometimes getting bad token value from hiera as Resolved.
Oct 27 2017, 6:03 PM · Patch-For-Review, Toolforge

Oct 26 2017

Andrew closed T178752: gblrename log_type missing on replicas as Resolved.
Oct 26 2017, 9:19 PM · cloud-services-team (Kanban), Patch-For-Review, Security-Team, Data-Services

Oct 25 2017

Andrew added a parent task for T178508: Update VPS puppetmasters to 4.8 or newer: T177254: Upgrade to puppet 4 (4.8 or newer).
Oct 25 2017, 3:27 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew added a subtask for T177254: Upgrade to puppet 4 (4.8 or newer): T178508: Update VPS puppetmasters to 4.8 or newer.
Oct 25 2017, 3:27 PM · Patch-For-Review, cloud-services-team (FY2017-18), Puppet, User-Joe, Operations

Oct 24 2017

Andrew added a comment to T178332: Request increased quota for cyberbot Cloud VPS project.

I've added the 'bigdisk' flavor to this project, so now when you create a new VM you should see that as an option. Let me know if you run into any trouble.

Oct 24 2017, 4:16 PM · Cloud-VPS (Quota-requests)

Oct 23 2017

Andrew added a comment to T178841: Beta cluster is down.

I've only looked at deployment-cache-text04. Puppet had been broken there for ages due to several puppet changes.

Oct 23 2017, 10:05 PM · Release-Engineering-Team (Kanban), User-Ryasmeen, User-greg, Patch-For-Review, Traffic, Operations, Beta-Cluster-Infrastructure

Oct 22 2017

Andrew added a comment to T177603: Proposal: Improvements for the Toolforge 'webservice' command.

@Sowjanyavemuri, the Outreachy coordinators that I've spoken to are firm that your schedule does not qualify you for an internship for this cycle. I encourage you to re-apply for the May program which should fit your schedule better.

Oct 22 2017, 3:51 PM · Cloud-Services, Outreachy (Round-15)

Oct 20 2017

Andrew added a parent task for T177254: Upgrade to puppet 4 (4.8 or newer): T178717: Upgrade wmcs instances and masters to puppet 4.8.
Oct 20 2017, 7:13 PM · Patch-For-Review, cloud-services-team (FY2017-18), Puppet, User-Joe, Operations
Andrew added a subtask for T178717: Upgrade wmcs instances and masters to puppet 4.8: T177254: Upgrade to puppet 4 (4.8 or newer).
Oct 20 2017, 7:13 PM · Patch-For-Review, Cloud-Services
Andrew created T178717: Upgrade wmcs instances and masters to puppet 4.8.
Oct 20 2017, 7:12 PM · Patch-For-Review, Cloud-Services
Andrew added a comment to T171473: labvirt1015 crashes.

yeah, I see that they are failed again. I don't know why...i tried swapping another spare from the decom ms-be servers and it lights up but still shows failed. The R720's have a history of killing off a PSU and every subsequent PSU you replace it with. These are out of warranty and probably should should be replaced.

Oct 20 2017, 3:34 PM · cloud-services-team (Kanban), DC-Ops, Operations, ops-eqiad
Andrew added a comment to T171473: labvirt1015 crashes.

I installed 20 VMs, and ran stress-ng on each of them, like this:

Oct 20 2017, 3:33 PM · cloud-services-team (Kanban), DC-Ops, Operations, ops-eqiad
Andrew added a comment to T177603: Proposal: Improvements for the Toolforge 'webservice' command.

Hello @Sowjanyavemuri! I'm doubt that we'll be able to bend the scheduling rules although I agree that the numbers are close. For number-crunching purposes, can you please answer each of the questions specifically? I know that these are answered in bits and pieces above but I'm a bit lost in the back-scroll.

Oct 20 2017, 3:00 PM · Cloud-Services, Outreachy (Round-15)

Oct 19 2017

Andrew created T178625: puppetdb failures.
Oct 19 2017, 9:34 PM · Puppet, Operations
Andrew added a comment to T178332: Request increased quota for cyberbot Cloud VPS project.

@Cyberpower678 Can you provide any kind of long-term estimate about how big you expect this database to grow?

Oct 19 2017, 4:57 PM · Cloud-VPS (Quota-requests)

Oct 18 2017

Andrew added a comment to T171473: labvirt1015 crashes.

I think the VM creation failure was a (mostly? completely?) unrelated issue. I've rescheduled some actually running VMs there, and will see how they do.

Oct 18 2017, 7:43 PM · cloud-services-team (Kanban), DC-Ops, Operations, ops-eqiad
Andrew updated subscribers of T178510: Upgrade puppetmaster on toolsbeta and test.

cc: everyone who has been active in the toolsbeta project

Oct 18 2017, 3:53 PM · cloud-services-team
Andrew edited projects for T178508: Update VPS puppetmasters to 4.8 or newer, added: cloud-services-team (Kanban); removed cloud-services-team.
Oct 18 2017, 3:46 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew created T178510: Upgrade puppetmaster on toolsbeta and test.
Oct 18 2017, 3:45 PM · cloud-services-team
Andrew added a comment to T178508: Update VPS puppetmasters to 4.8 or newer.

The future parser has few complaints, so we're ready to move on to actual upgrade testing.

Oct 18 2017, 3:44 PM · Patch-For-Review, cloud-services-team (Kanban)
Andrew created T178508: Update VPS puppetmasters to 4.8 or newer.
Oct 18 2017, 3:42 PM · Patch-For-Review, cloud-services-team (Kanban)

Oct 17 2017

Andrew added a comment to T177944: k8s nodes sometimes getting bad token value from hiera.

Current theory is that this happens when the labs-private repo is in the process of being rebased.

Oct 17 2017, 9:29 PM · Patch-For-Review, Toolforge

Oct 16 2017

Andrew closed T178052: pagetranslation log_type missing on replicas as Resolved.

All set.

Oct 16 2017, 3:18 PM · Patch-For-Review, Security-Team, DBA, Data-Services

Oct 12 2017

Andrew added a comment to T97081: toolsbeta: set up puppet-compiler / temporary-apply.

Here's my latest attempt to describe what works. Once the concerned patches are merged I'll try to get this down on wikitech someplace.

Oct 12 2017, 9:01 PM · Patch-For-Review, cloud-services-team (Kanban), puppet-compiler, Toolforge
Andrew closed T178082: Shutdown ocg Cloud VPS project? as Resolved.

Horizon can't quite delete everything yet, so I generally delete everything that Horizon can see first and then use the 'delete' link in wikitech. Horizon is /close/ to being able to do everything but it needs a bit of work.

Oct 12 2017, 5:57 PM · OfflineContentGenerator, Cloud-VPS
Andrew added a comment to T178052: pagetranslation log_type missing on replicas.

It looks to me like this is filter in maintain-views.yaml via logging_whitelist. However, I don't see that 'pagetranslation' has ever been in that list (or at least not since 2016-10-12 which is when the history becomes murky.)

Oct 12 2017, 2:14 PM · Patch-For-Review, Security-Team, DBA, Data-Services
Andrew triaged T177855: Difficulty applying profile class parameters in Horizon interface as Normal priority.
Oct 12 2017, 2:06 PM · cloud-services-team (Kanban), Horizon

Oct 11 2017

Andrew created T177959: Should VPS puppetmasters include labs-ns0/ns-1 in their resolv.confs?.
Oct 11 2017, 3:53 PM · cloud-services-team (Kanban)
Andrew closed T177944: k8s nodes sometimes getting bad token value from hiera as Resolved.

When I refreshed puppet on the affected host, it included this diff:

Oct 11 2017, 3:23 PM · Patch-For-Review, Toolforge

Oct 10 2017

Andrew created T177880: Automatically run maintain-views and and maintain-meta_p when config changes on cloud replicas.
Oct 10 2017, 8:21 PM · cloud-services-team (Kanban), Data-Services
Andrew added a comment to T167114: Open view for term_full_entity_id in wb_terms table in labs.

ok -- I was expecting this table to be present in enwiki. If it's wikidata-specific then we're probably done. @Ladsgroup can you confirm?

Oct 10 2017, 7:34 PM · cloud-services-team (Kanban), Data-Services, User-Ladsgroup, Wikidata-Former-Sprint-Board, Wikidata
Andrew updated subscribers of T167114: Open view for term_full_entity_id in wb_terms table in labs.

I've run maintain-views, but the wb_terms table isn't getting replicated at all. I don't see any evidence of filtering in the sanitarium files but I may be looking in the wrong place... @Marostegui, any ideas?

Oct 10 2017, 7:02 PM · cloud-services-team (Kanban), Data-Services, User-Ladsgroup, Wikidata-Former-Sprint-Board, Wikidata
Andrew added a comment to T177299: Revert temporary increase of floating-ip quota for 'cvn' project (trusty to debian migration).

ok! I've raised the quota to 4 IPs. Lets' leave this task open and you can nudge me when you're ready to clean up.

Oct 10 2017, 5:40 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Andrew renamed T177299: Revert temporary increase of floating-ip quota for 'cvn' project (trusty to debian migration) from Temporary increase of floating-ip quota for 'cvn' project (trusty to debian migration) to Revert temporary increase of floating-ip quota for 'cvn' project (trusty to debian migration).
Oct 10 2017, 5:39 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Andrew closed T177500: Request creation of MWStake VPS project as Resolved.

We don't support CamelCase in project names, so I've created a project called 'mwstake'. @MarkAHershberger is a project admin and can add other users or admins as needed.

Oct 10 2017, 5:37 PM · Cloud-VPS (Project-requests)
Andrew claimed T177299: Revert temporary increase of floating-ip quota for 'cvn' project (trusty to debian migration).
Oct 10 2017, 3:40 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Andrew added a comment to T177299: Revert temporary increase of floating-ip quota for 'cvn' project (trusty to debian migration).

Do you already have ram/CPU quota to create the additional instances? Is it really just the IPs that are holding you back?

Oct 10 2017, 3:39 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Andrew claimed T177500: Request creation of MWStake VPS project.

Approved, will do shortly

Oct 10 2017, 3:37 PM · Cloud-VPS (Project-requests)
Andrew created T177851: Page if fullstack test fails more than once in a row.
Oct 10 2017, 3:35 PM · cloud-services-team (Kanban)
Andrew created T177850: Page if the grid engine master is unreachable.
Oct 10 2017, 3:35 PM · monitoring, Toolforge, cloud-services-team (Kanban)
Andrew closed T177834: Wikimedia Cloud (labs) dns is intermittingly failing as Resolved.

This seems to have been caused by https://gerrit.wikimedia.org/r/#/c/382415/, which has now been reverted.

Oct 10 2017, 2:03 PM · Operations, Cloud-Services

Oct 9 2017

Andrew added a comment to T177427: Remove non-interactive bots from #wikimedia-cloud.

I don't mind the idea of unbreak now still showing up via wikibugs. [in the main channel]

Is that possible? +1 if so!

Oct 9 2017, 5:57 PM · Patch-For-Review, Cloud-Services, cloud-services-team (Kanban)

Oct 6 2017

Andrew added a comment to T97081: toolsbeta: set up puppet-compiler / temporary-apply.

I'm trying to reproduce the tools puppet compiler described here. A few things have clearly changed since this was last built... The hiera setup I seem to need looks like this:

Oct 6 2017, 6:21 PM · Patch-For-Review, cloud-services-team (Kanban), puppet-compiler, Toolforge

Oct 5 2017

Andrew added a comment to T177450: Not all content is getting replicated to wikitech-static.

I ran the export and import by hand just now, and I think we're getting the complete wiki.

Oct 5 2017, 8:32 PM · cloud-services-team (Kanban), wikitech.wikimedia.org
Andrew added a comment to T177427: Remove non-interactive bots from #wikimedia-cloud.

I've directed shinken-wm to talk in #wikimedia-cloud-feed.

Oct 5 2017, 8:11 PM · Patch-For-Review, Cloud-Services, cloud-services-team (Kanban)
mmodell awarded T177427: Remove non-interactive bots from #wikimedia-cloud a Like token.
Oct 5 2017, 7:34 PM · Patch-For-Review, Cloud-Services, cloud-services-team (Kanban)
Andrew renamed T167973: Move wikitech and labstestwiki to s3 from move wikitech and labstestwiki to s3 (needs discussion) to move wikitech and labstestwiki to s3.
Oct 5 2017, 4:03 PM · Data-Services, wikitech.wikimedia.org, cloud-services-team, DBA
Andrew added a comment to T177450: Not all content is getting replicated to wikitech-static.

A --current dump is 8.6M, a --full dump is 7.2G. So doing --full may not be practical.

Oct 5 2017, 3:56 PM · cloud-services-team (Kanban), wikitech.wikimedia.org
Andrew closed T176090: wikitech-static sync failing as Resolved.
Oct 5 2017, 3:55 PM · MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), MediaWiki-Maintenance-scripts, Operations
Andrew added a comment to T167973: Move wikitech and labstestwiki to s3.

Is there anything I can do to nudge this along, short of 'clone Jaime'?

Oct 5 2017, 2:45 PM · Data-Services, wikitech.wikimedia.org, cloud-services-team, DBA
Andrew closed T167820: rack/setup/install labweb100[12].wikimedia.org as Resolved.

These boxes are up and installed and seem ok. Actual service implementation is T168470

Oct 5 2017, 2:34 PM · Patch-For-Review, Operations, Cloud-Services
Andrew added a comment to T176757: CamelCase vs. VPS instance naming.

Adding a regex validation to the instance name in Horizon turns out to be non-trivial in the current version.

Oct 5 2017, 2:33 PM · cloud-services-team (Kanban)
Andrew closed T170492: figure out if nodepool is overwhelming rabbitmq and/or nova as Resolved.

rabbit is now much quieter, so this is /maybe/ better. Closing for now, optimistically.

Oct 5 2017, 2:32 PM · cloud-services-team (Kanban), Release-Engineering-Team (Watching / External), Nodepool, Cloud-VPS, Continuous-Integration-Infrastructure, Patch-For-Review
Andrew added a comment to T177450: Not all content is getting replicated to wikitech-static.

Wikitech is dumped using

Oct 5 2017, 12:29 PM · cloud-services-team (Kanban), wikitech.wikimedia.org
Andrew created T177450: Not all content is getting replicated to wikitech-static.
Oct 5 2017, 12:11 AM · cloud-services-team (Kanban), wikitech.wikimedia.org