Page MenuHomePhabricator

Andrew (Andrew Bogott)
User

Projects (10)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2014, 11:35 PM (245 w, 1 d)
Availability
Available
IRC Nick
andrewbogott
LDAP User
Unknown
MediaWiki User
Andrewbogott [ Global Accounts ]

Recent Activity

Yesterday

Andrew added a comment to T228056: Puppet times out on newly created instance in the puppet-diffs project.

So... @hashar is the work around for this just to wait a while after building a new compiler node? Or is that not adequate?

Mon, Jul 15, 9:19 PM · Cloud-VPS, cloud-services-team
Andrew added a comment to T228056: Puppet times out on newly created instance in the puppet-diffs project.

disappointingly, fixing that mistaken newline issue doesn't resolve anything

Mon, Jul 15, 9:07 PM · Cloud-VPS, cloud-services-team
Andrew added a comment to T228056: Puppet times out on newly created instance in the puppet-diffs project.

Thanks to an extremely tedious binary search, I've determined that this is not a puppetmaster issue. It correlates with this hiera setting:

Mon, Jul 15, 8:59 PM · Cloud-VPS, cloud-services-team
Andrew added a comment to T228056: Puppet times out on newly created instance in the puppet-diffs project.

Note that puppet installs a bunch of files happily before getting to this point. So it's not a total failure of file-serving, it's something specific.

Mon, Jul 15, 3:11 PM · Cloud-VPS, cloud-services-team
Andrew renamed T228056: Puppet times out on newly created instance in the puppet-diffs project from Puppet times out on newly created instance to Puppet times out on newly created instance in the puppet-diffs project.
Mon, Jul 15, 3:03 PM · Cloud-VPS, cloud-services-team

Fri, Jul 12

Andrew updated subscribers of T227918: labpuppetmaster1001/1002 complaining about unmerged patches.

Since there have been some other unexpected monitoring issues today related to monitoring refactors, pinging @fgiunchedi before I dig in too much.

Fri, Jul 12, 8:15 PM · observability, cloud-services-team (Kanban)
Andrew created T227918: labpuppetmaster1001/1002 complaining about unmerged patches.
Fri, Jul 12, 8:14 PM · observability, cloud-services-team (Kanban)
Andrew closed T226415: wmcs-cold-migrate doesn't completely clean up local VMs as Resolved.

Thanks Jason!

Fri, Jul 12, 6:42 PM · cloud-services-team (Kanban)

Thu, Jul 11

Andrew added a comment to T227830: alert when a script fired by a systemd timer hangs.

<mutante> JobTimeoutSec=, JobRunningTimeoutSec=
5:40 PM when i asked if it looks at return codes i was told "what if it's still running". but this ^
5:40 PM and then there is "write another timer that is scheduled to run at the shutdown times you would like. They can run" a service which stops the you want to top, by running ExecStart=/bin/systemctl stop other.service in the service file called your shutdown timer"
5:42 PM but it if it times out that also does not mean it goes into "failed" state. "When a job for this unit is queued, a timeout JobTimeoutSec= may be configured. Similarly, JobRunningTimeoutSec= starts counting when the queued job is actually started. If either time limit is reached, the job will be cancelled, the unit however will not change state or even enter the "failed" mode. "
5:43 PM ah, you can kill it with "JobTimeoutAction= optionally configures an additional action to take when the timeout is hit, "
5:44 PM so you could have a script that does both, kill the process and tell monitoring about it and have that as your TimeoutAction command

Thu, Jul 11, 10:53 PM · Patch-For-Review, observability, cloud-services-team (Kanban)
Andrew added projects to T227830: alert when a script fired by a systemd timer hangs: cloud-services-team (Kanban), observability.
Thu, Jul 11, 10:29 PM · Patch-For-Review, observability, cloud-services-team (Kanban)
Andrew updated the task description for T227830: alert when a script fired by a systemd timer hangs.
Thu, Jul 11, 10:28 PM · Patch-For-Review, observability, cloud-services-team (Kanban)
Andrew created T227830: alert when a script fired by a systemd timer hangs.
Thu, Jul 11, 10:28 PM · Patch-For-Review, observability, cloud-services-team (Kanban)
Andrew added a comment to T227785: wmcs-dns-floating-ip-updater (and other scripts using mwopenstackclients + designate) have been failing.

Regarding the failure to alert:

Thu, Jul 11, 10:06 PM · Patch-For-Review
Andrew moved T227600: Make graph-tool available on tool forge from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
Thu, Jul 11, 7:37 PM · cloud-services-team (Kanban), Toolforge
Andrew added a project to T227600: Make graph-tool available on tool forge: cloud-services-team (Kanban).
Thu, Jul 11, 7:37 PM · cloud-services-team (Kanban), Toolforge
Andrew closed T227716: Drop DB references from WMCS indexes and replicas for now-deleted zerowiki, if appropriate, a subtask of T187716: Sunset Wikipedia Zero, as Resolved.
Thu, Jul 11, 7:28 PM · MW-1.34-notes (1.34.0-wmf.14; 2019-07-16), Release-Engineering-Team-TODO (201907), Epic, Reading-Infrastructure-Team-Backlog, Wikimedia-Site-requests
Andrew closed T227716: Drop DB references from WMCS indexes and replicas for now-deleted zerowiki, if appropriate as Resolved.

There's an alias for zerowiki, which suggests that it's replicated. There's no actual database of that name present on the replicas, though, so maybe it was never actually set up. In any case, I'm running the cleanup steps.

Thu, Jul 11, 7:28 PM · Cloud-Services, Release-Engineering-Team-TODO
Andrew created T227785: wmcs-dns-floating-ip-updater (and other scripts using mwopenstackclients + designate) have been failing.
Thu, Jul 11, 3:13 PM · Patch-For-Review

Wed, Jul 10

Andrew closed T227474: New (public) buster base image on clouvps as Resolved.
Wed, Jul 10, 6:29 PM · cloud-services-team (Kanban)
Andrew closed T227475: Use sssd by default in cloud-vps buster base images, a subtask of T227474: New (public) buster base image on clouvps, as Resolved.
Wed, Jul 10, 6:29 PM · cloud-services-team (Kanban)
Andrew closed T227475: Use sssd by default in cloud-vps buster base images as Resolved.
Wed, Jul 10, 6:29 PM · cloud-services-team (Kanban)

Tue, Jul 9

Andrew added a comment to T227377: Request creation of Linkwatcher and COIBot VPS project.

A VM of this size will be quite difficult for us to manage -- among other things, it would take many hours to move off of a hypervisor. Generally when we create large VMs (although so far we have never created one of this size) it's with the understanding that I may need to delete it as part of routine maintenance and leave it to be rebuilt by the users.

Tue, Jul 9, 8:32 PM · Cloud-VPS (Project-requests)
Andrew renamed T227475: Use sssd by default in cloud-vps buster base images from Use sssd by default in cloud-vps base images to Use sssd by default in cloud-vps buster base images.
Tue, Jul 9, 2:13 PM · cloud-services-team (Kanban)

Mon, Jul 8

Andrew updated the task description for T210850: WMCS-related dashboards using Diamond metrics.
Mon, Jul 8, 9:08 PM · cloud-services-team (Kanban), Operations
Andrew updated the task description for T210850: WMCS-related dashboards using Diamond metrics.
Mon, Jul 8, 8:34 PM · cloud-services-team (Kanban), Operations
Andrew updated the task description for T210850: WMCS-related dashboards using Diamond metrics.
Mon, Jul 8, 8:30 PM · cloud-services-team (Kanban), Operations
Andrew added a comment to T214275: Deprecate the usage of nutcracker for memcached.

The latter should be doable, but the former seems a bit more complicated. Is there any plan to deprecate the labswiki infra and fold it into the appserver layer?

Mon, Jul 8, 2:35 PM · Wikimedia-General-or-Unknown, serviceops, Performance-Team (Radar), User-Elukey, Operations
Andrew created T227475: Use sssd by default in cloud-vps buster base images.
Mon, Jul 8, 12:51 PM · cloud-services-team (Kanban)
Andrew created T227474: New (public) buster base image on clouvps.
Mon, Jul 8, 12:50 PM · cloud-services-team (Kanban)

Thu, Jul 4

fgiunchedi awarded T210850: WMCS-related dashboards using Diamond metrics a Like token.
Thu, Jul 4, 8:14 AM · cloud-services-team (Kanban), Operations

Wed, Jul 3

Andrew added a comment to T210850: WMCS-related dashboards using Diamond metrics.

I've made a new dashboard, https://grafana.wikimedia.org/d/ebJoA6VWz/nova-fullstack -- once I'm convinced that it's doing what I expect I'll delete the older labs-nova-fullstack board.

Wed, Jul 3, 9:46 PM · cloud-services-team (Kanban), Operations

Tue, Jul 2

Andrew added a comment to T227041: Three small ganeti VMs to host haproxy for OpenStack endpoints.

How is corosync/pacemaker going to work then with a single VIP?

Tue, Jul 2, 10:11 PM · vm-requests, cloud-services-team (Kanban), Operations
Andrew closed T227105: puppet failures on labmon1001/1002 -- sqlite3-pcre as Resolved.

Thanks @CDanis

Tue, Jul 2, 6:07 PM · cloud-services-team (Kanban)
Andrew updated the task description for T227105: puppet failures on labmon1001/1002 -- sqlite3-pcre.
Tue, Jul 2, 6:01 PM · cloud-services-team (Kanban)
Andrew added a comment to T227105: puppet failures on labmon1001/1002 -- sqlite3-pcre.

Most likely from https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/520147/

Tue, Jul 2, 6:00 PM · cloud-services-team (Kanban)
Andrew closed T226534: Request creation of machine-vision VPS project as Resolved.

done!

Tue, Jul 2, 5:24 PM · Machine vision, cloud-services-team (Kanban), Reading-Infrastructure-Team-Backlog, Cloud-VPS (Project-requests)
Andrew added a comment to T227041: Three small ganeti VMs to host haproxy for OpenStack endpoints.

(Let's use Buster for this if it's available on ganeti)

Tue, Jul 2, 5:01 PM · vm-requests, cloud-services-team (Kanban), Operations
Andrew claimed T226534: Request creation of machine-vision VPS project.
Tue, Jul 2, 4:35 PM · Machine vision, cloud-services-team (Kanban), Reading-Infrastructure-Team-Backlog, Cloud-VPS (Project-requests)
Andrew added a comment to T226688: Block web crawlers from accessing Cloud Services.

I suggest that rather than a global block we think about this as a tooling issue -- maybe provide a default 'block everything' robots.txt (or even an actual service block) and a well-documented way for users to manage this.

Tue, Jul 2, 4:32 PM · cloud-services-team (Kanban), Data-Services, Toolforge
Andrew created T227105: puppet failures on labmon1001/1002 -- sqlite3-pcre.
Tue, Jul 2, 3:54 PM · cloud-services-team (Kanban)
Andrew added a comment to T227041: Three small ganeti VMs to host haproxy for OpenStack endpoints.

Sounds fine to me. Please use row_A in eqiad for this as it has more resources available. Also, I guess all three VMs will have to go on the same row anyway due to the requirement that all 3 nodes share the network.

Tue, Jul 2, 3:28 PM · vm-requests, cloud-services-team (Kanban), Operations

Mon, Jul 1

Andrew added a comment to T227029: Prevent catalog breakage on cloud instances by decoupling core cloud puppetmaster from custom puppetmasters.
  • What will this setup look like to the admins of a Cloud VPS instance? Is it all easy to find from /etc/puppet files or will there be instance based config put in some other location(s)?
Mon, Jul 1, 10:52 PM · Puppet, cloud-services-team (Kanban)
Andrew updated the task description for T227041: Three small ganeti VMs to host haproxy for OpenStack endpoints.
Mon, Jul 1, 10:22 PM · vm-requests, cloud-services-team (Kanban), Operations
Andrew added a comment to T227041: Three small ganeti VMs to host haproxy for OpenStack endpoints.

btw, I'm happy to actually set up the VMs, only assigning to Alex to approve the resource usage.

Mon, Jul 1, 10:17 PM · vm-requests, cloud-services-team (Kanban), Operations
Restricted Application added a project to T227041: Three small ganeti VMs to host haproxy for OpenStack endpoints: Operations.
Mon, Jul 1, 10:16 PM · vm-requests, cloud-services-team (Kanban), Operations
Andrew added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

If we want single-purpose proxies we can create ganeti VMs for that. I'm still trying to determine if we can have proper three-server redundancy that way...

Mon, Jul 1, 8:42 PM · cloud-services-team (Kanban)
Andrew added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

We can try to buy more hardware or we can just declare that our HA cluster is [cloudcontrol1003, cloudcontrol1004, cloudservices1003]. The puppet would be a bit ugly but they're all on public IPs.

Mon, Jul 1, 8:18 PM · cloud-services-team (Kanban)
Andrew added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

fwiw I'm totally down with deciding we need a third cloudcontrol. As I understand it it's only on the front-end proxy that we'd need three, not for each API backend right?

Mon, Jul 1, 8:13 PM · cloud-services-team (Kanban)
Andrew renamed T227029: Prevent catalog breakage on cloud instances by decoupling core cloud puppetmaster from custom puppetmasters from Decouple core cloud puppetmaster from custom puppetmasters to Prevent catalog breakage on cloud instances by decoupling core cloud puppetmaster from custom puppetmasters.
Mon, Jul 1, 8:01 PM · Puppet, cloud-services-team (Kanban)
Andrew updated subscribers of T227029: Prevent catalog breakage on cloud instances by decoupling core cloud puppetmaster from custom puppetmasters.
Mon, Jul 1, 7:58 PM · Puppet, cloud-services-team (Kanban)
Andrew created T227029: Prevent catalog breakage on cloud instances by decoupling core cloud puppetmaster from custom puppetmasters.
Mon, Jul 1, 7:56 PM · Puppet, cloud-services-team (Kanban)
Andrew added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

I have never used HAproxy, so there is no plan as yet -- that's up to you :)

Mon, Jul 1, 7:25 PM · cloud-services-team (Kanban)
Andrew added a comment to T223907: Set up HA endpoints for keystone, glance, nova, designate apis.

For the short term I've been assuming we'd just put a proxy in front of the existing endpoints. That means two each:

Mon, Jul 1, 7:03 PM · cloud-services-team (Kanban)

Fri, Jun 28

Andrew added a comment to T223906: Active/active rabbitMQ servers on wmcs controller nodes.

btw, if you update the live config please adjust the docs here, accordingly: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Rabbitmq

Fri, Jun 28, 9:13 PM · cloud-services-team (Kanban)
Andrew added a comment to T223906: Active/active rabbitMQ servers on wmcs controller nodes.

yep, I definitely just did what the HA guide said to do :/ If we can do active/active with two disk nodes that seems fine!

Fri, Jun 28, 9:13 PM · cloud-services-team (Kanban)

Thu, Jun 27

Andrew placed T223905: HA for openstack services up for grabs.
Thu, Jun 27, 5:14 PM · cloud-services-team (Kanban)
Andrew added a comment to T226731: Implement nova host-aggregates.

I'm certainly in favor of replacing custom code with upstream code! In particular it seems like we'll need this in order to make live-migration work sensibly between different CPU-typed cloudvirts, right? I do have a few concerns:

Thu, Jun 27, 4:49 PM · Cloud-VPS, cloud-services-team
Andrew closed T225484: cloudvirt servers: SSL certificate expiring as Resolved.

This is resolved -- we now use puppet certs rather than the libvirt* cert. I removed the monitor for the old cert.

Thu, Jun 27, 3:42 PM · cloud-services-team (Kanban)

Wed, Jun 26

Andrew claimed T225484: cloudvirt servers: SSL certificate expiring.

Note that ever since https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/418945/, hypervisor<->hypervisor communication has been broken so these certs are moot (we really need libvirtd -l for that). I'm going to try to fix the certs anyway because we'll want them working to get live migration going eventually, but at present this is a very low-stakes issue.

Wed, Jun 26, 11:56 PM · cloud-services-team (Kanban)
Andrew created T226647: nova-fullstack crashed from a keystone timeout.
Wed, Jun 26, 3:25 PM · cloud-services-team (Kanban)
Andrew closed T226632: cloudvps: capacity -- add cloudvirt1030 to scheduling pool or change documentation as Resolved.

thanks!

Wed, Jun 26, 3:05 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Tue, Jun 25

Andrew added a comment to T225484: cloudvirt servers: SSL certificate expiring.

We could use puppet certs if we use a SAN.
Theoretically it should be pretty straight forward, since it only requires a hiera key per host (or per role):

profile::base::puppet::dns_alt_names: cloudvirt*.eqiad.wmnet
Tue, Jun 25, 8:45 PM · cloud-services-team (Kanban)
Andrew assigned T225823: Request creation of asyncwiki VPS project to JHedden.
Tue, Jun 25, 4:19 PM · Cloud-VPS (Project-requests)
Andrew added a comment to T225823: Request creation of asyncwiki VPS project.

Approved! We'll create this shortly.

Tue, Jun 25, 4:19 PM · Cloud-VPS (Project-requests)
Andrew merged T226518: labvirt-star.eqiad.wmnet.crt about to expire into T225484: cloudvirt servers: SSL certificate expiring.
Tue, Jun 25, 3:14 PM · cloud-services-team (Kanban)
Andrew merged task T226518: labvirt-star.eqiad.wmnet.crt about to expire into T225484: cloudvirt servers: SSL certificate expiring.
Tue, Jun 25, 3:14 PM
Andrew created T226518: labvirt-star.eqiad.wmnet.crt about to expire.
Tue, Jun 25, 3:12 PM

Mon, Jun 24

Andrew created P8649 libvirt failure on cloudvirt1015.
Mon, Jun 24, 6:32 PM
Andrew closed T201247: Sporadic puppet failures as Resolved.

Haven't seen this in ages.

Mon, Jun 24, 4:06 PM · cloud-services-team (Kanban), Operations
Andrew triaged T226415: wmcs-cold-migrate doesn't completely clean up local VMs as Low priority.
Mon, Jun 24, 3:27 PM · cloud-services-team (Kanban)
Andrew created T226415: wmcs-cold-migrate doesn't completely clean up local VMs.
Mon, Jun 24, 3:27 PM · cloud-services-team (Kanban)

Fri, Jun 21

Andrew added a comment to T226270: Reduce the effects of puppet breakage on VPS.

I have an (ironically) unpuppetized example of a dual-run setup running now:

Fri, Jun 21, 8:10 PM · Puppet, cloud-services-team (Kanban)
Andrew added a comment to T226270: Reduce the effects of puppet breakage on VPS.

And, here is some info about the base classes applied to a VM vs a production machine:

Fri, Jun 21, 3:57 PM · Puppet, cloud-services-team (Kanban)
Andrew created P8639 Puppet base role on VPS vs. production.
Fri, Jun 21, 3:56 PM
Andrew added a comment to T226270: Reduce the effects of puppet breakage on VPS.

Here are some usage stats:

Fri, Jun 21, 3:56 PM · Puppet, cloud-services-team (Kanban)
Andrew created P8638 Puppet customization of VPS instances.
Fri, Jun 21, 3:54 PM
Andrew created T226270: Reduce the effects of puppet breakage on VPS.
Fri, Jun 21, 3:53 PM · Puppet, cloud-services-team (Kanban)

Thu, Jun 20

Andrew closed T225025: Request new Flavor for integration Cloud VPS project as Resolved.

I added a new flavor named 'mediumram' to the integration project. Thanks for conserving RAM!

Thu, Jun 20, 4:49 PM · cloud-services-team (Kanban), Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, Cloud-VPS (Quota-requests)
Andrew assigned T226188: relocate/reimage cloudvirt1014 with 10G interfaces to Cmjohnson.
Thu, Jun 20, 3:11 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Thu, Jun 20, 3:10 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew updated the task description for T216195: Move cloudvirt hosts to 10Gb ethernet.
Thu, Jun 20, 2:21 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Andrew created T226188: relocate/reimage cloudvirt1014 with 10G interfaces.
Thu, Jun 20, 2:21 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)

Tue, Jun 18

Andrew added a comment to T166337: wsexport tool leaking files in /tmp.

There are files in /tmp/calibre_2.75.1_tmp_imITvt on tools-sgewebgrid-lighttpd-0921.tools.eqiad.wmflabs but it looks like they're being actively created/deleted so that's maybe fine... I don't see /tmp litter on other hosts.

Tue, Jun 18, 1:46 PM · Community-Tech (Resolved 2018-19 Q4), E-Book-Export-Reliability, Tools, Toolforge

Mon, Jun 17

Andrew removed a member for acl*sre-team: GTirloni.
Mon, Jun 17, 3:59 PM
Andrew added a member for acl*sre-team: JHedden.
Mon, Jun 17, 3:58 PM
Andrew placed T225932: support ssl for openstack REST endpoints up for grabs.
Mon, Jun 17, 1:05 PM · cloud-services-team (Kanban)
Andrew added a comment to T217474: labstore1006 nfsd not started after reboot.

Just to double-check: IIRC, back in the day we avoided this because we had multiple controllers attached to a shared shelf and if two controllers ran at the same time then terrible, terrible things happened. Is it safe to say that there's no current situation where having 'too many' nfs services running at once causes harm?

Mon, Jun 17, 12:45 PM · Patch-For-Review, Data-Services, observability, cloud-services-team (Kanban)
Andrew triaged T225932: support ssl for openstack REST endpoints as Normal priority.
Mon, Jun 17, 12:34 PM · cloud-services-team (Kanban)
Andrew created T225932: support ssl for openstack REST endpoints.
Mon, Jun 17, 12:30 PM · cloud-services-team (Kanban)
Andrew updated the task description for T223905: HA for openstack services.
Mon, Jun 17, 12:27 PM · cloud-services-team (Kanban)
Andrew updated the task description for T223905: HA for openstack services.
Mon, Jun 17, 12:27 PM · cloud-services-team (Kanban)
Andrew renamed T223907: Set up HA endpoints for keystone, glance, nova, designate apis from Set up HA for keystone, glance, nova, designate apis to Set up HA endpoints for keystone, glance, nova, designate apis.
Mon, Jun 17, 12:26 PM · cloud-services-team (Kanban)
Andrew added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

The remaining task here is to make/update a wiki page about this.

Mon, Jun 17, 12:25 PM · Operations, Traffic, Cloud-VPS, cloud-services-team (Kanban)
Andrew added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

We had a session about this during the SRE summit. The conclusions were:

Mon, Jun 17, 12:25 PM · Operations, Traffic, Cloud-VPS, cloud-services-team (Kanban)
Andrew renamed T223907: Set up HA endpoints for keystone, glance, nova, designate apis from Set up LVS for keystone, glance, nova apis to Set up HA for keystone, glance, nova, designate apis.
Mon, Jun 17, 12:23 PM · cloud-services-team (Kanban)

Sun, Jun 16

Andrew added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

I'm put eight test VMs on 1015, will let them run for a few days and then see if they're still up :)

Sun, Jun 16, 4:06 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)

Jun 13 2019

Andrew awarded T204840: wikitech-static: not synced a Party Time token.
Jun 13 2019, 12:35 PM · cloud-services-team (Kanban), wikitech.wikimedia.org

Jun 7 2019

Andrew added a comment to T225312: role::wikibase in wikidata-dev Cloud VPS project broken (⇒ can’t SSH into wikibase-* instances).

I wouldn't say that the role is necessarily broken; it may just be that it needs the hiera args provided if the class is applied. You could provide a good default in the git puppet tree, though, if there is a good universal default. Easier would be to just add it to the project-wide hiera setting on Horizon.

Jun 7 2019, 4:48 PM · Puppet, Cloud-VPS, Wikidata
Andrew added a comment to T204840: wikitech-static: not synced.

Some extensions were not bundled with the branch and so the usual 'check them out as submodules' trick did not work for
updating them; they had to be cloned manually. @Andrew do you want to add a description of what you did?

Jun 7 2019, 4:17 PM · cloud-services-team (Kanban), wikitech.wikimedia.org

Jun 6 2019

Andrew created T225258: Get live hacks on wikitech-static into git.
Jun 6 2019, 10:26 PM · wikitech.wikimedia.org