Krenair (Alex Monk)
Wikimedia volunteer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:34 PM (176 w, 3 d)
Availability
Available
IRC Nick
Krenair
LDAP User
Alex Monk
MediaWiki User
Krenair

I am a Wikimedia volunteer helping in various technical ways. These days it's usually Beta cluster related. I've previously spent significant amounts of time involved in MediaWiki development, software deployments to the Wikimedia cluster, and various other things. I am also an OTRS agent.

Some of my old VisualEditor work can be found under @AlexMonk-WMF instead

I have opinions on things, which do not necessarily represent those of any organisation I am, have previously been, or will in the future be affiliated with.

Recent Activity

Yesterday

Krenair awarded T187716: Undeploy Zero extensions and kill zerowiki a Goat token.
Mon, Feb 19, 7:36 PM · Wikimedia-Site-requests

Fri, Feb 16

Krenair added a comment to T186247: Hebrew Wikivoyage (via Tool "wikivoyage") loads assets by default from third-party sites.

Am I missing something here or has something gone wrong in the handling of this ticket? How was it known for two weeks but stayed online?

Fri, Feb 16, 11:50 PM · Collaboration-Team-Triage (Collab-Team-This-Quarter), Vuln-Infoleak, Community-Liaisons, Collaboration-Feature-Rollouts (Collaboration-Maps), Discovery, Privacy, Toolforge-standards-committee, Maps, WMF-Legal, Tools

Sun, Feb 11

Krenair added a comment to T186675: Add 'centralauth' to meta_p.wiki so that apps can re-use the appropriate slice.

I would suggest splitting the table into two: one that has a list of wikis (in the current setup all columns except slice) and and one that lists databases and their properties (dbname, slice). The original wiki table can then be replaced with a view that performs the relevant join to maintain backwards compatibility.

Sun, Feb 11, 11:16 PM · Toolforge, Data-Services, cloud-services-team

Mon, Feb 5

Krenair added a comment to T186415: Create trusted group in gerrit.

But we can block it for all users right?

Mon, Feb 5, 7:47 PM · Developer-Relations, Gerrit
Krenair added a comment to T186415: Create trusted group in gerrit.

Especially with the things I've heard about it, enabling private changes
would be a bad idea, regardless of whether they can be used for security
patches or not

Mon, Feb 5, 7:42 PM · Developer-Relations, Gerrit

Wed, Jan 31

Krenair added a comment to T186133: Login session bug on Beta Commons.

Session unreliability is a long-standing issue on beta (see e.g. T172560: "Loss of session data" on Beta Cluster; there was another task that I can't find now where @Krenair tracked it down to redis replication failures)

Wed, Jan 31, 7:33 PM · Beta-Cluster-reproducible, Beta-Cluster-Infrastructure

Wed, Jan 24

Krenair added a project to T185670: Request for allowance of multiple account registers from same IP for 2018-01-25 14:00UTC: Wikimedia-Site-requests.

I wouldn't count on it seeing how last-minute this request is. You'll need to provide the list of wikis to allow it on, and preferably a link to info about the event.

Wed, Jan 24, 11:56 PM · Wikimedia-Site-requests
Krenair added a comment to T185606: wikistream.wmflabs.org 502 Varnish Error.

I won't be able to SSH to that as I am not a project member.

Wed, Jan 24, 8:59 PM · Cloud-VPS

Tue, Jan 23

Krenair edited projects for T185606: wikistream.wmflabs.org 502 Varnish Error, added: Cloud-VPS; removed Tools.

http://tools.wmflabs.org/openstack-browser/proxy/ shows that it goes to ws-web.wikistream.eqiad.wmflabs on port 80
The error message I get doesn't reference Varnish, it is from an nginx instance.
Do you perhaps have nginx running on ws-web? When you say 'I can see that the service itself is running fine', what service is that?

Tue, Jan 23, 10:58 PM · Cloud-VPS
Krenair added a comment to T185319: IRC RecentChanges feed: code stewardship request.

Pretty sure MW has supported having multiple destinations for these streams for years now. So you could have multiple servers receiving the changes from MW and being available for clients to connect to.

Tue, Jan 23, 2:01 AM · Tools, Operations, Analytics, Wikimedia-IRC-RC-Server, Code-Stewardship-Reviews

Jan 20 2018

Krenair added a comment to T184478: Puppet broken on deployment-ores01 due to missing hieradata.

So you hold your resolved tasks open for up to a month after resolution? But yeah, this is very much a Beta-Cluster-Infrastructure task.

Jan 20 2018, 4:44 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184478: Puppet broken on deployment-ores01 due to missing hieradata.

What do you mean 'before reporting them'?

Jan 20 2018, 2:31 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure

Jan 19 2018

Krenair closed T173554: Puppet broken on deployment-sentry01 as Resolved.

This took a frankly ridiculous amount of time to solve considering how simple the problem and patch was.

Jan 19 2018, 9:38 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair closed T173554: Puppet broken on deployment-sentry01, a subtask of T132259: Deployment-prep hosts with puppet errors (tracking), as Resolved.
Jan 19 2018, 9:38 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair closed T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing as Resolved.
Jan 19 2018, 9:36 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair closed T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing, a subtask of T132259: Deployment-prep hosts with puppet errors (tracking), as Resolved.
Jan 19 2018, 9:36 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair added a comment to T184478: Puppet broken on deployment-ores01 due to missing hieradata.

Looks like its fixed now? Wanna mark this as resolved?

Jan 19 2018, 9:31 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure

Jan 17 2018

Krenair awarded T184230: Disavow emails from wikipedia.com a Burninate token.
Jan 17 2018, 10:27 PM · Patch-For-Review, Operations, Mail

Jan 16 2018

Krenair added a comment to T185028: Beta cluster login broken.

Reproduced bug on enwiki

Jan 16 2018, 7:41 PM · MinervaNeue, Readers-Web-Backlog, Beta-Cluster-Infrastructure
Krenair renamed T185028: Beta cluster login broken from Unable to login commons wmflabs to Beta cluster login broken.
Jan 16 2018, 7:41 PM · MinervaNeue, Readers-Web-Backlog, Beta-Cluster-Infrastructure

Jan 15 2018

Krenair added a project to T184957: en:wikiversity Draft Namespace: Wikimedia-Site-requests.
Jan 15 2018, 10:45 PM · Patch-For-Review, User-Jayprakash12345, Wikimedia-Site-requests
Krenair added a comment to T167060: en.wiki domain owned by us, but isn't hosted by us??.

wikibooks.wiki too - https://meta.wikimedia.org/wiki/Requests_for_comment/Domain_parking

Jan 15 2018, 9:51 PM · WMF-Legal, Privacy, Domains, Operations, DNS, Traffic

Jan 11 2018

Krenair updated subscribers of T184234: Puppet broken on deployment-cache-text04 due to varnishkafka issues.
Jan 11 2018, 12:34 AM · Puppet, Beta-Cluster-Infrastructure

Jan 10 2018

Krenair added a comment to T184244: Puppet broken on deployment-mx due to systemd on trusty.

Created a new system, ran into the problem that https://gerrit.wikimedia.org/r/#/c/403326/ fixes

Jan 10 2018, 11:10 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure

Jan 9 2018

Krenair closed T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?, a subtask of T132259: Deployment-prep hosts with puppet errors (tracking), as Resolved.
Jan 9 2018, 9:56 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair closed T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'? as Resolved.

<Krenair> ottomata, -eventlogging04?
<ottomata> superset is there?
<ottomata> ???
<Krenair> looks like it yep
<Krenair> is it not supposed to be?
<Krenair> ottomata?
<ottomata> Krenair: no
<ottomata> no idea why it would be...
<Krenair> hm, ok
<Krenair> looks like someone added profile::superset to the instance's roles list in horizon puppet data
<ottomata> weird, did I? is it posssible I did that accidentally? we don't run druid in deployment-prep, dunno why i would...
<ottomata> maybe i had the wrong tab open?
<Krenair> it's possible
<ottomata> Krenair: removing.

Jan 9 2018, 9:56 PM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?.

I ran the exact same command that puppet does (as the user specified in the puppet file), and it appears to have worked. I don't know why, but it now succeeds. :/

Jan 9 2018, 9:47 PM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair edited projects for T184555: All IP addresses used for sending emails by Wikimedia's services, added: Mail; removed Wikimedia-Mailing-lists.

Could potentially give them the IPs for mx1001.wikimedia.org / mx2001.wikimedia.org, but they might change in future... And other stuff (misc services) might also be sending mail without going via the MX hosts?

Jan 9 2018, 9:09 PM · Mail, Operations
Krenair edited projects for T184555: All IP addresses used for sending emails by Wikimedia's services, added: Operations; removed MediaWiki-Email.
Jan 9 2018, 9:07 PM · Mail, Operations
Krenair added a comment to T184540: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running.

Is mysql-upgrade going to ensure it doesn't run while anything else is doing DDL?

Jan 9 2018, 7:54 PM · DBA, Data-Services, cloud-services-team

Jan 8 2018

Krenair created T184482: analytics VPS project puppet errors.
Jan 8 2018, 10:18 PM · Analytics-Kanban, User-Elukey, Puppet
Krenair added a comment to T132259: Deployment-prep hosts with puppet errors (tracking).

-snapshot01 is T184270 (package it wants is missing from stretch, moritz to fix when higher priority things are done)

Jan 8 2018, 9:32 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair added a comment to T184478: Puppet broken on deployment-ores01 due to missing hieradata.

It actually looks like no one but me has logged onto this thing

Jan 8 2018, 9:21 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184478: Puppet broken on deployment-ores01 due to missing hieradata as Normal priority.
Jan 8 2018, 9:21 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure
Krenair renamed T184477: Puppet disabled for a month on deployment-restbase0[12] instances from Puppet disabled for a month on deployment-restbase instances to Puppet disabled for a month on deployment-restbase0[12] instances.
Jan 8 2018, 9:16 PM · Services (done), Puppet, Beta-Cluster-Infrastructure
Krenair added a parent task for T184477: Puppet disabled for a month on deployment-restbase0[12] instances: T132259: Deployment-prep hosts with puppet errors (tracking).
Jan 8 2018, 9:16 PM · Services (done), Puppet, Beta-Cluster-Infrastructure
Krenair added a subtask for T132259: Deployment-prep hosts with puppet errors (tracking): T184477: Puppet disabled for a month on deployment-restbase0[12] instances.
Jan 8 2018, 9:16 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair created T184477: Puppet disabled for a month on deployment-restbase0[12] instances.
Jan 8 2018, 9:15 PM · Services (done), Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T148843: GPU upgrade for stats machine.

lucky you didn't go with nvidia: https://www.theregister.co.uk/2018/01/03/nvidia_server_gpus/

Jan 8 2018, 7:14 PM · Operations, Analytics-Cluster, Analytics, Research-management
Krenair added a comment to T184176: Scap not working in Beta.

@Krenair: that should be fixed as soon as jenkins is finished building https://integration.wikimedia.org/ci/job/phabricator-jessie-commits/896/

Jan 8 2018, 7:07 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
Krenair claimed T184236: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module.

Found a syntax problem in the latest version of it too (jenkins confirmed), fixed that, and added a dependent patch that allows the names in use on deployment-prep

Jan 8 2018, 12:57 AM · Patch-For-Review, Operations, media-storage, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184236: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module.

Looks like the reason is we have an old broken version of https://gerrit.wikimedia.org/r/#/c/361648/7 cherry-picked

Jan 8 2018, 12:27 AM · Patch-For-Review, Operations, media-storage, Puppet, Beta-Cluster-Infrastructure
Krenair added a project to T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing: Patch-For-Review.
Jan 8 2018, 12:08 AM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure

Jan 7 2018

Krenair added a comment to T184236: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module.

I think the code was always broken and it actually wanted to do this:

diff --git a/modules/swift/manifests/init_device.pp b/modules/swift/manifests/init_device.pp
index 69ab253328..cb8cb20250 100644
--- a/modules/swift/manifests/init_device.pp
+++ b/modules/swift/manifests/init_device.pp
@@ -1,5 +1,5 @@
 define swift::init_device($partition_nr='1') {
-    if (! $title =~ /^[hvs]d[a-z]+$/) {
+    if (!($title =~ /^[hvs]d[a-z]+$/)) {
         fail("Invalid name ${title} for swift::init_device")
     }
Jan 7 2018, 11:46 PM · Patch-For-Review, Operations, media-storage, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?.

have tried cloning from gerrit to deployment-tin:/srv/deployment/analytics/superset/deploy/ but then it needs some missing DEPLOY_HEAD file, no idea where that comes from

Jan 7 2018, 11:35 PM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?.

Probably not helping matters is puppet errors on deployment-tin - cf T184176

Jan 7 2018, 11:31 PM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184176: Scap not working in Beta.

Is this problem related? I was about to go and report a separate bug, but...

Info: Applying configuration version '1515367485'
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.8.0-1~20180105205453.271 to 3.7.4-3+0~20180106122359.272~1.gbp3819c6 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Jan 7 2018, 11:27 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
Krenair placed T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing up for grabs.

I have a feeling that https://gerrit.wikimedia.org/r/402432 which we cherry-picked for T184239 may have fixed this. The hosts no longer have errors

Jan 7 2018, 11:22 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair renamed T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing from Puppet broken on deployment-kafka-jump-[12] due to version of a package being missing to Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing.
Jan 7 2018, 11:20 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T174742: deployment-kafka01 - disk is full.

T184235 might be a repeat of this?

Jan 7 2018, 11:16 PM · Analytics-Kanban, Beta-Cluster-Infrastructure

Jan 6 2018

Krenair added a comment to T184245: Create some mechanism for instances in projects to modify the project Designate records.

Might also be worth looking into TSIG, dunno if what we run (pdns IIRC?) supports it in a way we can easily configure or not

Jan 6 2018, 5:01 PM · Operations, DNS, Beta-Cluster-reproducible, Cloud-VPS

Jan 5 2018

Krenair renamed T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances from Ferm/DNS library weirdness causing puppet errors on 12 deployment-prep instances to Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances.
Jan 5 2018, 11:31 PM · Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, DNS, Traffic
Krenair added a comment to T132259: Deployment-prep hosts with puppet errors (tracking).

It's fine with me if you want to move them all to a particular workboard column instead of a tracking task

Jan 5 2018, 11:24 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair added a subtask for T132259: Deployment-prep hosts with puppet errors (tracking): T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances.
Jan 5 2018, 11:22 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair added a parent task for T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances: T132259: Deployment-prep hosts with puppet errors (tracking).
Jan 5 2018, 11:22 PM · Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, DNS, Traffic
Krenair renamed T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances from Ferm/DNS library weirdness on deployment-mediawiki boxes to Ferm/DNS library weirdness causing puppet errors on 12 deployment-prep instances.
Jan 5 2018, 11:03 PM · Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, DNS, Traffic
Krenair added a comment to T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances.

Gave up waiting for that (it's been almost a year), sent a message anyway and it's been held for moderation.

Jan 5 2018, 10:51 PM · Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, DNS, Traffic
Krenair added a comment to T184239: Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch.

Actually the remaining ones appear to be T153468

Jan 5 2018, 10:39 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184239: Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch.

Patch handles the errors, got some more ones on some of the hosts

Jan 5 2018, 10:28 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair merged T173469: Non-existent wiki urls on beta cluster gives Unsafe/Insecure connection message into T182927: various .beta.wmflabs.org domains use an invalid ssl certificate.
Jan 5 2018, 10:19 PM · Beta-Cluster-Infrastructure
Krenair merged task T173469: Non-existent wiki urls on beta cluster gives Unsafe/Insecure connection message into T182927: various .beta.wmflabs.org domains use an invalid ssl certificate.
Jan 5 2018, 10:19 PM · Upstream, Beta-Cluster-Infrastructure
Krenair added a comment to T184239: Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch.

Dug into this a bit more with some help from paladox and mutante, it seems the problem is apt-get update failing due to errors relating to including a stretch-wikimedia experimental sources entry, which seems broken on the apt.wikimedia.org end

Jan 5 2018, 9:10 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184239: Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch.

Nope, it just plain doesn't exist:

alex@alex-laptop:~$ ssh deployment-mediawiki07
Linux deployment-mediawiki07 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64
Debian GNU/Linux 9.2 (stretch)
deployment-mediawiki07 is mediawiki::appserver
The last Puppet run was at Fri Jan  5 20:32:39 UTC 2018 (13 minutes ago). 
Last login: Fri Jan  5 01:50:07 2018 from 10.68.18.65
krenair@deployment-mediawiki07:~$ apt-cache policy prometheus-nutcracker-exporter
N: Unable to locate package prometheus-nutcracker-exporter
Jan 5 2018, 8:47 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T180935: Various puppet issues in deployment-prep.

As for puppet being broken on several instances, indeed we could use some new tasks. The reasons listed in this are no more accurate, so I am declining as outdated.

Jan 5 2018, 8:41 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure
Krenair placed T184245: Create some mechanism for instances in projects to modify the project Designate records up for grabs.

(alternatively we could just not use designate and instead run our own DNS server and stick an NS record in designate, but that kind of sucks)

Jan 5 2018, 2:56 AM · Operations, DNS, Beta-Cluster-reproducible, Cloud-VPS
Krenair created T184245: Create some mechanism for instances in projects to modify the project Designate records.
Jan 5 2018, 2:55 AM · Operations, DNS, Beta-Cluster-reproducible, Cloud-VPS
Krenair added a comment to T184234: Puppet broken on deployment-cache-text04 due to varnishkafka issues.

hiera part:

diff --git a/hieradata/labs/deployment-prep/host/deployment-cache-text04.yaml b/hieradata/labs/deployment-prep/host/deployment-cache-text04.yaml
index a4e902a4ea..908d7c7ed7 100644
--- a/hieradata/labs/deployment-prep/host/deployment-cache-text04.yaml
+++ b/hieradata/labs/deployment-prep/host/deployment-cache-text04.yaml
@@ -1,3 +1,4 @@
+profile::cache::kafka::statsv::kafka_cluster_name: main-eqiad
 profile::cache::base::varnish_version: 5
 nginx::variant: extras
 cache::lua_support: true
Jan 5 2018, 2:48 AM · Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184244: Puppet broken on deployment-mx due to systemd on trusty as Normal priority.
Jan 5 2018, 2:30 AM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184243: Puppet broken on deployment-redis0[12] due to systemd on trusty as Normal priority.
Jan 5 2018, 2:29 AM · Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T182927: various .beta.wmflabs.org domains use an invalid ssl certificate.

https://community.letsencrypt.org/t/staging-endpoint-for-acme-v2/49605

Jan 5 2018, 2:23 AM · Beta-Cluster-Infrastructure
Krenair triaged T184242: Puppet broken on deployment-netbox, looks like it thinks its a prod box as Normal priority.
Jan 5 2018, 2:21 AM · Operations, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184241: Puppet broken on deployment-trending01 due to removal of role as Normal priority.
Jan 5 2018, 2:17 AM · Services (done), Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing as Normal priority.
Jan 5 2018, 2:15 AM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184239: Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch as Normal priority.
Jan 5 2018, 2:12 AM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184235: Puppet broken on deployment-kafka03 due to full disk.

Repeat of T174742 ?

Jan 5 2018, 1:54 AM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'? as Normal priority.
Jan 5 2018, 1:53 AM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184236: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module as Normal priority.
Jan 5 2018, 1:51 AM · Patch-For-Review, Operations, media-storage, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184235: Puppet broken on deployment-kafka03 due to full disk.

2.7G /var/log/daemon.log
2.6G /var/log/daemon.log.1
605M /var/spool/kafka/eventlogging-client-side-0
604M /var/spool/kafka/eventlogging-valid-mixed-0
588M /var/spool/kafka/eventlogging-valid-mixed-1
572M /var/spool/kafka/eventlogging-client-side-1
221M /var/log/kafka/controller.log
257M /var/log/kafka/kafka-mirror-main-deployment-prep_to_analytics.log.1
257M /var/log/kafka/kafka-mirror-main-deployment-prep_to_analytics.log.2
257M /var/log/kafka/kafka-mirror-main-deployment-prep_to_analytics.log.3
257M /var/log/kafka/kafka-mirror-main-deployment-prep_to_analytics.log.4
257M /var/log/kafka/server.log.1
257M /var/log/kafka/server.log.2
257M /var/log/kafka/server.log.3
257M /var/log/kafka/server.log.4
193M /var/spool/kafka/webrequest_text-21
193M /var/spool/kafka/webrequest_text-11
193M /var/spool/kafka/webrequest_text-23
193M /var/spool/kafka/webrequest_text-6
192M /var/spool/kafka/webrequest_text-17
193M /var/spool/kafka/webrequest_text-9
193M /var/spool/kafka/webrequest_text-14
193M /var/spool/kafka/webrequest_text-13
193M /var/spool/kafka/webrequest_text-7
193M /var/spool/kafka/webrequest_text-19
193M /var/spool/kafka/webrequest_text-10
193M /var/spool/kafka/webrequest_text-22
193M /var/spool/kafka/webrequest_text-20
193M /var/spool/kafka/webrequest_text-2
193M /var/spool/kafka/webrequest_text-15
192M /var/spool/kafka/webrequest_text-5
193M /var/spool/kafka/webrequest_text-18
193M /var/spool/kafka/webrequest_text-8
193M /var/spool/kafka/webrequest_text-3
193M /var/spool/kafka/webrequest_text-4
193M /var/spool/kafka/webrequest_text-0
193M /var/spool/kafka/webrequest_text-1
193M /var/spool/kafka/webrequest_text-12
193M /var/spool/kafka/webrequest_text-16
166M /var/log/kafka/state-change.log

Jan 5 2018, 1:46 AM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184235: Puppet broken on deployment-kafka03 due to full disk.
krenair@deployment-kafka03:~$ sudo puppet agent -tv
Warning: Setting configtimeout is deprecated. 
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1146:in `issue_deprecation_warning')
Error: Could not run Puppet configuration client: No space left on device @ fptr_finalize - /var/lib/puppet/state/agent_catalog_run.lock
Error: Could not run: no implicit conversion of Puppet::Util::Log into Integer
krenair@deployment-kafka03:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs           401M   41M  361M  11% /run
/dev/vda3        19G   19G     0 100% /
tmpfs          1003M     0 1003M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs          1003M     0 1003M   0% /sys/fs/cgroup
Jan 5 2018, 1:43 AM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184235: Puppet broken on deployment-kafka03 due to full disk as Normal priority.
Jan 5 2018, 1:42 AM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184234: Puppet broken on deployment-cache-text04 due to varnishkafka issues as Normal priority.
Jan 5 2018, 1:39 AM · Puppet, Beta-Cluster-Infrastructure
Krenair removed a project from T184233: deployment-phab completely broken: Tracking.
Jan 5 2018, 1:38 AM · Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184233: deployment-phab completely broken as Normal priority.
Jan 5 2018, 1:37 AM · Puppet, Beta-Cluster-Infrastructure
Restricted Application added a project to T184230: Disavow emails from wikipedia.com: Operations.
Jan 5 2018, 1:35 AM · Patch-For-Review, Operations, Mail

Jan 4 2018

Krenair added a comment to T182927: various .beta.wmflabs.org domains use an invalid ssl certificate.

https://letsencrypt.org/2017/12/07/looking-forward-to-2018.html

First, we’re planning to introduce an ACME v2 protocol API endpoint and support for wildcard certificates along with it. Wildcard certificates will be free and available globally just like our other certificates. We are planning to have a public test API endpoint up by January 4, and we’ve set a date for the full launch: Tuesday, February 27.

Jan 4 2018, 11:55 PM · Beta-Cluster-Infrastructure

Jan 3 2018

Krenair created P6524 Tools group membership.
Jan 3 2018, 9:30 PM · Tools
Krenair added a comment to T182407: Strip 2FA for 'Martin Urbanec' account at arbcom-cs.wikipedia.org.
Jan 3 2018, 7:25 PM · Support-and-Safety, Wikimedia-Site-requests, User-Urbanecm

Jan 2 2018

Krenair added a comment to T182407: Strip 2FA for 'Martin Urbanec' account at arbcom-cs.wikipedia.org.

It'll be named arbcom_cswiki IIRC. You should be able to find it on the S3
DB servers, though the sql arbcom_cswiki command from terbium (or modern
equivalent) should find it for you. I don't recommend doing it yourself
without a dev/ops person

Jan 2 2018, 1:45 PM · Support-and-Safety, Wikimedia-Site-requests, User-Urbanecm
Krenair added a comment to T182407: Strip 2FA for 'Martin Urbanec' account at arbcom-cs.wikipedia.org.

(best done by someone who is used to messing with mysql directly. Plenty of
people can do it though)

Jan 2 2018, 1:29 PM · Support-and-Safety, Wikimedia-Site-requests, User-Urbanecm
Krenair added a comment to T182407: Strip 2FA for 'Martin Urbanec' account at arbcom-cs.wikipedia.org.

IIRC it can't be done on-wiki anywhere anyway, has to be done by
restricted/deployment/ops in mysql directly, like the other wikis. I think
you have restricted access though.

Jan 2 2018, 1:25 PM · Support-and-Safety, Wikimedia-Site-requests, User-Urbanecm
Krenair added a comment to T183916: Create subdomain for Research landing page.

Why is it not simply a redirect to a page on meta?

Jan 2 2018, 1:03 PM · Research, Operations, Domains, Traffic
Krenair added a comment to T183790: Request for steward rights on the Beta Cluster.

I don't think one is provided, most probably just set up their own MW
installs

Jan 2 2018, 12:29 PM · Beta-Cluster-Infrastructure

Jan 1 2018

Krenair added a project to T183862: Recent Changes is broken on Dutch Wikipedia Beta on Beta Cluster: Beta-Cluster-Infrastructure.

Thanks @Ladsgroup. Do you know if this will occur on other wikis too?

Jan 1 2018, 10:08 PM · Beta-Cluster-Infrastructure, User-Ladsgroup, Scoring-platform-team (Current), MediaWiki-extensions-ORES, Beta-Cluster-reproducible
Krenair added a project to T183862: Recent Changes is broken on Dutch Wikipedia Beta on Beta Cluster: ORES.
Jan 1 2018, 3:13 PM · Beta-Cluster-Infrastructure, User-Ladsgroup, Scoring-platform-team (Current), MediaWiki-extensions-ORES, Beta-Cluster-reproducible

Dec 31 2017

Krenair edited projects for T183826: "Error 503 Backend fetch failed" on wikistream.wmflabs.org, added: Tools; removed Cloud-Services.
Dec 31 2017, 11:55 AM · VPS-Projects

Dec 30 2017

Krenair updated subscribers of T102367: Migrate tools.wmflabs.org to https only (and set HSTS).
Dec 30 2017, 2:54 AM · Operations, Traffic, HTTPS, Toolforge
Krenair added a comment to T102367: Migrate tools.wmflabs.org to https only (and set HSTS).

If I have my tool set a strict-transport-security: max-age=86400 header, will that impact other tools as well since they're on the same subdomain? Or will it just affect my tool?

Dec 30 2017, 2:54 AM · Operations, Traffic, HTTPS, Toolforge

Dec 26 2017

Krenair added a comment to T182927: various .beta.wmflabs.org domains use an invalid ssl certificate.

https://letsencrypt.org/2017/12/07/looking-forward-to-2018.html

First, we’re planning to introduce an ACME v2 protocol API endpoint and support for wildcard certificates along with it. Wildcard certificates will be free and available globally just like our other certificates. We are planning to have a public test API endpoint up by January 4, and we’ve set a date for the full launch: Tuesday, February 27.

Dec 26 2017, 11:51 AM · Beta-Cluster-Infrastructure

Dec 22 2017

Krenair added a comment to T183549: Arbcom wikis are in both wikipedia.dblist and special.dblist.

Be very careful messing with this.

Dec 22 2017, 11:56 PM · Wikimedia-Site-requests, User-Urbanecm
Krenair added a comment to T180179: Evaluate the possibility to add Juniper images to Openstack.

Noting here that proprietary software is not usually installed on WMCS environments per https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#What_uses_of_Labs_do_we_not_like.3F (Proprietary Software).

Dec 22 2017, 11:12 PM · cloud-services-team (Kanban), Cloud-VPS, netops, Operations, Traffic