Krenair (Alex Monk)
Wikimedia volunteer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:34 PM (185 w, 4 d)
Availability
Available
IRC Nick
Krenair
LDAP User
Alex Monk
MediaWiki User
Krenair

I am a Wikimedia volunteer helping in various technical ways. These days it's usually Beta cluster related. I've previously spent significant amounts of time involved in MediaWiki development, software deployments to the Wikimedia cluster, and various other things. I am also an OTRS agent.

Some of my old VisualEditor work can be found under @AlexMonk-WMF instead

I have opinions on things, which do not necessarily represent those of any organisation I am, have previously been, or will in the future be affiliated with.

Recent Activity

Tue, Mar 27

Krenair renamed T190781: Secure deployment-prep sudo access to prevent member -> projectadmin escalation by dns-manager credentials from Secure deployment-prep sudo access to prevent member -> admin escalation by dns-manager credentials to Secure deployment-prep sudo access to prevent member -> projectadmin escalation by dns-manager credentials.
Tue, Mar 27, 7:53 AM · Beta-Cluster-Infrastructure
Krenair created T190781: Secure deployment-prep sudo access to prevent member -> projectadmin escalation by dns-manager credentials.
Tue, Mar 27, 7:52 AM · Beta-Cluster-Infrastructure

Mar 24 2018

Krenair added a comment to T190629: mwgate-npm-node-6-docker stylelint test failing for certain css/less files, but providing only dots for the issues, with no explanation.

<Isarra> But what doesn't it like about it?
<Krenair> I don't know, I think it's trying to tell us in some sort of bizarre morse code
<Krenair> just like with whitespace instead of dashes

Mar 24 2018, 7:18 PM · Continuous-Integration-Infrastructure
Krenair added a comment to T182927: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains.

Once the commit above is approved, I've got a WIP script based on a more modern (ACME v2 supporting) acme_tiny.py from https://github.com/diafygi/acme-tiny, into which I've mixed some of Adrien Dorsaz's DNS-01 changes from https://github.com/Trim/acme-dns-tiny/blob/master/acme_dns_tiny.py, along with some of my own to hook it up to the Designate API instead of TSIG keys.

Mar 24 2018, 4:37 PM · Patch-For-Review, Beta-Cluster-Infrastructure

Mar 15 2018

Krenair awarded T189759: InteractionTimeline loads fonts from Google font CDN a Burninate token.
Mar 15 2018, 9:27 PM · Anti-Harassment (AHT Sprint 17), Privacy, Tools, InteractionTimeline

Mar 13 2018

Krenair added a comment to T182927: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains.

https://community.letsencrypt.org/t/acme-v2-and-wildcard-certificate-support-is-live/55579
We'll want to have a look at how much needs to change in acme_tiny to handle this (unless upstream is already there)

Mar 13 2018, 11:08 PM · Patch-For-Review, Beta-Cluster-Infrastructure

Mar 12 2018

Krenair added a comment to T77858: Make Privacy Policy/ToS on Beta Cluster explain the privacy implications of using Beta.
In T77858#4045141, @Tgr wrote:

Anyway it creates confusion :-) The Labs ToU is between tool developers and the WMF, not users and the WMF. It does not really make sense to link from as generic site footer as a ToU (and even less as a privacy policy, which it is not).

Mar 12 2018, 11:39 PM · Beta-Cluster-Infrastructure
Krenair added a comment to T156733: Setup TorBlock cron on silver to update exit node list.

Do we need some equivalent change on labweb1001/1002 for this? Or do the varnishes take care of it now?

Mar 12 2018, 11:32 PM · Cloud-Services, wikitech.wikimedia.org
Krenair added a comment to T77858: Make Privacy Policy/ToS on Beta Cluster explain the privacy implications of using Beta.
In T77858#4045113, @Tgr wrote:

I don't think making the privacy policy link to the labs terms of use is helpful. The kind of confusion it creates can be evidenced in T161051.

Mar 12 2018, 11:28 PM · Beta-Cluster-Infrastructure
Krenair created P6838 List wmflabs.org recordsets.
Mar 12 2018, 7:00 PM · Cloud-VPS

Mar 9 2018

Krenair added a comment to T189334: Don't require captchas in userspace.

Aren't user pages a frequent spambot target?

Mar 9 2018, 9:55 PM · Community-consensus-needed, Wikimedia-Site-requests

Mar 7 2018

Krenair added a comment to T189168: How to purge misc-web varnishes for wikitech changes?.

did you try 239.128.0.115 ?

Mar 7 2018, 11:36 PM · Patch-For-Review, Operations, cloud-services-team (Kanban), Traffic, wikitech.wikimedia.org
Krenair added a comment to T188839: When editing source, sometimes characters get put in strange places if you have too many characters on a line (?).

Not sure if this is the same bug @Krenair is thinking about; it sounds similar but not obviously related to line length in my experience.

Mar 7 2018, 7:35 PM · VisualEditor (Current work), VisualEditor-ContentEditable, VisualEditor-MediaWiki-2017WikitextEditor

Mar 5 2018

Krenair added a comment to T188915: Move wikitech images to swift.

create a swift container for wikitech images

Mar 5 2018, 6:34 PM · Patch-For-Review, User-ArielGlenn, Goal, cloud-services-team (FY2017-18), Cloud-Services
Krenair added a comment to T188839: When editing source, sometimes characters get put in strange places if you have too many characters on a line (?).

I can reproduce it without pasting at all, I just fill the line with some character ('a'), press space three times, and then it breaks when I put another character in.

Mar 5 2018, 6:24 PM · VisualEditor (Current work), VisualEditor-ContentEditable, VisualEditor-MediaWiki-2017WikitextEditor

Mar 4 2018

Krenair added a comment to T188871: support custom wikilink procedure per-wiki.

If you paste in the full URL as an external link to VE, Parsoid actually appears to insert an interwiki link where possible
E.g. if I make some text like 'asd', and turn it into a link to https://www.mediawiki.org/wiki/MediaWiki then the wikitext it will save it is actually [[mw:MediaWiki|asd]]

Mar 4 2018, 9:39 PM · VisualEditor
Krenair added a comment to T188862: Disable the collection of private information on abusefilter log for Beta Cluster wikis.

sure but they could turn it back on, or dig up IPs etc. some other way

Mar 4 2018, 7:31 PM · Privacy, Beta-Cluster-Infrastructure, User-MarcoAurelio
Krenair added a comment to T188862: Disable the collection of private information on abusefilter log for Beta Cluster wikis.

I don't think this will actually prevent people without NDAs from accessing it as they can just get shell access and put some other logging in

Mar 4 2018, 7:28 PM · Privacy, Beta-Cluster-Infrastructure, User-MarcoAurelio
Krenair closed T164047: Captchas sent with wrong mime type on beta as Resolved.

Well that did the trick

krenair@deployment-tin:/srv/mediawiki-staging/php-master/extensions/ConfirmEdit$ mwscript extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php aawiki --wordlist=/tmp/words --font=/usr/share/fonts/truetype/freefont/FreeMonoBoldOblique.ttf --blacklist=/tmp/badwords --fill=10000 --verbose --delete
Generating 10000 new captchas.. Done.
Mar 4 2018, 2:13 AM · Beta-Cluster-Infrastructure
Krenair added a comment to T164047: Captchas sent with wrong mime type on beta.

Couldn't quite just use https://wikitech.wikimedia.org/wiki/Generating_CAPTCHAs because it expects prod (antispam) secrets to be in place (the ones in labs/private.git are predictably useless)
Had to download equivalents from a couple of helpful github repos
Am regenerating now

Mar 4 2018, 1:49 AM · Beta-Cluster-Infrastructure
Krenair added a comment to T188776: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches.

(Some of) the stuff I wondered across with a quick grep that doesn't look to be covered by that page:

Mar 4 2018, 1:43 AM · Wiki-Setup (Rename), Release-Engineering-Team, Traffic, DNS, Operations, WMF-Communications
Krenair added a comment to T188776: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches.

Okay but the existing site presumably has to continue to live somewhere, and the special config for it has to be cleaned up a little probably so it can continue to function properly

Mar 4 2018, 1:30 AM · Wiki-Setup (Rename), Release-Engineering-Team, Traffic, DNS, Operations, WMF-Communications
Krenair updated subscribers of T188776: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches.

@Bawolff, is that site approved to sit under wikimedia.org, seeing as it has wgRawHtml?

Mar 4 2018, 1:09 AM · Wiki-Setup (Rename), Release-Engineering-Team, Traffic, DNS, Operations, WMF-Communications
Krenair added a project to T188776: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches: Wikimedia-Site-requests.

The procedure for handling such a change, if a developer decides to accept this request for some reason, live at https://wikitech.wikimedia.org/wiki/Rename_a_wiki_domain
Beware of undocumented hidden cases (or new services that got deployed and forgot to update the docs) that completely take down the site, especially seeing as this site likely has a bunch of special cases

Mar 4 2018, 1:05 AM · Wiki-Setup (Rename), Release-Engineering-Team, Traffic, DNS, Operations, WMF-Communications
Krenair added a comment to T164047: Captchas sent with wrong mime type on beta.

The answer probably lies in some python-swiftclient or python-requests version used by the Trusty -ms-fe01 host on which I originally did the migration (-ms-fe02 came along in T162247 and it runs Jessie)
Let's see if we can just regenerate all captcha images

Mar 4 2018, 12:12 AM · Beta-Cluster-Infrastructure

Mar 3 2018

Krenair added a comment to T164047: Captchas sent with wrong mime type on beta.

That task shows that I ran root@deployment-ms-fe01:/data/project/upload7/private/captcha# swift upload global-data-captcha-render * (I'm so glad I kept records of important commands I was running so we could refer back to them later)
Looks like swiftclient does content sniffing based on the file extension though:

root@deployment-ms-fe02:/etc/swift# touch testfile.html
root@deployment-ms-fe02:/etc/swift# swift upload global-data-captcha-render testfile.html
testfile.html
root@deployment-ms-fe02:/etc/swift# swift --debug download global-data-captcha-render testfile.html --no-download 2>&1 | grep Content-Type | tail -n1
DEBUG:swiftclient:RESP HEADERS: [('Content-Length', '0'), ('Content-Type', 'text/html'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 03 Mar 2018 23:14:02 GMT'), ('Etag', 'd41d8cd98f00b204e9800998ecf8427e'), ('X-Timestamp', '1520118841.60094'), ('X-Object-Meta-Mtime', '1520118831.473796'), ('X-Trans-Id', 'tx46d161c44565461a9d53f-005a9b2c3e'), ('Date', 'Sat, 03 Mar 2018 23:14:06 GMT'), ('Connection', 'keep-alive')]
Mar 3 2018, 11:15 PM · Beta-Cluster-Infrastructure
Krenair added a comment to T164047: Captchas sent with wrong mime type on beta.

That last-modified date was from near my NFS -> Swift migration: T64835#2459268
So the file itself may be very old

Mar 3 2018, 11:02 PM · Beta-Cluster-Infrastructure
Krenair added a comment to T164047: Captchas sent with wrong mime type on beta.

Loaded the login page, it downloaded this URL:

alex@alex-laptop:~$ curl -sI 'https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:Captcha/image&wpCaptchaId=1063782475' | grep ^Content-Type
Content-Type: application/x-www-form-urlencoded
Mar 3 2018, 10:45 PM · Beta-Cluster-Infrastructure
Krenair created T188839: When editing source, sometimes characters get put in strange places if you have too many characters on a line (?).
Mar 3 2018, 9:23 PM · VisualEditor (Current work), VisualEditor-ContentEditable, VisualEditor-MediaWiki-2017WikitextEditor
Krenair created T188838: Source editor: Trying to insert a comment results in a mess in the editor (but not in the output?).
Mar 3 2018, 8:48 PM · VisualEditor-MediaWiki-2017WikitextEditor, VisualEditor
Krenair added a comment to T188837: Extra hidden character seems to be at the beginning/end of the document, causing odd left/right key behaviour.

Also once you've moved past one of these with the keyboard you can insert stuff after it, but it still remains hidden in the middle of your content

Mar 3 2018, 8:41 PM · VisualEditor
Krenair added a comment to T188837: Extra hidden character seems to be at the beginning/end of the document, causing odd left/right key behaviour.

You can hold shift during the final step and it appears to highlight an img.ve-ce-chimera.ve-ce-chimera-webkit (am using Chrome 63)

Mar 3 2018, 8:34 PM · VisualEditor
Krenair updated the task description for T188837: Extra hidden character seems to be at the beginning/end of the document, causing odd left/right key behaviour.
Mar 3 2018, 8:33 PM · VisualEditor
Krenair added a comment to T188837: Extra hidden character seems to be at the beginning/end of the document, causing odd left/right key behaviour.

is it maybe allowing the cursor to go outside the paragraph or something?

Mar 3 2018, 8:25 PM · VisualEditor
Krenair renamed T188837: Extra hidden character seems to be at the beginning/end of the document, causing odd left/right key behaviour from Extra hidden character seems to be at the end of the document, causing odd left key behaviour to Extra hidden character seems to be at the beginning/end of the document, causing odd left/right key behaviour.
Mar 3 2018, 8:21 PM · VisualEditor
Krenair updated the task description for T188837: Extra hidden character seems to be at the beginning/end of the document, causing odd left/right key behaviour.
Mar 3 2018, 8:19 PM · VisualEditor
Krenair created T188837: Extra hidden character seems to be at the beginning/end of the document, causing odd left/right key behaviour.
Mar 3 2018, 8:15 PM · VisualEditor
Krenair added a comment to T186463: Uninstall Flow from Commons.

The comment posted on the village pump doesn't reflect Wikimedia best practices, in particular where it suggests that once an extension is installed we commit to keep it forever because otherwise the logs would not look pretty. The real policy is that we install things when they pass some strict requirements and we uninstall them when they do more harm than good, otherwise our technical debt and interface clutter would always go up.

Mar 3 2018, 2:49 PM · Commons, Patch-For-Review, Wikimedia-Site-requests, Collaboration-Team-Triage, StructuredDiscussions
Krenair added a comment to T188577: Add a config setting making all Flow boards read-only.

The original task description doesn't follow How to report a bug , in that it doesn't describe a problem or goal but only a specific proposed solution.

Mar 3 2018, 2:25 PM · MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), Patch-For-Review, Collaboration-Team-Triage, StructuredDiscussions

Feb 28 2018

IKhitron awarded T111439: Current he.wiki interface-editor group config doesn't comply with privacy policy a Like token.
Feb 28 2018, 11:33 AM · Patch-For-Review, Privacy, Wikimedia-Site-requests

Feb 26 2018

dcausse awarded T125976: Run mediawiki::maintenance scripts in Beta Cluster a Like token.
Feb 26 2018, 1:43 PM · User-Addshore, Beta-Cluster-Infrastructure

Feb 19 2018

Krenair awarded T187716: Sunset Wikipedia Zero (tracking) a Goat token.
Feb 19 2018, 7:36 PM · Wikimedia-Site-requests

Feb 16 2018

Krenair added a comment to T186247: Hebrew Wikivoyage (via Tool "wikivoyage") loads assets by default from third-party sites.

Am I missing something here or has something gone wrong in the handling of this ticket? How was it known for two weeks but stayed online?

Feb 16 2018, 11:50 PM · Collaboration-Team-Triage (Collab-Team-This-Quarter), Vuln-Infoleak, Community-Liaisons, Collaboration-Feature-Rollouts (Collaboration-Maps), Discovery, Privacy, Toolforge-standards-committee, Maps, WMF-Legal, Tools

Feb 11 2018

Krenair added a comment to T186675: Add 'centralauth' to meta_p.wiki so that apps can re-use the appropriate slice.

I would suggest splitting the table into two: one that has a list of wikis (in the current setup all columns except slice) and and one that lists databases and their properties (dbname, slice). The original wiki table can then be replaced with a view that performs the relevant join to maintain backwards compatibility.

Feb 11 2018, 11:16 PM · Toolforge, Data-Services, cloud-services-team

Feb 5 2018

Krenair added a comment to T186415: Create trusted group in gerrit.

But we can block it for all users right?

Feb 5 2018, 7:47 PM · Developer-Relations, Gerrit
Krenair added a comment to T186415: Create trusted group in gerrit.

Especially with the things I've heard about it, enabling private changes
would be a bad idea, regardless of whether they can be used for security
patches or not

Feb 5 2018, 7:42 PM · Developer-Relations, Gerrit

Jan 31 2018

Krenair added a comment to T186133: Login session bug on Beta Commons.

Session unreliability is a long-standing issue on beta (see e.g. T172560: "Loss of session data" on Beta Cluster; there was another task that I can't find now where @Krenair tracked it down to redis replication failures)

Jan 31 2018, 7:33 PM · Beta-Cluster-reproducible, Beta-Cluster-Infrastructure

Jan 24 2018

Krenair added a project to T185670: Request for allowance of multiple account registers from same IP for 2018-01-25 14:00UTC: Wikimedia-Site-requests.

I wouldn't count on it seeing how last-minute this request is. You'll need to provide the list of wikis to allow it on, and preferably a link to info about the event.

Jan 24 2018, 11:56 PM · Wikimedia-Site-requests
Krenair added a comment to T185606: wikistream.wmflabs.org 502 Varnish Error.

I won't be able to SSH to that as I am not a project member.

Jan 24 2018, 8:59 PM · Cloud-VPS

Jan 23 2018

Krenair edited projects for T185606: wikistream.wmflabs.org 502 Varnish Error, added: Cloud-VPS; removed Tools.

http://tools.wmflabs.org/openstack-browser/proxy/ shows that it goes to ws-web.wikistream.eqiad.wmflabs on port 80
The error message I get doesn't reference Varnish, it is from an nginx instance.
Do you perhaps have nginx running on ws-web? When you say 'I can see that the service itself is running fine', what service is that? Does the ws-web instance have a security rule allowing port 80 in?

Jan 23 2018, 10:58 PM · Cloud-VPS
Krenair added a comment to T185319: IRC RecentChanges feed: code stewardship request.

Pretty sure MW has supported having multiple destinations for these streams for years now. So you could have multiple servers (not in an IRC network) receiving the changes from MW and being available for clients to connect to. The same set of changes (that introduced RCFeeds) would've been the ones that deprecated and eventually killed $wgRC2UDPAddress.

Jan 23 2018, 2:01 AM · Tools, Operations, Analytics, Wikimedia-IRC-RC-Server, Code-Stewardship-Reviews

Jan 20 2018

Krenair added a comment to T184478: Puppet broken on deployment-ores01 due to missing hieradata.

So you hold your resolved tasks open for up to a month after resolution? But yeah, this is very much a Beta-Cluster-Infrastructure task.

Jan 20 2018, 4:44 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184478: Puppet broken on deployment-ores01 due to missing hieradata.

What do you mean 'before reporting them'?

Jan 20 2018, 2:31 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure

Jan 19 2018

Krenair closed T173554: Puppet broken on deployment-sentry01 as Resolved.

This took a frankly ridiculous amount of time to solve considering how simple the problem and patch was.

Jan 19 2018, 9:38 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair closed T173554: Puppet broken on deployment-sentry01, a subtask of T132259: Deployment-prep hosts with puppet errors (tracking), as Resolved.
Jan 19 2018, 9:38 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair closed T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing as Resolved.
Jan 19 2018, 9:36 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair closed T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing, a subtask of T132259: Deployment-prep hosts with puppet errors (tracking), as Resolved.
Jan 19 2018, 9:36 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair added a comment to T184478: Puppet broken on deployment-ores01 due to missing hieradata.

Looks like its fixed now? Wanna mark this as resolved?

Jan 19 2018, 9:31 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure

Jan 17 2018

Krenair awarded T184230: Disavow emails from wikipedia.com a Burninate token.
Jan 17 2018, 10:27 PM · Patch-For-Review, Operations, Mail

Jan 16 2018

Krenair added a comment to T185028: Beta cluster login broken.

Reproduced bug on enwiki

Jan 16 2018, 7:41 PM · MinervaNeue, Readers-Web-Backlog, Beta-Cluster-Infrastructure
Krenair renamed T185028: Beta cluster login broken from Unable to login commons wmflabs to Beta cluster login broken.
Jan 16 2018, 7:41 PM · MinervaNeue, Readers-Web-Backlog, Beta-Cluster-Infrastructure

Jan 15 2018

Krenair added a project to T184957: en:wikiversity Draft Namespace: Wikimedia-Site-requests.
Jan 15 2018, 10:45 PM · Patch-For-Review, User-Jayprakash12345, Wikimedia-Site-requests
Krenair added a comment to T167060: en.wiki domain owned by us, but isn't hosted by us??.

wikibooks.wiki too - https://meta.wikimedia.org/wiki/Requests_for_comment/Domain_parking

Jan 15 2018, 9:51 PM · WMF-Legal, Privacy, Domains, Operations, DNS, Traffic

Jan 11 2018

Krenair updated subscribers of T184234: Puppet broken on deployment-cache-text04 due to varnishkafka issues.
Jan 11 2018, 12:34 AM · Puppet, Beta-Cluster-Infrastructure

Jan 10 2018

Krenair added a comment to T184244: Puppet broken on deployment-mx due to systemd on trusty.

Created a new system, ran into the problem that https://gerrit.wikimedia.org/r/#/c/403326/ fixes

Jan 10 2018, 11:10 PM · Puppet, Beta-Cluster-Infrastructure

Jan 9 2018

Krenair closed T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?, a subtask of T132259: Deployment-prep hosts with puppet errors (tracking), as Resolved.
Jan 9 2018, 9:56 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair closed T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'? as Resolved.

<Krenair> ottomata, -eventlogging04?
<ottomata> superset is there?
<ottomata> ???
<Krenair> looks like it yep
<Krenair> is it not supposed to be?
<Krenair> ottomata?
<ottomata> Krenair: no
<ottomata> no idea why it would be...
<Krenair> hm, ok
<Krenair> looks like someone added profile::superset to the instance's roles list in horizon puppet data
<ottomata> weird, did I? is it posssible I did that accidentally? we don't run druid in deployment-prep, dunno why i would...
<ottomata> maybe i had the wrong tab open?
<Krenair> it's possible
<ottomata> Krenair: removing.

Jan 9 2018, 9:56 PM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?.

I ran the exact same command that puppet does (as the user specified in the puppet file), and it appears to have worked. I don't know why, but it now succeeds. :/

Jan 9 2018, 9:47 PM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair edited projects for T184555: All IP addresses used for sending emails by Wikimedia's services, added: Mail; removed Wikimedia-Mailing-lists.

Could potentially give them the IPs for mx1001.wikimedia.org / mx2001.wikimedia.org, but they might change in future... And other stuff (misc services) might also be sending mail without going via the MX hosts?

Jan 9 2018, 9:09 PM · Mail, Operations
Krenair edited projects for T184555: All IP addresses used for sending emails by Wikimedia's services, added: Operations; removed MediaWiki-Email.
Jan 9 2018, 9:07 PM · Mail, Operations
Krenair added a comment to T184540: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running.

Is mysql-upgrade going to ensure it doesn't run while anything else is doing DDL?

Jan 9 2018, 7:54 PM · DBA, Data-Services, cloud-services-team

Jan 8 2018

Krenair created T184482: analytics VPS project puppet errors.
Jan 8 2018, 10:18 PM · Analytics-Kanban, User-Elukey, Puppet
Krenair added a comment to T132259: Deployment-prep hosts with puppet errors (tracking).

-snapshot01 is T184270 (package it wants is missing from stretch, moritz to fix when higher priority things are done)

Jan 8 2018, 9:32 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair added a comment to T184478: Puppet broken on deployment-ores01 due to missing hieradata.

It actually looks like no one but me has logged onto this thing

Jan 8 2018, 9:21 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure
Krenair triaged T184478: Puppet broken on deployment-ores01 due to missing hieradata as Normal priority.
Jan 8 2018, 9:21 PM · User-Ladsgroup, Scoring-platform-team (Current), ORES, Puppet, Beta-Cluster-Infrastructure
Krenair renamed T184477: Puppet disabled for a month on deployment-restbase0[12] instances from Puppet disabled for a month on deployment-restbase instances to Puppet disabled for a month on deployment-restbase0[12] instances.
Jan 8 2018, 9:16 PM · Services (done), Puppet, Beta-Cluster-Infrastructure
Krenair added a parent task for T184477: Puppet disabled for a month on deployment-restbase0[12] instances: T132259: Deployment-prep hosts with puppet errors (tracking).
Jan 8 2018, 9:16 PM · Services (done), Puppet, Beta-Cluster-Infrastructure
Krenair added a subtask for T132259: Deployment-prep hosts with puppet errors (tracking): T184477: Puppet disabled for a month on deployment-restbase0[12] instances.
Jan 8 2018, 9:16 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair created T184477: Puppet disabled for a month on deployment-restbase0[12] instances.
Jan 8 2018, 9:15 PM · Services (done), Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T148843: GPU upgrade for stats machine.

lucky you didn't go with nvidia: https://www.theregister.co.uk/2018/01/03/nvidia_server_gpus/

Jan 8 2018, 7:14 PM · Operations, Analytics-Cluster, Analytics, Research-management
Krenair added a comment to T184176: Scap not working in Beta.

@Krenair: that should be fixed as soon as jenkins is finished building https://integration.wikimedia.org/ci/job/phabricator-jessie-commits/896/

Jan 8 2018, 7:07 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
Krenair claimed T184236: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module.

Found a syntax problem in the latest version of it too (jenkins confirmed), fixed that, and added a dependent patch that allows the names in use on deployment-prep

Jan 8 2018, 12:57 AM · Patch-For-Review, Operations, media-storage, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184236: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module.

Looks like the reason is we have an old broken version of https://gerrit.wikimedia.org/r/#/c/361648/7 cherry-picked

Jan 8 2018, 12:27 AM · Patch-For-Review, Operations, media-storage, Puppet, Beta-Cluster-Infrastructure
Krenair added a project to T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing: Patch-For-Review.
Jan 8 2018, 12:08 AM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure

Jan 7 2018

Krenair added a comment to T184236: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module.

I think the code was always broken and it actually wanted to do this:

diff --git a/modules/swift/manifests/init_device.pp b/modules/swift/manifests/init_device.pp
index 69ab253328..cb8cb20250 100644
--- a/modules/swift/manifests/init_device.pp
+++ b/modules/swift/manifests/init_device.pp
@@ -1,5 +1,5 @@
 define swift::init_device($partition_nr='1') {
-    if (! $title =~ /^[hvs]d[a-z]+$/) {
+    if (!($title =~ /^[hvs]d[a-z]+$/)) {
         fail("Invalid name ${title} for swift::init_device")
     }
Jan 7 2018, 11:46 PM · Patch-For-Review, Operations, media-storage, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?.

have tried cloning from gerrit to deployment-tin:/srv/deployment/analytics/superset/deploy/ but then it needs some missing DEPLOY_HEAD file, no idea where that comes from

Jan 7 2018, 11:35 PM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?.

Probably not helping matters is puppet errors on deployment-tin - cf T184176

Jan 7 2018, 11:31 PM · Analytics, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184176: Scap not working in Beta.

Is this problem related? I was about to go and report a separate bug, but...

Info: Applying configuration version '1515367485'
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.8.0-1~20180105205453.271 to 3.7.4-3+0~20180106122359.272~1.gbp3819c6 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Jan 7 2018, 11:27 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
Krenair placed T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing up for grabs.

I have a feeling that https://gerrit.wikimedia.org/r/402432 which we cherry-picked for T184239 may have fixed this. The hosts no longer have errors

Jan 7 2018, 11:22 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair renamed T184240: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing from Puppet broken on deployment-kafka-jump-[12] due to version of a package being missing to Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing.
Jan 7 2018, 11:20 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T174742: deployment-kafka01 - disk is full.

T184235 might be a repeat of this?

Jan 7 2018, 11:16 PM · Analytics-Kanban, Beta-Cluster-Infrastructure

Jan 6 2018

Krenair added a comment to T184245: Create some mechanism for instances in projects to modify the project Designate records.

Might also be worth looking into TSIG, dunno if what we run (pdns IIRC?) supports it in a way we can easily configure or not

Jan 6 2018, 5:01 PM · Operations, DNS, Beta-Cluster-reproducible, Cloud-VPS

Jan 5 2018

Krenair renamed T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances from Ferm/DNS library weirdness causing puppet errors on 12 deployment-prep instances to Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances.
Jan 5 2018, 11:31 PM · Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
Krenair added a comment to T132259: Deployment-prep hosts with puppet errors (tracking).

It's fine with me if you want to move them all to a particular workboard column instead of a tracking task

Jan 5 2018, 11:24 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair added a subtask for T132259: Deployment-prep hosts with puppet errors (tracking): T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances.
Jan 5 2018, 11:22 PM · Puppet, Tracking, Beta-Cluster-Infrastructure
Krenair added a parent task for T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances: T132259: Deployment-prep hosts with puppet errors (tracking).
Jan 5 2018, 11:22 PM · Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
Krenair renamed T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances from Ferm/DNS library weirdness on deployment-mediawiki boxes to Ferm/DNS library weirdness causing puppet errors on 12 deployment-prep instances.
Jan 5 2018, 11:03 PM · Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
Krenair added a comment to T153468: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances.

Gave up waiting for that (it's been almost a year), sent a message anyway and it's been held for moderation.

Jan 5 2018, 10:51 PM · Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
Krenair added a comment to T184239: Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch.

Actually the remaining ones appear to be T153468

Jan 5 2018, 10:39 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Krenair added a comment to T184239: Puppet broken on deployment-mediawiki07, deployment-imagescaler02, deployment-redis06, deployment-videoscaler01 due to prometheus exporter packages being missing in stretch.

Patch handles the errors, got some more ones on some of the hosts

Jan 5 2018, 10:28 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure