Krenair (Alex Monk)
Wikimedia volunteer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:34 PM (155 w, 3 d)
Availability
Available
IRC Nick
Krenair
LDAP User
Alex Monk
MediaWiki User
Krenair

I am a Wikimedia volunteer helping in various technical ways - e.g. MediaWiki development, software deployments to the Wikimedia cluster, and various other things. I am also an OTRS agent.

Some of my old VisualEditor work can be found under @AlexMonk-WMF instead

I have opinions on things, which do not necessarily represent those of any organisation I am or have previously been affiliated with.

Recent Activity

Sun, Sep 24

Krenair added a comment to T176576: http://tools.wmflabs.org/static-browser/ and http://tools.wmflabs.org/static gives Error 500.

Error 500 is never intentional, it always indicates something is wrong

Sun, Sep 24, 3:54 PM · Tools

Thu, Sep 21

Krenair added a comment to T93483: Add script_path to meta_p.wiki database.

I agree that it should be script_path and not api_url

Thu, Sep 21, 6:15 AM · Data-Services

Wed, Sep 20

Krenair added a comment to T172356: Decom RCStream in Beta Cluster.

and a DNS entry instance-deployment-stream.deployment-prep.wmflabs.org. see http://instance-deployment-stream.deployment-prep.wmflabs.org

Wed, Sep 20, 6:09 PM · Analytics-Kanban, Patch-For-Review, Beta-Cluster-Infrastructure, Wikimedia-Stream

Mon, Sep 18

Krenair added a comment to T138915: OTRS database is "too large".

I'm guessing this got worse with the recent spam problem

Mon, Sep 18, 5:38 PM · DBA, OTRS

Sun, Sep 17

Krenair added a comment to T174860: Define naming scheme for connecting to new wiki replica cluster.

I am curious as to what your designateclient hack is though

Sun, Sep 17, 9:55 PM · Patch-For-Review, cloud-services-team (Kanban), User-bd808, Data-Services
Krenair added a comment to T174860: Define naming scheme for connecting to new wiki replica cluster.

Managing this domain will be a bit of a pain, since it's in noauth-project. [...]

Sun, Sep 17, 9:53 PM · Patch-For-Review, cloud-services-team (Kanban), User-bd808, Data-Services
Krenair awarded T176090: wikitech-static sync failing a Manufacturing Defect? token.
Sun, Sep 17, 6:46 PM · MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), MediaWiki-Maintenance-scripts, Operations
Krenair added a project to T176090: wikitech-static sync failing: MediaWiki-Maintenance-scripts.
Sun, Sep 17, 6:46 PM · MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), MediaWiki-Maintenance-scripts, Operations

Sat, Sep 16

Krenair awarded T175962: Issue with maintenance script: SELECTing revisions with high rev_id is painfully slow a Orange Medal token.
Sat, Sep 16, 9:15 PM · MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), Patch-For-Review, Community-Tech-Sprint, DBA
Krenair added a comment to T176057: Designate api error when trying to create an A record in the web.db.svc.eqiad.wmflabs zone.

another job well done phabricator

Sat, Sep 16, 8:56 PM · Cloud-VPS, Upstream
Krenair added a project to T176057: Designate api error when trying to create an A record in the web.db.svc.eqiad.wmflabs zone: Upstream.

<Krenair> bd808, did you try the FQDN for that record you had trouble creating?
<Krenair> i.e. "name": "s1.web.db.svc.eqiad.wmflabs."
<bd808> I did not... would the records need to contain FQDNs?
<bd808> easy enough to try I guess
<Krenair> I'm not sure but try it
<Krenair> I'd report the error upstream regardless
<bd808> using an FQDN works

Sat, Sep 16, 8:56 PM · Cloud-VPS, Upstream
Krenair added a comment to T176042: Create amwikimedia.

Ok.

Sat, Sep 16, 8:02 PM · Patch-For-Review, Wiki-Setup (Create)
Krenair added a comment to T176042: Create amwikimedia.

who?

Sat, Sep 16, 7:38 PM · Patch-For-Review, Wiki-Setup (Create)
Krenair added a comment to T176042: Create amwikimedia.

Has someone from WMAM requested this?

Sat, Sep 16, 7:36 PM · Patch-For-Review, Wiki-Setup (Create)
Krenair added a comment to T175917: Beta cluster rights clarification.

The rights those groups grant on beta are fine to hand out (within reason) there, but I'd at least change the name of the group before granting it to avoid impersonation concerns

Sat, Sep 16, 5:39 PM · Beta-Cluster-Infrastructure

Thu, Sep 14

Krenair awarded T172035: Blockers for Wikimedia wiki domain renaming a Goat token.
Thu, Sep 14, 5:43 PM · Wikimedia-Site-requests

Wed, Sep 13

Krenair added a comment to T172035: Blockers for Wikimedia wiki domain renaming.

I think that token (and fire in general) is generally associated with destruction but sure.

Wed, Sep 13, 6:31 PM · Wikimedia-Site-requests

Tue, Sep 12

Krenair added a comment to T175643: Grant Bmansurov access to "Recommendation-api" Cloud VPS Project.

*mumbles something about phabricator's conflict detection or lack thereof*

Tue, Sep 12, 8:43 PM · VPS-Projects, Recommendation-API, Cloud-VPS
Krenair awarded T175643: Grant Bmansurov access to "Recommendation-api" Cloud VPS Project a Like token.
Tue, Sep 12, 8:43 PM · VPS-Projects, Recommendation-API, Cloud-VPS
Krenair added a comment to T175643: Grant Bmansurov access to "Recommendation-api" Cloud VPS Project.

I did some searching around and it looks to me like DarTar's approval is good for this project

Tue, Sep 12, 8:43 PM · VPS-Projects, Recommendation-API, Cloud-VPS
Krenair added a comment to T175643: Grant Bmansurov access to "Recommendation-api" Cloud VPS Project.

schana was on Phabricator under a week ago talking about a request they made in relation to this project. Is this urgent?

Tue, Sep 12, 8:26 PM · VPS-Projects, Recommendation-API, Cloud-VPS

Mon, Sep 11

Krenair updated subscribers of T175643: Grant Bmansurov access to "Recommendation-api" Cloud VPS Project.

adding current admins to process request

Mon, Sep 11, 11:05 PM · VPS-Projects, Recommendation-API, Cloud-VPS

Thu, Sep 7

Krenair added a comment to T162910: Update Collection not to use deprecated wfSetupSession call.

https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/GlobalFunctions.php;5ec0132821f4dca98a53b466a3d35b7a2853c616$2952-2955

Thu, Sep 7, 9:23 PM · Readers-Web-Backlog (Tracking), Collection, Technical-Debt, Wikimedia-log-errors

Tue, Sep 5

Krenair added a comment to T166712: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation.

<yuvipanda> Krenair: you should ask some of the cloud team people about it :)

Tue, Sep 5, 6:16 PM · cloud-services-team (Kanban), Analytics

Sun, Sep 3

Krenair added a comment to T174850: wikistream.wmflabs.org down - unable to ssh to ws-web.

I believe that auth-related service would likely be nslcd/nscd (I forget which).

Sun, Sep 3, 6:09 PM · Cloud-VPS
Krenair added a comment to T174859: Image File is not working on hi.wikiversity.

Yes, you'd need a separate task to report that.

Sun, Sep 3, 2:51 AM · media-storage, Hindi-Sites
Krenair closed T174859: Image File is not working on hi.wikiversity as Resolved.

That fixed it, the URL provided above by @Urbanecm works now.

Sun, Sep 3, 1:00 AM · media-storage, Hindi-Sites

Sat, Sep 2

Krenair updated subscribers of T174859: Image File is not working on hi.wikiversity.
Sat, Sep 2, 11:27 PM · media-storage, Hindi-Sites
Krenair added a comment to T174859: Image File is not working on hi.wikiversity.

addWiki.php is supposed to run setZoneAccess.php

Sat, Sep 2, 11:26 PM · media-storage, Hindi-Sites
Krenair added a comment to T173968: Preview the wikitext message.

could also just stick it in action=parse

Sat, Sep 2, 10:17 PM · Huggle
Krenair added a comment to T174867: Wikivoyage is missing from PAWS system user-config.py.

also wikiboots instead of wikibooks?
what about wikinews?

Sat, Sep 2, 9:29 PM · PAWS
Krenair added a comment to T174866: tool extreg-wos not updating its table.

I changed the title on them. Let's see if it updates now.

Sat, Sep 2, 9:21 PM · Tools
Krenair renamed T174483: Convert MagicNoCache to use extension registration from Convert MagicNoCache to use extension registration to Convert MagicNoCache to use extension registration.
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-Other
Krenair renamed T174657: Convert PageNotice to use extension registration from Convert Extension:PageNotice to use extension registration to Convert PageNotice to use extension registration.
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-PageNotice
Krenair renamed T174661: Convert SimpleChanges to use extension registration from Convert Extension:SimpleChanges to use extension registration to Convert SimpleChanges to use extension registration.
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-Other
Krenair renamed T174663: Convert SubpageFun to use extension registration from Convert Extension:SubpageFun to use extension registration to Convert SubpageFun to use extension registration.
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-Other
Krenair renamed T174456: Convert WebChat to use extension registration from Convert Extension:WebChat to use extension registration to Convert WebChat to use extension registration.
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-Other
Krenair renamed T174478: Convert ArticleToCategory2 to use extension registration from Convert Extension:ArticleToCategory2 to use extension registration to Convert ArticleToCategory2 to use extension registration.
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-Other
Krenair renamed T174529: Convert GroupsSidebar to use extension registration. from Convert Extension:GroupsSidebar to use extension registration. to Convert GroupsSidebar to use extension registration..
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-Other
Krenair renamed T174530: Convert MsLinks to use extension registration. from Convert Extension:MsLinks to use extension registration. to Convert MsLinks to use extension registration..
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-Other
Krenair renamed T174483: Convert MagicNoCache to use extension registration from Convert Extension:MagicNoCache to use extension registration to Convert MagicNoCache to use extension registration.
Sat, Sep 2, 9:18 PM · Patch-For-Review, MediaWiki-extensions-Other
Krenair renamed T171819: Convert FlickrAPI to use extension registration from Update FlickrAPI extension to use registration to Convert FlickrAPI to use extension registration.
Sat, Sep 2, 9:17 PM · Patch-For-Review
Krenair renamed T173058: Convert AdminLinks to use extension registration from Convert Extension:AdminLinks to use extension registration to Convert AdminLinks to use extension registration.
Sat, Sep 2, 9:17 PM · Patch-For-Review, MediaWiki-extensions-AdminLinks
Krenair added a comment to T174866: tool extreg-wos not updating its table.

A lot of tasks are like that - see the related objects on T98668

Sat, Sep 2, 9:10 PM · Tools
Krenair added a comment to T174861: CAPTCHA ineffective, consider using reCAPTCHA.

My understanding is that it was rejected over 6 years ago, so this is probably a duplicate of a wontfix.

Sat, Sep 2, 7:19 PM · Wikimedia-Site-requests

Thu, Aug 31

Krenair added a comment to T174746: deployment-imagescaler0[12] puppet broken due to missing nginx lua support.
  • Installed libnginx-mod-http-lua
  • Manually installed /etc/nginx/prometheus.lua file to avoid puppet catch-22, need to make the nginx service depend on that file somehow?
  • Killed existing deployment-imagescaler01 (was fine on -imagescaler02) nginx processes that were refusing to stop (preventing a restart due to port 8800 being in use)
  • Puppet now works
Thu, Aug 31, 11:31 PM · Patch-For-Review, Thumbor, Performance-Team, Beta-Cluster-Infrastructure
Krenair updated the task description for T174746: deployment-imagescaler0[12] puppet broken due to missing nginx lua support.
Thu, Aug 31, 11:22 PM · Patch-For-Review, Thumbor, Performance-Team, Beta-Cluster-Infrastructure
Krenair created T174746: deployment-imagescaler0[12] puppet broken due to missing nginx lua support.
Thu, Aug 31, 11:20 PM · Patch-For-Review, Thumbor, Performance-Team, Beta-Cluster-Infrastructure
Krenair updated the task description for T174742: deployment-kafka01 - disk is full.
Thu, Aug 31, 10:59 PM · Analytics, Beta-Cluster-Infrastructure
Krenair created T174742: deployment-kafka01 - disk is full.
Thu, Aug 31, 10:57 PM · Analytics, Beta-Cluster-Infrastructure
Krenair claimed T162977: Create top-level domain for a project on project creation.
Thu, Aug 31, 10:34 PM · Patch-For-Review, Cloud-Services
Krenair added a comment to T162977: Create top-level domain for a project on project creation.

I think we'll have to do the makedomain-style subdomain creation dance with wmflabsdotorg
Actually let's just import that code and call it?

Thu, Aug 31, 10:23 PM · Patch-For-Review, Cloud-Services
Krenair added a comment to T174720: letsencrypt::cert::integrated and non-http servers.

acme_tiny - modules/letsencrypt/files/acme_tiny.py in puppet

Thu, Aug 31, 9:24 PM · Patch-For-Review, Operations
Krenair updated subscribers of T166712: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation.
Thu, Aug 31, 5:57 PM · cloud-services-team (Kanban), Analytics
Krenair added a comment to T166712: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation.

modules/toollabs/files/log-command-invocation in puppet

Thu, Aug 31, 5:55 PM · cloud-services-team (Kanban), Analytics

Wed, Aug 30

Krenair added a comment to T168677: Add new Cloud Services domains to public suffix list.

I just checked and that domain doesn't even have the right nameservers (it's set to ns[0-2].wikimedia.org), so this might be a bit premature :)

Wed, Aug 30, 9:55 PM · Cloud-Services
Krenair added a watcher for Cloud-Services: Krenair.
Wed, Aug 30, 9:48 PM
Krenair added a watcher for Cloud-VPS (Project-requests): Krenair.
Wed, Aug 30, 9:48 PM
Krenair added a comment to T166845: monitor some things on all Cloud instances (discussion).

Like shinken?

Yes! But as I understand it Shinken currently only monitors select projects... I'd like to monitor a narrower set of things on every single instance.

Wed, Aug 30, 9:22 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Krenair added a comment to T168433: Deprecate DSA (ssh-dss) SSH keys for Labs users.

Currently 178 users with these

Wed, Aug 30, 9:15 PM · Cloud-VPS, Toolforge, cloud-services-team (Kanban)
Krenair added a comment to T168433: Deprecate DSA (ssh-dss) SSH keys for Labs users.

Audit script:

import ldap
ldap_conn = ldap.initialize('ldap://ldap-labs.eqiad.wikimedia.org:389')
lc = ldap.controls.libldap.SimplePagedResultsControl(size=10000, cookie='')
while True:
    rtype, rdata, rmsgid, serverctrls = ldap_conn.result3(ldap_conn.search_ext(
        'ou=people,dc=wikimedia,dc=org',
        ldap.SCOPE_ONELEVEL,
        filterstr='(&(objectClass=ldapPublicKey)(sshPublicKey=*))',
        attrlist=['uid', 'sshPublicKey'],
        serverctrls=[lc]
    ))
    for userDN, userAttributes in rdata:
        badkeys = []
        for key in userAttributes['sshPublicKey']:
            if key.startswith('ssh-dss'):
                badkeys.append(key)
        if len(badkeys) > 0:
            print(userAttributes['uid'][0], len(badkeys))
    for control in serverctrls:
        if control.controlType == ldap.controls.SimplePagedResultsControl.controlType:
            lc.cookie = control.cookie
    if not lc.cookie:
        break
Wed, Aug 30, 9:14 PM · Cloud-VPS, Toolforge, cloud-services-team (Kanban)
Krenair added a comment to T171136: dss keys disabled prematurely.

I just tested this and it seems to work on all the non-stretch instances that I tried. Where did you test?

Wed, Aug 30, 8:20 PM · Cloud-VPS, cloud-services-team
Krenair edited projects for T150828: Horizon prefix puppet dialog puts you in wrong prefix after you create a new prefix, added: Horizon; removed Cloud-Services.
Wed, Aug 30, 8:06 PM · Horizon
Krenair added a parent task for T166349: Support per project user email address $project-admins@wmflabs.org: T174608: Improve SMTP inbound/outbound services.
Wed, Aug 30, 7:58 PM · Mail, Cloud-Services
Krenair added a subtask for T174608: Improve SMTP inbound/outbound services: T166349: Support per project user email address $project-admins@wmflabs.org.
Wed, Aug 30, 7:58 PM · Epic, Cloud-Services
Krenair added a comment to T41785: Create a labs SMTP smarthost.

I think so

Wed, Aug 30, 7:56 PM · Operations, Cloud-Services, Mail
Krenair added a comment to T166712: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation.

Ping @Nuria

Wed, Aug 30, 7:52 PM · cloud-services-team (Kanban), Analytics
Krenair removed a project from T152767: Missing Labs hiera entry in labs-private repo: Patch-For-Review.
Wed, Aug 30, 7:47 PM · Operations, Cloud-Services
Krenair removed a project from T153608: Migrate references from $instance.eqiad.wmflabs to $instance.$project.eqiad.wmflabs: Patch-For-Review.
Wed, Aug 30, 7:46 PM · Puppet, Cloud-Services
Krenair added a project to T154099: Deactivate repository labs/invisible-unicorn: Gerrit.
Wed, Aug 30, 7:43 PM · Repository-Admins, Gerrit, Cloud-Services
Krenair added a project to T166349: Support per project user email address $project-admins@wmflabs.org: Mail.
Wed, Aug 30, 7:39 PM · Mail, Cloud-Services
Krenair edited projects for T168349: enwiki_p logging vs logging_userindex returning dramatically different results, added: Data-Services; removed Cloud-Services.
Wed, Aug 30, 7:37 PM · Data-Services, DBA
Krenair edited projects for T147859: Add a note to the launch instance window about instance names being global, added: Horizon; removed Cloud-VPS, Cloud-Services.

upstream?

Wed, Aug 30, 7:18 PM · Horizon
Krenair added a comment to T41785: Create a labs SMTP smarthost.

! In T41785#3567277, @Krenair wrote:

You mean on separate physical hosts, right? I think we're still limited to eqiad if it is to be a Cloud VPS instance :)

Nah, this doesn't seem like a workload that justifies physical hardware. I just mean that if we can avoid a SPOF that's a good thing, and if multi-site is possible all the better.

Wed, Aug 30, 7:06 PM · Operations, Cloud-Services, Mail
Krenair awarded T162977: Create top-level domain for a project on project creation a Like token.
Wed, Aug 30, 6:48 PM · Patch-For-Review, Cloud-Services
Krenair added a comment to T166845: monitor some things on all Cloud instances (discussion).

Like shinken?

Wed, Aug 30, 6:44 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Krenair added a comment to T171570: Rename database labswiki to wikitech.

Surely it'd be wikitechwiki anyway :p

Wed, Aug 30, 6:41 PM · cloud-services-team, DBA, wikitech.wikimedia.org
Krenair added a comment to T167973: move wikitech and labstestwiki to s3 (needs discussion).

I cannot think any reason why wikitech's database (and the similar labstestwiki) is not part of s3

Historically, wikitech was separate because it held important documentation that ops would want to continue to access if most of the site went down. Now we have wikitech-static, so...

Wed, Aug 30, 6:40 PM · Data-Services, wikitech.wikimedia.org, cloud-services-team, DBA
Krenair added a comment to T41785: Create a labs SMTP smarthost.

Next tricky step is that <projectname>.wmflabs.org does not exist in DNS by default.

Wed, Aug 30, 6:29 PM · Operations, Cloud-Services, Mail
Krenair added a comment to T41785: Create a labs SMTP smarthost.

For sure! Maybe a pair of instances in different locations for durability?

Wed, Aug 30, 6:25 PM · Operations, Cloud-Services, Mail
Krenair added a comment to T174596: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour.

See also T167357 where this task will probably become obsolete, I just wanted to document the effect of this really.

Wed, Aug 30, 6:19 PM · netops, Operations, Cloud-VPS
Krenair updated the task description for T174596: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour.
Wed, Aug 30, 6:18 PM · netops, Operations, Cloud-VPS
Krenair created T174596: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour.
Wed, Aug 30, 6:17 PM · netops, Operations, Cloud-VPS

Tue, Aug 29

Krenair added a comment to T174488: Request for additional edit permissions on wikitech.wikimedia.org for dr0ptp4kt.

The edit to the page after that one doesn't make a lot of sense...

Tue, Aug 29, 8:16 PM · User-bd808, Cloud-Services, cloud-services-team
Krenair added a comment to T174488: Request for additional edit permissions on wikitech.wikimedia.org for dr0ptp4kt.

What kind of edits would you like to make?

Tue, Aug 29, 7:39 PM · User-bd808, Cloud-Services, cloud-services-team
Krenair added a comment to T168765: Create Wikiversity Hindi.

Was the maintain-views step not performed?

MariaDB [hiwikiversity_p]> show tables;
Empty set (0.00 sec)
Tue, Aug 29, 1:22 PM · Wiki-Setup (Create), Analytics-Wikistats, Analytics, MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), Wikidata, User-Urbanecm, Patch-For-Review, Hindi-Sites, Wikimedia-Language-setup, Operations

Aug 26 2017

Krenair added a comment to T172560: "Loss of session data" on Beta Cluster.

Is there any chance it might be caused by redis misbehaviour? I vaguely recall sessions being stored in object cache, in which case this might be another symptom of the problem in T173646 ?

Good call.

I don't know if that is the cause, but they are indeed stored in Redis at least on Beta (and as I recall production too): P5918

Aug 26 2017, 11:20 AM · MediaWiki-extensions-CentralAuth, MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), Patch-For-Review, Beta-Cluster-Infrastructure, MediaWiki-Authentication-and-authorization

Aug 24 2017

Krenair added a comment to T172560: "Loss of session data" on Beta Cluster.

Is there any chance it might be caused by redis misbehaviour? I vaguely recall sessions being stored in object cache, in which case this might be another symptom of the problem in T173646 ?

Aug 24 2017, 11:07 PM · MediaWiki-extensions-CentralAuth, MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), Patch-For-Review, Beta-Cluster-Infrastructure, MediaWiki-Authentication-and-authorization

Aug 23 2017

Krenair added a comment to T154371: Review and deploy Timeless skin.

Good luck with that: https://incubator.wikimedia.org/wiki/Special:RandomByTest?testwiki=Wp/ase

Aug 23 2017, 9:15 PM · wikimedia-extension-review-queue, Patch-For-Review, User-notice, Wikimedia-Extension-setup, Community-consensus-needed, Timeless
Krenair added a comment to T173646: FancyCaptcha reloading sometimes broken in beta due to differing caches.

I didn't fake my console output above. I wasn't piping these IDs into redis-cli anyway, I put them in manually. They appeared exactly as they did in my phabricator code snippet above.
I wouldn't pipe straight in from curl to localhost because you need to check that a captcha generated by one machine shows up when the other machine queries the cache
This, however, demonstrates the problem (again):

krenair@deployment-mediawiki04:~$ for i in {1..10}; do ID=`curl 'http://deployment-mediawiki05/w/api.php' -H 'Host: en.wikipedia.beta.wmflabs.org' --data 'action=fancycaptchareload&format=json' 2>/dev/null | jq --raw-output '.fancycaptchareload.index'`; redis-cli -a $password_here -s /var/run/nutcracker/redis_eqiad.sock get enwiki:captcha:$ID; done
(nil)
(nil)
"a:4:{s:4:\"salt\";s:8:\"eccb8227\";s:4:\"hash\";s:16:\"e9d9c5e6958e32ee\";s:6:\"viewed\";b:0;s:5:\"index\";s:9:\"586427543\";}"
"a:4:{s:4:\"salt\";s:8:\"3b0e1f3f\";s:4:\"hash\";s:16:\"b8ec26025159982a\";s:6:\"viewed\";b:0;s:5:\"index\";s:9:\"779560224\";}"
"a:4:{s:4:\"salt\";s:8:\"a3fe4b71\";s:4:\"hash\";s:16:\"51c249a3fc6e1a6e\";s:6:\"viewed\";b:0;s:5:\"index\";s:9:\"151118027\";}"
(nil)
"a:4:{s:4:\"salt\";s:8:\"63a67563\";s:4:\"hash\";s:16:\"8d8be10b9ca8671b\";s:6:\"viewed\";b:0;s:5:\"index\";s:10:\"1117844497\";}"
"a:4:{s:4:\"salt\";s:8:\"68f07044\";s:4:\"hash\";s:16:\"0a05113e069dae94\";s:6:\"viewed\";b:0;s:5:\"index\";s:10:\"1020645645\";}"
"a:4:{s:4:\"salt\";s:8:\"815fd2ed\";s:4:\"hash\";s:16:\"c7d5ec5d75e7cfb7\";s:6:\"viewed\";b:0;s:5:\"index\";s:9:\"854439982\";}"
"a:4:{s:4:\"salt\";s:8:\"d9419c5d\";s:4:\"hash\";s:16:\"f28c71919060cad5\";s:6:\"viewed\";b:0;s:5:\"index\";s:10:\"1146404443\";}"
krenair@deployment-mediawiki05:~$ for i in {1..10}; do ID=`curl 'http://deployment-mediawiki04/w/api.php' -H 'Host: en.wikipedia.beta.wmflabs.org' --data 'action=fancycaptchareload&format=json' 2>/dev/null | jq --raw-output '.fancycaptchareload.index'`; redis-cli -a $password_here -s /var/run/nutcracker/redis_eqiad.sock get enwiki:captcha:$ID; done
(nil)
(nil)
"a:4:{s:4:\"salt\";s:8:\"5c0ed654\";s:4:\"hash\";s:16:\"af173e59bfe70595\";s:6:\"viewed\";b:0;s:5:\"index\";s:10:\"1542499225\";}"
"a:4:{s:4:\"salt\";s:8:\"b1206b38\";s:4:\"hash\";s:16:\"063b80acbdfbb1f5\";s:6:\"viewed\";b:0;s:5:\"index\";s:10:\"1602094256\";}"
(nil)
"a:4:{s:4:\"salt\";s:8:\"77595383\";s:4:\"hash\";s:16:\"e3eae84215acd764\";s:6:\"viewed\";b:0;s:5:\"index\";s:10:\"2091943011\";}"
(nil)
(nil)
"a:4:{s:4:\"salt\";s:8:\"4d1bd27e\";s:4:\"hash\";s:16:\"2c134012bb69d2a4\";s:6:\"viewed\";b:0;s:5:\"index\";s:9:\"429683122\";}"
"a:4:{s:4:\"salt\";s:8:\"469ae441\";s:4:\"hash\";s:16:\"ceceb3ca4239fbd7\";s:6:\"viewed\";b:0;s:5:\"index\";s:10:\"1953322325\";}"
Aug 23 2017, 8:17 PM · MediaWiki-Cache, Beta-Cluster-reproducible, ConfirmEdit (CAPTCHA extension)

Aug 22 2017

Krenair added a comment to T173646: FancyCaptcha reloading sometimes broken in beta due to differing caches.

by the way, you can get a new one of these IDs generated with curl 'http://localhost/w/api.php' -H 'Host: en.wikipedia.beta.wmflabs.org' --data 'action=fancycaptchareload&format=json' 2>/dev/null | jq '.fancycaptchareload.index'

Aug 22 2017, 11:24 PM · MediaWiki-Cache, Beta-Cluster-reproducible, ConfirmEdit (CAPTCHA extension)
Krenair added a comment to T173646: FancyCaptcha reloading sometimes broken in beta due to differing caches.

I didn't use mwscript eval.php, I used mwrepl.

Aug 22 2017, 11:11 PM · MediaWiki-Cache, Beta-Cluster-reproducible, ConfirmEdit (CAPTCHA extension)

Aug 21 2017

Krenair closed T173571: Disk full on deployment-jobrunner02 as Resolved.

hopefully that should stop this happening again

Aug 21 2017, 6:35 PM · Patch-For-Review, Beta-Cluster-Infrastructure
Krenair added a comment to T173602: Revert "norsk bokmål" -> "norsk" change.

Although I consider changing the no.wikipedia.org to nb.wikipedia.org the best solution in an ideal world, I'm only in favor of such a change if this does not cause any problems for the project in terms of dead links and such.

If it were to be done then you'd get redirects e.g. https://be-x-old.wikipedia.org/wiki/Main_Page goes to https://be-tarask.wikipedia.org/wiki/Main_Page

Aug 21 2017, 12:51 AM · Wikimedia-Site-requests

Aug 20 2017

Krenair added a comment to T173602: Revert "norsk bokmål" -> "norsk" change.

I renamed be-x-old to be-tarask a couple of years ago and there are still unfixed problems with it (e.g. T111822, T112426, T111853, T111852). I'm not sure it's a good idea to be considering an no -> nb rename right now

Aug 20 2017, 8:11 PM · Wikimedia-Site-requests
Krenair reopened T173665: error while loading shared libraries: libircclient.so: cannot open shared object file as "Open".
Aug 20 2017, 8:00 PM · Huggle
Krenair added a comment to T173665: error while loading shared libraries: libircclient.so: cannot open shared object file.

Does installing it fix the problem?

Aug 20 2017, 4:18 PM · Huggle
Krenair added a comment to T173665: error while loading shared libraries: libircclient.so: cannot open shared object file.

Do you have the libircclient1 package installed?

Aug 20 2017, 2:38 PM · Huggle
Krenair added a comment to T173646: FancyCaptcha reloading sometimes broken in beta due to differing caches.

When you run that PHP on -mediawiki05 you get the same as -tin. Here's what it looks like against the redis boxes directly:

Aug 20 2017, 3:36 AM · MediaWiki-Cache, Beta-Cluster-reproducible, ConfirmEdit (CAPTCHA extension)
Krenair added a project to T173646: FancyCaptcha reloading sometimes broken in beta due to differing caches: MediaWiki-Cache.

might be Beta-Cluster-Infrastructure

Aug 20 2017, 3:33 AM · MediaWiki-Cache, Beta-Cluster-reproducible, ConfirmEdit (CAPTCHA extension)