Krenair (Alex Monk)
Wikimedia volunteer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:34 PM (215 w, 3 d)
Availability
Available
IRC Nick
Krenair
LDAP User
Alex Monk
MediaWiki User
Krenair [ Global Accounts ]

I am a Wikimedia volunteer helping in various technical ways. These days it's usually Beta Cluster, Cloud VPS, or Operations related. Since 2012 I've spent significant amounts of time involved in MediaWiki development, software deployments to the Wikimedia cluster, OTRS (email response to e.g. info-en@wikimedia.org addresses), and various other things.

Some of my old VisualEditor and other work (2014-2016) can be found under @AlexMonk-WMF instead.

I have opinions on things, which do not necessarily represent those of any organisation I am, have previously been, or will in the future be affiliated with.

Recent Activity

Yesterday

Krenair updated subscribers of T208431: Add Marble to `wmf` LDAP group.

maybe the other one was created via toolsadmin?

Mon, Nov 19, 9:50 PM · Patch-For-Review, Security-Team, LDAP-Access-Requests
Krenair added a parent task for T209841: LibreNMS IRC bot registration: T48252: IRC bots account pending nickserv registration (tracking).
Mon, Nov 19, 3:35 PM · Patch-For-Review, monitoring
Krenair added a subtask for T48252: IRC bots account pending nickserv registration (tracking): T209841: LibreNMS IRC bot registration.
Mon, Nov 19, 3:35 PM · Tracking
Krenair added a comment to T171188: Move the main WMCS puppetmaster into the Labs realm.

JFTR, I don't know what cloudinfra-puppetmaster-01 is. Maybe @Krenair or someone else set up that?

Mon, Nov 19, 2:22 PM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations

Sat, Nov 17

Krenair added a comment to T209770: Unable to login to Striker.
krenair@bastion-01:~$ ldapsearch -xLLL "cn=DannyS712 bot" uid cn
dn: uid=dannys712,ou=people,dc=wikimedia,dc=org
cn: DannyS712 bot
uid: dannys712
Sat, Nov 17, 11:07 PM · Toolforge
Krenair renamed T209770: Unable to login to Striker from Unable to login to toolforge to Unable to login to Striker.
Sat, Nov 17, 11:06 PM · Toolforge
Krenair added a comment to T209770: Unable to login to Striker.

Did you try DannyS712 bot instead of DannyS712_bot, i.e. with the underscore changed to a space?

Sat, Nov 17, 11:05 PM · Toolforge
Krenair added a comment to T209769: redirect toolserver.org to eqiad1-r .

Come the thought of it why is it handled by prod auth DNS if all the actual servers behind it are in Cloud VPS?

Sat, Nov 17, 11:00 PM · Patch-For-Review, cloud-services-team (Kanban)
Krenair added a comment to T209769: redirect toolserver.org to eqiad1-r .

I've updated designate DNS records to point to the new IP but we may have things registered upstream or elsewhere.

Sat, Nov 17, 10:59 PM · Patch-For-Review, cloud-services-team (Kanban)

Thu, Nov 15

Krenair committed rOSCCe28ae8871f3a: certcentral: split base path in config and certificates path (authored by Vgutierrez).
certcentral: split base path in config and certificates path
Thu, Nov 15, 3:02 PM

Wed, Nov 14

Krenair added a project to T209474: addWiki broken for wiktionaries: Addwiki.
Wed, Nov 14, 12:35 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), User-Addshore, Addwiki, Cognate, MediaWiki-extensions-WikimediaMaintenance
Krenair added a comment to T209460: CloudVPS: our ideal future model.

The document is: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron_ideal_model (edits welcome).

I put in some basic ones: https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Neutron_ideal_model&diff=1808555&oldid=1808492

Wed, Nov 14, 10:06 AM · Operations, cloud-services-team (Kanban), Epic

Tue, Nov 13

Krenair added a comment to T208636: Give users a download of their "User Data'.

Okay. Will it be clear to the users that this is essentially a beta with data possibly missing? And that we know certain types of data are missing (deliberately excluded) and that some data may be missing in error?

Tue, Nov 13, 9:27 PM · Data-Portability, Community-Tech
Krenair updated subscribers of T208101: Migrate deployment-prep to eqiad1.

I think you mean @dduvall :)

Tue, Nov 13, 9:08 PM · Patch-For-Review, Release-Engineering-Team, Beta-Cluster-Infrastructure, Epic, Cloud-Services
Krenair added a comment to T208636: Give users a download of their "User Data'.

Okay but crucially it's not a data dump of *all* data relating to them, is it? There are some deliberate omissions?

Tue, Nov 13, 9:04 PM · Data-Portability, Community-Tech
Krenair added a watcher for SEO: Krenair.
Tue, Nov 13, 3:27 PM
Krenair added a comment to T209374: dns-floating-ip-updater.py not working with 155.80.208.in-addr.arpa.

<Krenair> There should be two instances of that cron running
<Krenair> One for each region
<Krenair> Each region should not touch records for the other region
<Krenair> So you should expect a *lot* of 'Not handling something.x because it doesn't end with y'

Tue, Nov 13, 3:13 PM · Patch-For-Review, cloud-services-team (Kanban)
Krenair added a comment to T209298: Server Access for 3 formal collaborators.

Are collaborators going to be able to log in to officewiki?
bastiononly hasn't existed for years.

Tue, Nov 13, 1:47 PM · SRE-Access-Requests, Epic, Research-Programs, Operations
Restricted Application added a project to T174388: LoginNotify should inform users of the IP address of failed login attempts to their account: Growth-Team.
Tue, Nov 13, 12:28 PM · Growth-Team, Patch-For-Review, Collaboration-Team-Triage, Notifications, User-Huji, Community-Tech, Privacy, WMF-Legal, MediaWiki-extensions-LoginNotify
Krenair added a comment to T208636: Give users a download of their "User Data'.

It should probably be made very clear to users exactly what data they are not being provided. E.g. it sounds like this will not include their CU data, and various deleted/suppressed things about them that are actually still held by the site.

Tue, Nov 13, 12:17 PM · Data-Portability, Community-Tech
Krenair added a comment to T174596: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour.

BTW, I'm focusing on the eqiad1 deployment setting. Not paying much attention to the setting in main, since the new is the one we will be living with for upcoming times.
Please @Krenair @ayounsi do all your tests and checks from VMs in this deployment.

Tue, Nov 13, 9:56 AM · cloud-services-team (Kanban), netops, Cloud-VPS, Operations
Krenair added a comment to T209031: Not able to scoop comment table in labs for mediawiki reconstruction process.
  • Access to underlying tables - We could query the underlying tables, and that would bypass any performance problems we have with the views. We would duplicate the sanitizing logic from the views, and maintain it to be always the same as it is in cloud db. This would require special permissions to the cloud db.
Tue, Nov 13, 9:46 AM · Analytics-Kanban, DBA, Data-Services, Analytics

Mon, Nov 12

Krenair added a comment to T209119: Create a beta host.

It may be worth making the puppet manifest that can be told which environment it's in and choose appropriately.

Mon, Nov 12, 11:03 PM · Quarry

Sun, Nov 11

Liuxinyu970226 awarded T100373: U2F integration for Extension:OATHAuth a Like token.
Sun, Nov 11, 6:52 AM · MediaWiki-extensions-OATHAuth

Sat, Nov 10

Krenair closed T209217: Merge WikiEditor into the MediaWiki Core as Declined.
Sat, Nov 10, 8:00 PM · MediaWiki-Page-editing, WikiEditor
Krenair edited projects for T209217: Merge WikiEditor into the MediaWiki Core, added: MediaWiki-Page-editing; removed MediaWiki-General-or-Unknown.
Sat, Nov 10, 7:05 PM · MediaWiki-Page-editing, WikiEditor
Krenair added a project to T209217: Merge WikiEditor into the MediaWiki Core: MediaWiki-General-or-Unknown.

Yeah I don't know if it's necessary to merge it into the core repository. It's bundled already, people can disable it and get a plain text box if they want.

Sat, Nov 10, 7:05 PM · MediaWiki-Page-editing, WikiEditor
Krenair added a comment to T204160: Create a security issue task type with additional attributes.

Subscribers are the most borderline - non-security-members who are more familiar with the relevant community inviting people knowing more about an issue is a good thing, OTOH it can result (even with good intentions) in the task being visible to too many people / potentially easily accessible to an attacker who can steal wiki accounts. Not sure about that one.

Sat, Nov 10, 5:44 PM · Release-Engineering-Team (Kanban), Security-Team, User-MModell, Phabricator
Krenair added a project to T209202: Page Move Internal Error: MediaWiki-General-or-Unknown.
Sat, Nov 10, 1:37 AM · Multi-Content-Revisions, MediaWiki-General-or-Unknown, Wikimedia-production-error
Krenair added a project to T209202: Page Move Internal Error: Wikimedia-production-error.

need someone to pull the details for that exception hash

Sat, Nov 10, 1:37 AM · Multi-Content-Revisions, MediaWiki-General-or-Unknown, Wikimedia-production-error

Fri, Nov 9

Krenair added a watcher for ScienceSource: Krenair.
Fri, Nov 9, 10:08 PM
Krenair added a comment to T153468: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs.

So that seems to work. If we can get updated ferm and libnet-dns-perl packages this might be done.

Fri, Nov 9, 9:34 PM · monitoring, Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
Krenair added a comment to T153468: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs.

it doesn't look like there'll be any movement on that problem inside Net::DNS any time soon

Fri, Nov 9, 9:01 PM · monitoring, Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
Krenair awarded T199207: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) a Evil Spooky Haunted Tree token.
Fri, Nov 9, 8:06 PM · Phabricator (Upstream), Upstream, User-MModell, Release-Engineering-Team (Kanban), User-Ryasmeen
Krenair updated subscribers of T199207: 404 on workboard for an existing project (due to custom filter applied which did not exist in database).

Fixed OOUI (noticed by @Volker_E)

Fri, Nov 9, 8:04 PM · Phabricator (Upstream), Upstream, User-MModell, Release-Engineering-Team (Kanban), User-Ryasmeen
Krenair awarded T209166: analytics - hadoop-worker-3 can't talk to kdc a Evil Spooky Haunted Tree token.
Fri, Nov 9, 7:21 PM · Cloud-VPS, cloud-services-team (Kanban)
Krenair updated the task description for T207536: Move various support services for Cloud VPS currently in prod into their own instances.
Fri, Nov 9, 6:18 PM · cloud-services-team (Kanban), Operations, Cloud-VPS
Krenair updated the task description for T207536: Move various support services for Cloud VPS currently in prod into their own instances.
Fri, Nov 9, 6:17 PM · cloud-services-team (Kanban), Operations, Cloud-VPS
Krenair updated the task description for T207536: Move various support services for Cloud VPS currently in prod into their own instances.
Fri, Nov 9, 6:15 PM · cloud-services-team (Kanban), Operations, Cloud-VPS
Krenair added a comment to T207536: Move various support services for Cloud VPS currently in prod into their own instances.
  • OpenStack Horizon (dashboard)
  • Wikimedia Striker (toolsadmin)

Both of these services receive developer account (LDAP) authentication credentials from end users (the usernames and passwords that also allow access to things like Gerrit, Wikitech, Phabricator). They also both hold credentials for highly privileged accounts in other services (LDAP, Keystone). I think this means that we should not be considering hosting these services inside the Cloud VPS environment as the terms of use and sane operational practices for Cloud VPS prohibit collecting LDAP passwords. Am I missing some subtly here or do others agree?

Fri, Nov 9, 6:15 PM · cloud-services-team (Kanban), Operations, Cloud-VPS
Krenair added a comment to T153468: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs.

Ferm merged my pull request for handling of NOERROR empty responses. So now if we can get the next release of ferm, and add the trailing full stops (it doesn't look like there'll be any movement on that problem inside Net::DNS any time soon, if ever), we should be good.

What do you mean with trailing full stops? What change would need to be made to a ferm rule to fall back gracefully in combination with the patch of yours which was merged upstream?

Fri, Nov 9, 4:27 PM · monitoring, Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
Krenair added a comment to T208909: [Bug] Update old nonuniformly distributed page_random values.

I have no idea what this task is about but please do not think of running the query on the description AS IS. You need to create batching to update 100-1000 rows on each transaction and select (update) based on the primary key. Not sure if there is somewhere else other than page that stores supposedly random values. This is such a common task that we should have "perPageBatched" and "perRevisionBatched" skeletons/classes precreated on mediawiki/maintenance.

Fri, Nov 9, 1:47 AM · MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), Patch-For-Review, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), DBA, MediaWiki-General-or-Unknown

Thu, Nov 8

Krenair added a comment to T209124: stashbot ignores some public access request tickets because it considers them to be security tickets.

If something private actually gets caught in that check then something has already failed. stashbot should not be given (or left with) access to private security tasks.

Thu, Nov 8, 11:28 PM · Stashbot
Krenair added a comment to T209031: Not able to scoop comment table in labs for mediawiki reconstruction process.

@Krenair: we are looking how to best import the public dataset from labs, we have already looked into scooping data from the non public data hosts and the sanitization is a lot harder than you might (by no means as simple as "runing your own views") so we need to come up with a strategy for us to scoop data from labs efficiently.

Thu, Nov 8, 4:03 PM · Analytics-Kanban, DBA, Data-Services, Analytics
Krenair added a comment to T209031: Not able to scoop comment table in labs for mediawiki reconstruction process.

By the way, the query discussed on IRC was SELECT MIN(comment_id), MAX(comment_id) FROM (select comment_id, convert(comment_text using utf8) as comment_text from comment where (1 = 1) ) AS t1 which simplifies to SELECT MIN(comment_id), MAX(comment_id) FROM comment;

Thu, Nov 8, 2:53 PM · Analytics-Kanban, DBA, Data-Services, Analytics
Krenair edited projects for T209031: Not able to scoop comment table in labs for mediawiki reconstruction process, added: Data-Services; removed Cloud-Services.

I'm wondering if MySQL's query plan for this (full table scan) is as efficient as it could be. That said I doubt it's something that could realistically be changed without a different query.

Thu, Nov 8, 2:50 PM · Analytics-Kanban, DBA, Data-Services, Analytics
Krenair awarded T209031: Not able to scoop comment table in labs for mediawiki reconstruction process a Evil Spooky Haunted Tree token.
Thu, Nov 8, 2:46 PM · Analytics-Kanban, DBA, Data-Services, Analytics
Krenair added a comment to T208843: WMCS - Remove unused legacy code.

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/463325/
https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/470100/
https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/470101/

Thu, Nov 8, 1:27 PM · Patch-For-Review, cloud-services-team (Kanban)

Wed, Nov 7

Krenair created P7770 shinken alerts about integration-slave-docker SSH problems.
Wed, Nov 7, 4:56 PM
Krenair closed T208959: Update Herald (H187) to include project 3563 as Resolved.
Wed, Nov 7, 4:37 PM · Phabricator, Wikipedia-iOS-App-Backlog
Krenair closed T208958: Update Herald (H187) to include project 3688 as Resolved.
Wed, Nov 7, 4:31 PM · Phabricator, Wikipedia-iOS-App-Backlog
Krenair added a comment to T195861: Create a committee to improve the math support in Wikimedia projects.

I would expect so, possibly unless T208188 goes through depending on the details.

Wed, Nov 7, 2:49 PM · User-mobrovac, Patch-For-Review, Math
Krenair added a comment to T195861: Create a committee to improve the math support in Wikimedia projects.

They allow JavaScript, but there will be CSP protections that will restrict where you can load external resources from. Users aren't careful enough about what external JS they load into wikimedia users' sessions.

Wed, Nov 7, 2:31 PM · User-mobrovac, Patch-For-Review, Math
Krenair added a comment to T208870: TCP connections between analytics and deployment-prep.

I think by the way that if we do open this for use outside the project, it should be to specific authorised hosts on an individual IP basis only.

Wed, Nov 7, 1:58 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
Krenair added a comment to T208870: TCP connections between analytics and deployment-prep.
Nov 06 18:19:08 <elukey>	I am wondering if there was any talk about what discussed in T208870 in the past 
Nov 06 18:19:08 <stashbot>	T208870: TCP connections between analytics and deployment-prep - https://phabricator.wikimedia.org/T208870
Nov 06 18:21:22 *	thcipriani reads
Nov 06 18:23:41 <elukey>	I am very ignorant in this, but I am looking for the best way to deploy in cloud/labs without getting crazy in maintaining a deployment server
Nov 06 18:24:23 <thcipriani>	elukey: we haven't talked about that before and Krenair is right that deployment-deploy01 was built sfor use inside deployment-prep only, but it's definitely true that maintaining a deployment server is a difficult task
Nov 06 18:25:42 <thcipriani>	I think it should be possible to deploy from one place to the other with more relaxed firewall rules. I don't think it would create much maintenance burden to do so
Nov 06 18:26:18 <elukey>	that would really be great, I already used it in the past and it worked very well
Nov 06 18:26:45 <thcipriani>	let me dump some thoughts on that ticket
Nov 06 18:27:03 <thcipriani>	I'll see if anyone else has strong opinions here
Nov 06 18:27:57 <Krenair>	We could open the firewall rules sure
Nov 06 18:28:37 <Krenair>	but I don't think we want to encourage people relying on that system
Nov 06 18:28:53 <Krenair>	we can change the name and not tell anyone outside the project at any time
Nov 06 18:29:00 <Krenair>	we can change the services available on that box at any time
Nov 06 18:29:31 <Krenair>	if we remove someone from deployment-prep they may find they are no longer able to deploy to their own project
Nov 06 18:29:42 <Krenair>	we might not add someone to deployment-prep for the sole purpose of deploying elsewhere
Nov 06 18:29:44 <Krenair>	etc. etc. etc.
Nov 06 18:33:00 <Krenair>	other important one: if a repo you want to deploy from deployment-deploy01 also needs to go to deployment-prep instances, deployment-prep should decide which version rather than your external project
Nov 06 18:34:21 <thcipriani>	you could define a different environment in scap. I don't think there are technical hurdles for this working reliably from the scap perspective.
Nov 06 18:34:50 <thcipriani>	but managing users would be a bit strange, that's true
Nov 06 18:43:48 <elukey>	it's strange though that analytics is the only project with this requirement
Nov 06 18:44:17 <elukey>	our use case is to deploy very sporadically so we could also try to use "unofficial" support for the moment
Nov 06 18:44:32 <elukey>	so just get the hole open and see if it works
Nov 06 18:45:30 <Krenair>	you're not the only project with deployment hosts
Nov 06 18:45:36 <Krenair>	there is e.g. striker-deploy
Nov 06 18:46:08 <thcipriani>	ah right, I think there was a good write-up for setting up deployment hosts that was written as a result of ^ project
Wed, Nov 7, 1:57 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
Krenair awarded T208909: [Bug] Update old nonuniformly distributed page_random values a Evil Spooky Haunted Tree token.
Wed, Nov 7, 11:30 AM · MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), Patch-For-Review, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), DBA, MediaWiki-General-or-Unknown
Krenair added a project to T208903: Ability to see what changed in an edit (diffs) on iOS app: Wikipedia-iOS-App-Backlog.
Wed, Nov 7, 9:33 AM · Wikipedia-iOS-App-Backlog
Krenair updated subscribers of T208879: Puppet errors on various deployment-prep hosts.

@Mathew.onipe is taking care of -maps05

Wed, Nov 7, 9:08 AM · Beta-Cluster-Infrastructure, Cloud-VPS
Krenair added a comment to T208916: cloudvps: neutron issue with split brain.

and when you do get onto an eqiad1-r host, you also see DUPs while pinging out to other stuff:

krenair@gerrit-mysql:~$ ping bastion.wmflabs.org
PING bastion.wmflabs.org (10.68.17.232) 56(84) bytes of data.
64 bytes from bastion-01.bastion.eqiad.wmflabs (10.68.17.232): icmp_seq=1 ttl=63 time=4.03 ms
64 bytes from bastion-01.bastion.eqiad.wmflabs (10.68.17.232): icmp_seq=2 ttl=63 time=2.08 ms
64 bytes from bastion-01.bastion.eqiad.wmflabs (10.68.17.232): icmp_seq=3 ttl=63 time=4.52 ms
64 bytes from bastion-01.bastion.eqiad.wmflabs (10.68.17.232): icmp_seq=5 ttl=63 time=6.64 ms
64 bytes from bastion-01.bastion.eqiad.wmflabs (10.68.17.232): icmp_seq=6 ttl=63 time=0.632 ms
64 bytes from bastion-01.bastion.eqiad.wmflabs (10.68.17.232): icmp_seq=4 ttl=63 time=2041 ms
64 bytes from bastion-01.bastion.eqiad.wmflabs (10.68.17.232): icmp_seq=5 ttl=63 time=1027 ms (DUP!)
64 bytes from bastion-01.bastion.eqiad.wmflabs (10.68.17.232): icmp_seq=7 ttl=63 time=9.56 ms
^C
--- bastion.wmflabs.org ping statistics ---
7 packets transmitted, 7 received, +1 duplicates, 0% packet loss, time 6021ms
rtt min/avg/max/mdev = 0.632/387.139/2041.842/709.455 ms, pipe 3
krenair@gerrit-mysql:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=124 time=0.759 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=124 time=2.59 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=124 time=1.03 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=124 time=1.05 ms (DUP!)
Wed, Nov 7, 12:28 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Krenair renamed T208916: cloudvps: neutron issue with split brain from Network failure around eqiad1-r to Network failure around eqiad1-r (cloud-vps neutron).
Wed, Nov 7, 12:26 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Krenair added a project to T208916: cloudvps: neutron issue with split brain: cloud-services-team.
Wed, Nov 7, 12:23 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Krenair added a comment to T208916: cloudvps: neutron issue with split brain.

pinging is getting duplicate packets, both from within labs:

krenair@bastion-01:~$ ping eqiad1.bastion.wmflabs.org
PING eqiad1.bastion.wmflabs.org (172.16.1.136) 56(84) bytes of data.
64 bytes from bastion-eqiad1-01.bastion.eqiad.wmflabs (172.16.1.136): icmp_seq=1 ttl=63 time=0.545 ms
64 bytes from bastion-eqiad1-01.bastion.eqiad.wmflabs (172.16.1.136): icmp_seq=1 ttl=63 time=0.592 ms (DUP!)
64 bytes from bastion-eqiad1-01.bastion.eqiad.wmflabs (172.16.1.136): icmp_seq=2 ttl=63 time=0.490 ms
64 bytes from bastion-eqiad1-01.bastion.eqiad.wmflabs (172.16.1.136): icmp_seq=3 ttl=63 time=1.09 ms
64 bytes from bastion-eqiad1-01.bastion.eqiad.wmflabs (172.16.1.136): icmp_seq=4 ttl=63 time=0.616 ms
64 bytes from bastion-eqiad1-01.bastion.eqiad.wmflabs (172.16.1.136): icmp_seq=5 ttl=63 time=0.784 ms
64 bytes from bastion-eqiad1-01.bastion.eqiad.wmflabs (172.16.1.136): icmp_seq=6 ttl=63 time=8.02 ms
64 bytes from bastion-eqiad1-01.bastion.eqiad.wmflabs (172.16.1.136): icmp_seq=6 ttl=63 time=8.04 ms (DUP!)

and from my own device:

alex@alex-laptop:~$ ping eqiad1.bastion.wmflabs.org
PING eqiad1.bastion.wmflabs.org (185.15.56.13) 56(84) bytes of data.
64 bytes from eqiad1.bastion.wmflabs.org (185.15.56.13): icmp_seq=1 ttl=48 time=90.5 ms
64 bytes from eqiad1.bastion.wmflabs.org (185.15.56.13): icmp_seq=4 ttl=48 time=92.2 ms
64 bytes from eqiad1.bastion.wmflabs.org (185.15.56.13): icmp_seq=5 ttl=48 time=89.3 ms
64 bytes from eqiad1.bastion.wmflabs.org (185.15.56.13): icmp_seq=6 ttl=48 time=90.7 ms
64 bytes from eqiad1.bastion.wmflabs.org (185.15.56.13): icmp_seq=6 ttl=48 time=90.7 ms (DUP!)
64 bytes from eqiad1.bastion.wmflabs.org (185.15.56.13): icmp_seq=7 ttl=48 time=89.8 ms
Wed, Nov 7, 12:22 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Krenair added a comment to T208916: cloudvps: neutron issue with split brain.

SSH sometimes completely breaks:

alex@alex-laptop:~$ ssh eqiad1.bastion.wmflabs.org
Connection reset by 185.15.56.13 port 22
alex@alex-laptop:~$ ssh eqiad1.bastion.wmflabs.org
Linux bastion-eqiad1-01 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64
Debian GNU/Linux 9.5 (stretch)
bastion-eqiad1-01 is a Cloud VPS bastion host (with mosh enabled) (labs::bastion)
The last Puppet run was at Wed Nov  7 00:10:55 UTC 2018 (10 minutes ago). 
Last login: Tue Nov  6 23:28:48 2018 from 31.48.107.117
krenair@bastion-eqiad1-01:~$ exit
logout
Connection to eqiad1.bastion.wmflabs.org closed.
alex@alex-laptop:~$ ssh eqiad1.bastion.wmflabs.org
Connection reset by 185.15.56.13 port 22
Wed, Nov 7, 12:21 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
Krenair created T208916: cloudvps: neutron issue with split brain.
Wed, Nov 7, 12:20 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Tue, Nov 6

Krenair added a comment to T208887: Puppet errors on puppenmeister.planet.eqiad.wmflabs.

What is the master set to in puppet.conf, and is its cert signed by that particular master?

Tue, Nov 6, 10:03 PM · Cloud-VPS
Krenair renamed T208856: Instances with Puppet failures from tools: Instances with Puppet failures to Instances with Puppet failures.
Tue, Nov 6, 7:34 PM · Tracking, Cloud-VPS, cloud-services-team (Kanban)
Krenair added a comment to T208870: TCP connections between analytics and deployment-prep.

Okay, I think you both happen to be deployment-prep members but that isn't necessarily true in the general case of project members from any labs project wanting a deployment host.

Tue, Nov 6, 6:13 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
Krenair added a comment to T208870: TCP connections between analytics and deployment-prep.

Okay. What does the workflow look like for projects relying on it? To deploy into those projects you also need deployment-prep membership?

Tue, Nov 6, 5:50 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
Krenair removed projects from T208870: TCP connections between analytics and deployment-prep: cloud-services-team (Kanban), Cloud-VPS.
Tue, Nov 6, 5:49 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
Krenair added a comment to T208870: TCP connections between analytics and deployment-prep.

This may have previously snuck under the radar while your analytics instance had a 10/8 IP address, but now you're in eqiad1-r. Why should a deployment-prep deployment host be serving analytics instances?

Tue, Nov 6, 5:34 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
Krenair added a project to T208870: TCP connections between analytics and deployment-prep: Beta-Cluster-Infrastructure.
Tue, Nov 6, 5:33 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
Krenair closed T207476: Create production LE accounts as Resolved.

<vgutierrez> so certcentral is running now against the LE ACMEv2 production environment, no ratelimits hitted, no internal services hammered, pinkunicorn.wm.o certificates happily issued \o/ 

Tue, Nov 6, 1:12 PM · Patch-For-Review, Certcentral, Traffic, Operations
Krenair closed T207476: Create production LE accounts, a subtask of T194962: Create and deploy a centralized letsencrypt service, as Resolved.
Tue, Nov 6, 1:12 PM · Certcentral, Patch-For-Review, Wikimedia-Hackathon-2018, Traffic, Operations
Krenair awarded T208760: Add phabricator token "stroopwafel" a Cookie token.
Tue, Nov 6, 12:38 PM · Phabricator
Krenair added a comment to T208763: Enable page schemas on the beta cluster.

☝️

  • A SWAT will be needed to turn it on. Given this is a beta cluster only change, if you find a willing deployer, you can do it outside the normal SWAT windows.

The config is automatically deployed to the Beta Cluster when a config change is merged by the beta-mediawiki-config-update-eqiad Jenkins job.

Tue, Nov 6, 11:53 AM · Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), SEO
Krenair added a comment to T208722: Add Michael Grosse to 'wmde' LDAP group.

*any* ldap group?

Tue, Nov 6, 10:10 AM · Operations, LDAP-Access-Requests
Krenair awarded T30856: Remove classic edit toolbar from core a Evil Spooky Haunted Tree token.
Tue, Nov 6, 1:43 AM · MW-1.32-notes, User-notice, MW-1.29-release-notes, Technical-Debt, JavaScript, MediaWiki-Page-editing

Mon, Nov 5

Krenair renamed T208799: Add page_id column to wb_items_per_site from Add page_id column to wikidatawiki.wb_items_per_site to Add page_id column to wb_items_per_site .
Mon, Nov 5, 11:51 PM · Wikidata
Krenair added a project to T208799: Add page_id column to wb_items_per_site : Wikidata.

Alright. On the assumption that no such column currently exists in the production tables or the master version of the schema somewhere in a git repository: Ordinarily a schema change like this would originate from the relevant developers, who would need to agree to populate such a column anyway. After that the production schema will get changed by DBAs, presumably including the analytics-store stuff. Then would it be possible to expose the new column as part of a view in the labs replicas.

Mon, Nov 5, 11:51 PM · Wikidata
Krenair added a comment to T208799: Add page_id column to wb_items_per_site .

Is the replicas/analytics-store? Surely this is a problem in the base MW/Wikibase tables?

Mon, Nov 5, 11:40 PM · Wikidata
Krenair claimed T100373: U2F integration for Extension:OATHAuth.

Given this now works in FF (albeit with some config) I think this has some value and I'm going to have another go.

Mon, Nov 5, 5:23 PM · MediaWiki-extensions-OATHAuth
Krenair added a comment to T150565: Support physical OATH/OTP devices.

FWIW I've been using a Yubikey in OTP mode with OATHAuth for some time.

Mon, Nov 5, 5:03 PM · MediaWiki-extensions-OATHAuth
Krenair added a comment to T197158: CheckUser should require elevated security.

And all of this in response to, if I remember correctly, one time in the past year when an attacker gained CU access.

Mon, Nov 5, 4:54 PM · Security, Stewards-and-global-tools, CheckUser, Security-Extensions
Krenair added a comment to T192532: Figure out a way to enable volunteers to use the puppet compiler.

Yeah you just repeat the 'Hosts: ' part on each line: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/471195/

Mon, Nov 5, 4:22 PM · Release-Engineering-Team (Backlog), Operations, Puppet, puppet-compiler, Continuous-Integration-Config
Krenair renamed T207476: Create production LE accounts from Create production LE account to Create production LE accounts.
Mon, Nov 5, 3:45 PM · Patch-For-Review, Certcentral, Traffic, Operations
Krenair added a comment to T206461: Provide a Let's Encrypt ACME v2 staging environment account.

(the logical followup being T207476: Create production LE accounts)

Mon, Nov 5, 3:21 PM · Traffic, Operations
Krenair closed T206461: Provide a Let's Encrypt ACME v2 staging environment account as Resolved.

This was completed and used for e.g. T208424.

Mon, Nov 5, 3:20 PM · Traffic, Operations
Krenair closed T206461: Provide a Let's Encrypt ACME v2 staging environment account, a subtask of T199711: Deploy a scalable service for ACME (LetsEncrypt) certificate management, as Resolved.
Mon, Nov 5, 3:20 PM · Certcentral, Patch-For-Review, Traffic, Operations, Goal
Krenair moved T208731: Clean up and Refactoring of the class Block Checklist(?) from Backlog to User blocking on the MediaWiki-User-management board.
Mon, Nov 5, 3:14 PM · MediaWiki-User-management
Krenair added a project to T208731: Clean up and Refactoring of the class Block Checklist(?): MediaWiki-User-management.
Mon, Nov 5, 3:14 PM · MediaWiki-User-management
Krenair added a comment to T208405: Check whether huggle project requires NFS or not.

Sounds like the huggle project could be migrated to local storage then, and loose the NFS dependency?

Mon, Nov 5, 2:56 PM · Huggle, Cloud-VPS
Krenair closed T208418: Check whether wikidata-query project requires NFS or not as Resolved.

Sounds like it's necessary then. Thanks @Smalyshev

Mon, Nov 5, 2:54 PM · Wikidata, Wikidata-Query-Service, Cloud-VPS
Krenair closed T208418: Check whether wikidata-query project requires NFS or not, a subtask of T102240: Audit projects' use of NFS, and remove it where not necessary, as Resolved.
Mon, Nov 5, 2:54 PM · Wikimedia-Incident, Labs-Sprint-106, Labs-Sprint-105, Labs-Sprint-104, Incident-20150617-LabsNFSOutage, Labs-Sprint-103, Labs-Sprint-102, Cloud-Services
Krenair added a comment to T208402: Check whether dumps project requires NFS or not.

Well the migrations are in the process of happening. I don't know when the dumps project is scheduled but it sounds like that will be unblocked soon.

Mon, Nov 5, 2:52 PM · Cloud-VPS
Krenair added a comment to T208416: Check whether wikidata-dev project requires NFS or not.

Well first thing would be to go through your instances and check which ones have nfs entries showing on df (they usually get mounted under /mnt/nfs/, and come from one of the labstore* servers). Then you should look at where those get mounted to, what files are in there, what applications write those files, and whether an application running on one host needs to be able to access files written to that NFS share on another host.

Mon, Nov 5, 2:43 PM · wikidata-tech-focus, Wikidata, Cloud-VPS
Krenair reassigned T208240: Ensure jenkins on puppet.git checks for yaml syntax errors from Krenair to hashar.
Mon, Nov 5, 2:34 PM · Patch-For-Review, Continuous-Integration-Config, Operations
Krenair added a project to T208695: Duplicate key on several s8 replicas breaking replication: Wikimedia-Incident.
Mon, Nov 5, 12:20 AM · Wikidata, Wikimedia-Incident, DBA

Sun, Nov 4

Krenair updated subscribers of T174469: LDAP account that is not attached on wikitech has no means for password reset.

A user has appeared in #wikimedia-dev asking about the account which turns out to be uid=siyam-_-,ou=people,dc=wikimedia,dc=org, cn: MD Abu Siyam. They've forgotten the password but can't reset because wikitech thinks it doesn't exist: https://wikitech.wikimedia.org/wiki/Special:Contributions/MD_Abu_Siyam
@bd808 please can you attach it?

Sun, Nov 4, 1:40 AM · Striker, wikitech.wikimedia.org

Sat, Nov 3

Krenair added a comment to T143238: CE Insights - Performance Survey - 2016.

Given that it's now 2018, is this task and all subtasks done?

Sat, Nov 3, 11:24 PM · Surveys, Community-Engagement-Insights
Krenair added a comment to T208519: wm-bot logs broken.

No there isn't any problem with a security rule, as I said I can connect to it from wm-bot2 instance using psql (postgre's CLI) just fine, it's wm-bot's npgsql library that isn't able to connect there, probably some kind of a bug in the library itself.

Huh, true, that is weird. I'd have a go with it but I don't have access to the wm-bot project.

Sat, Nov 3, 11:22 PM · WM-Bot