Page MenuHomePhabricator

Krenair (Alex Monk)
Wikimedia volunteer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:34 PM (270 w, 2 d)
Availability
Available
IRC Nick
Krenair
LDAP User
Alex Monk
MediaWiki User
Krenair [ Global Accounts ]

I am a Wikimedia volunteer helping in various technical ways. These days it's usually Beta Cluster, Cloud VPS, or Operations related. Since 2012 I've spent significant amounts of time involved in MediaWiki development, software deployments to the Wikimedia cluster, OTRS (email response to e.g. info-en@wikimedia.org addresses), and various other things.

Some of my old VisualEditor and other work (2014-2016) can be found under @AlexMonk-WMF instead.

I have opinions on things, which do not necessarily represent those of any organisation I am, have previously been, or will in the future be affiliated with.

Recent Activity

Today

Krenair added a comment to T231518: Add *.wmflabs.org to w.wiki shortener.

What sort of criteria and restrictions would we have to impose on such a
whitelisted tool?

Sun, Dec 8, 3:13 PM · Wikimedia-Site-requests, Security-Team, MediaWiki-extensions-UrlShortener
Krenair added a comment to T240094: Create required table for new Watchlist Expiry feature.

Why a new table instead of a new field on the existing table?

Sun, Dec 8, 6:34 AM · Community-Tech, Expiring-Watchlist-Items

Thu, Dec 5

Krenair added a comment to T236551: "wikistream" Cloud VPS project jessie deprecation.

Creating a new instance can be done through horizon and should be fairly
intuitive, you'll want to pick either Debian stretch (older, process is
likely to need repeating sooner) or Debian buster (newer, process shouldn't
need repeating for longer), and an instance size (can use the same as the
existing one shown in openstack-browser/horizon, though if it's currently
larger than required this is a good opportunity to downsize).
Once you've submitted the form in horizon it'll go through the setup
process and after a few minutes you should find yourself able to log into
it. You then configure it probably roughly the same way you did the
existing instance.
In your case it looks like instead of floating IPs and your own DNS you use
the provided HTTP(S) proxy. You'll want to delete the existing entry and
make a new one (if it needs to be done in place leave a note and maybe
someone can have a look to see if that's doable).
If you encounter issues you can request help on the task or maybe on IRC
(#wikimedia-cloud).

Thu, Dec 5, 9:39 PM · Cloud-VPS (Debian Jessie Deprecation)
Krenair added a comment to T236583: "discourse" Cloud VPS project jessie deprecation.

I would discourage enabling people to just dist upgrade more easily and
with support due to the implication that the instance is hard to replace
from scratch. People should be prepared to replace these. Especially
considering we can lose them irrecoverably.

Thu, Dec 5, 8:06 PM · Space (Oct-Dec-2019), Cloud-VPS (Debian Jessie Deprecation)
Krenair added a comment to T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.

What exactly is needed? Do we need to make a second logstash stretch host?
Do we need to move something else over from the old instance? How does the
Kafka cluster interact with this beyond firing logs at it?

Thu, Dec 5, 8:04 PM · Release-Engineering-Team-TODO, observability, Wikimedia-Logstash, Beta-Cluster-Infrastructure

Tue, Dec 3

Krenair added a comment to T239036: OBB-1022554 XSS in not-in-the-other-language.

other parameters this stuff can be passed in: lang1, lang2, proj1, proj2, starts_with, depth - and maybe others.

Tue, Dec 3, 1:42 AM · Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team, Security
Krenair added a comment to T239036: OBB-1022554 XSS in not-in-the-other-language.

By the looks of things this is just a straightforward classic use of a request var directly in output, missing an htmlspecialchars call:

krenair@tools-sgebastion-07:~$ grep pagepile ~tools.not-in-the-other-language/public_html/index.php 
require_once ( '/data/project/pagepile/public_html/pagepile.php' ) ;
$pagepile = get_request ( 'pagepile' , '' ) ;
if ( $pagepile_enabeled ) {
	print "<tr><th>PagePile</th><td><input class='span4' type='text' name='pagepile' id='pagepile' value='$pagepile' placeholder='PagePile input ID' /> (optional; check out <a href='/pagepile' target='_blank'>PagePile</a>)</td></tr>" ;
Tue, Dec 3, 1:36 AM · Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team, Security

Mon, Dec 2

Krenair created P9797 scholia crafting with uwsgi plugin issue.
Mon, Dec 2, 9:59 PM · Tools
Krenair added a comment to T132084: Notify editors that they are now eligible for the Wikipedia Library program.

I've just noticed this thing run in beta (P9795) after editing enwiki there for the first time in a few months - are we supposed to be directing people towards a wmflabs.org site?

Mon, Dec 2, 8:34 PM · Wikimedia-extension-review-queue, Growth-Team, Patch-For-Review, User-notice-collaboration, User-notice, Notifications, The-Wikipedia-Library
Krenair created P9795 Weird Echo email from beta inviting me to click a non-beta wmflabs.org link!?.
Mon, Dec 2, 8:22 PM

Sat, Nov 30

Krenair added a project to T239036: OBB-1022554 XSS in not-in-the-other-language: Toolforge-standards-committee.
Sat, Nov 30, 3:01 PM · Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team, Security
Krenair added a watcher for Toolforge-standards-committee: Krenair.
Sat, Nov 30, 2:58 PM
Krenair closed T172606: Tool "iacrop" redirects to localhost, a subtask of T172065: Hunt for Toolforge tools that load resources from third party sites, as Resolved.
Sat, Nov 30, 2:49 PM · Toolforge-standards-committee, Tools, Privacy
Krenair closed T172606: Tool "iacrop" redirects to localhost as Resolved.

It looks like the webservice is gone now but per the author it's discontinued, so closing this

Sat, Nov 30, 2:49 PM · Tools
Krenair added a comment to T172606: Tool "iacrop" redirects to localhost.

was it maybe this:

tools.iacrop@tools-sgebastion-07:~$ grep 127.0.0.1 . -r
./.lighttpd.conf:	"iacrop/(.*)" => "http://127.0.0.1/iacrop/public/$1"
Sat, Nov 30, 2:38 PM · Tools

Fri, Nov 29

Krenair added a project to T239036: OBB-1022554 XSS in not-in-the-other-language: cloud-services-team.

I've sent an email (from my gmail) to the address provided:

Fri, Nov 29, 9:03 PM · Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team, Security
Krenair updated the task description for T239036: OBB-1022554 XSS in not-in-the-other-language.
Fri, Nov 29, 9:01 PM · Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team, Security

Wed, Nov 27

Krenair closed T196797: When changes to a Designate zone occur (e.g. record creations/deletions), there is a brief period in which the entire zone is NXDOMAIN as Resolved.

Yep, I don't see this when creating/updating/deleting records.

Wed, Nov 27, 9:21 PM · cloud-services-team (Kanban), Beta-Cluster-reproducible, Cloud-VPS

Sun, Nov 24

Krenair updated the task description for T239036: OBB-1022554 XSS in not-in-the-other-language.
Sun, Nov 24, 10:14 PM · Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team, Security
Krenair added a project to T239036: OBB-1022554 XSS in not-in-the-other-language: Toolforge.
Sun, Nov 24, 10:14 PM · Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team, Security
Krenair created T239036: OBB-1022554 XSS in not-in-the-other-language.
Sun, Nov 24, 10:13 PM · Tools, Vuln-XSS, Toolforge-standards-committee, cloud-services-team, Security-Team, Security

Fri, Nov 22

Krenair added a comment to T238960: Conversion to volunteer NDA for MaxSem.

As far as I know, no NDA is required for beta cluster access.

Fri, Nov 22, 11:00 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations
Krenair added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

Alright, I:

root@deployment-logstash03:~# curl -X POST "localhost:9200/_reindex" -H 'Content-Type: application/json' -d"
  {
    \"source\": {
      \"remote\": {
        \"host\": \"http://deployment-logstash2.deployment-prep.eqiad.wmflabs:9200\"
      },
      \"index\": \".kibana\"
    },
    \"dest\": {
      \"index\": \".kibana\"
    }
  }
"
{"took":841,"timed_out":false,"total":151,"updated":0,"created":151,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[]}
  • re-enabled and ran puppet on those two hosts

and some dashboards have appeared at https://logstash-beta.wmflabs.org/app/kibana#/dashboards?_g=() just like on old-logstash-beta

Fri, Nov 22, 5:57 PM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
Krenair set Security to security-bug on T145703: Horizon loses credentials every day.

Combination of Arturo's sessionid and __mmapiwsid cookies logs one in as Arturo to Horizon, locking this task down

Fri, Nov 22, 1:34 AM · Security, cloud-services-team (Kanban), Horizon

Thu, Nov 21

Krenair added a comment to T131288: Make labs proxies https only.

We'd need to set https_upgrade to true either by changing the default or setting the key when we create the dynamicproxy class.

Thu, Nov 21, 11:33 PM · cloud-services-team (Kanban), Cloud-VPS

Wed, Nov 20

Krenair added a comment to T120486: add a https-only option to dynamicproxy.

done in https://gerrit.wikimedia.org/r/c/operations/puppet/+/482142 ?

Wed, Nov 20, 10:29 PM · cloud-services-team (Kanban), Traffic, HTTPS, Cloud-VPS, Operations
Krenair added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

looks like each of the logstash hosts runs its own elasticsearch cluster locally, would our source be something like deployment-logstash2.deployment-prep.eqiad.wmflabs:9200 or deployment-logstash2.deployment-prep.eqiad.wmflabs:9300 ? it seems we'd need to configure reindex.remote.whitelist somewhere too though I have no idea where

Wed, Nov 20, 9:40 PM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
Krenair added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

If the new elasticsearch cluster has access to the old one

Wed, Nov 20, 9:19 PM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
Krenair added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

Wondering what we need to do next. Do we need to copy dashboards over somehow?

Wed, Nov 20, 1:05 AM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
Krenair added a comment to T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.

logstash-beta now points at deployment-logstash03, deployment-logstash2 is still accessible at https://old-logstash-beta.wmflabs.org/ for the time being. Is stuff better now?
I do see results at https://logstash-beta.wmflabs.org/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:AW6GShqJw8E2oQN1xWTl,interval:auto,query:(query_string:(query:'type:mediawiki')),sort:!('@timestamp',desc))

Wed, Nov 20, 1:01 AM · Release-Engineering-Team-TODO, observability, Wikimedia-Logstash, Beta-Cluster-Infrastructure
Krenair updated the task description for T238711: Horizon proxy deletion UI has blank where we should name the proxy we're deleting.
Wed, Nov 20, 12:43 AM · Cloud-VPS
Krenair created T238711: Horizon proxy deletion UI has blank where we should name the proxy we're deleting.
Wed, Nov 20, 12:43 AM · Cloud-VPS
Krenair renamed T238710: Make it possible to move an existing dynamicproxy entry to a different backend from Make it possible to move an existing dynamicproxy entry to a new instance to Make it possible to move an existing dynamicproxy entry to a different backend.
Wed, Nov 20, 12:41 AM · Cloud-VPS
Krenair created T238710: Make it possible to move an existing dynamicproxy entry to a different backend.
Wed, Nov 20, 12:41 AM · Cloud-VPS
Krenair claimed T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).
Wed, Nov 20, 12:30 AM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
Krenair added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

horizon-based hieradata changes:
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/5f26dcdb608d31f477ec2f74de31f55c81fa4665%5E%21/#F0
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/0660c88726226d038e7da0546f9e1f6192f565c5%5E%21/#F0
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/9f14f09b6cfa2631ae11283118d12a297517bbee%5E%21/#F0
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/f0dd923a8cfdff1ee269a11614e774b032ef338e%5E%21/#F0
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/937bb910a49cb18418b6372ad56c4a3fc7d5b8b4%5E%21/#F0
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/943782a36eef45a4f4f45c0a6637b4c435adb03d%5E%21/#F0

Wed, Nov 20, 12:29 AM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
Krenair removed a project from T238708: Delete instance-puppet entries for deleted VMs: MW-1.35-notes (1.35.0-wmf.5; 2019-11-05).
Wed, Nov 20, 12:07 AM · cloud-services-team (Kanban), Cloud-VPS
Krenair added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

If deployment-logstash03 has the same classes applied than deployment-logstash2

Wed, Nov 20, 12:07 AM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
Krenair created T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).
Wed, Nov 20, 12:00 AM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure

Mon, Nov 18

Krenair assigned T238509: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors to Vgutierrez.

Thanks Valentin.

Mon, Nov 18, 8:49 PM · Operations, Traffic
Krenair updated the task description for T238509: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors.
Mon, Nov 18, 1:02 AM · Operations, Traffic
Krenair created T238509: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors.
Mon, Nov 18, 12:49 AM · Operations, Traffic

Sun, Nov 17

Krenair added a comment to T234612: Soft launch proxy-based access for a few partners.

Can someone clarify what is technically being launched here? I'm seeing references to proxies but also wmflabs.org URLs. Is a network proxy being run inside labs?

Sun, Nov 17, 1:18 AM · Library-Card-Platform-Proxy

Fri, Nov 15

Krenair added a comment to T218609: Figure out future for newly created deployment-prep jessie instances.

@Eevans: It has been 6 months, please respond.

Fri, Nov 15, 12:49 PM · Beta-Cluster-Infrastructure
Krenair added a comment to T218729: Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster.

@fgiunchedi Do we need to do anything else to get rid of deployment-logstash2 and use deployment-logstash03 instead? logstash2 now has a puppet error due to https://gerrit.wikimedia.org/r/c/operations/puppet/+/522406 (T198092)

Fri, Nov 15, 12:46 PM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure

Thu, Nov 14

Krenair placed T238222: Request creation of BishopFox VPS project up for grabs.
Thu, Nov 14, 9:05 PM · Cloud-VPS (Project-requests)
Krenair added a comment to T238222: Request creation of BishopFox VPS project.

This sounds fine, I do wonder about the pen test in general though - this is just a routine thing right? We're not considering putting anything more sensitive into Cloud VPS, or trusting it more than currently?

Thu, Nov 14, 9:04 PM · Cloud-VPS (Project-requests)

Mon, Nov 11

Krenair added a comment to T237889: Install php-ldap on all MW appservers.

Probably because this is a logical blocker to the parent task.

Mon, Nov 11, 9:12 AM · Operations, wikitech.wikimedia.org

Sun, Nov 10

Krenair added a comment to T204013: Horizon Designate dashboard not allowing creation of NS records.

(Stein got released on 10th April, Wikimedia probably wont have it for a while though)

Sun, Nov 10, 11:31 PM · cloud-services-team (Kanban), Operations, Traffic, Upstream, Horizon
Krenair added a comment to T237863: Cloud VPS proxies should handle subdomains.

We'd need some mechanism for it to be creating *.<yourname>.wmflabs.org certs for each one (right now everything is just on one big *.wmflabs.org cert which wouldn't match this), which would require some ACME automation we don't have at the moment.

That seems fairly easy though, and needed anyway at some point for things like custom domain names.

Sun, Nov 10, 7:44 PM · Cloud-VPS, MediaWiki-Vagrant
Krenair added a comment to T237863: Cloud VPS proxies should handle subdomains.

We'd need some mechanism for it to be creating *.<yourname>.wmflabs.org certs for each one, which would require some ACME automation we don't have at the moment. Also this is feeling dupe-y.

Sun, Nov 10, 5:05 PM · Cloud-VPS, MediaWiki-Vagrant
Krenair awarded T229934: Enable semantic relationship between code review changesets and maniphest tasks in phabricator a Mountain of Wealth token.
Sun, Nov 10, 4:20 PM · GerritBot, Phabricator, Code-Review-Workgroup
Krenair reassigned T51553: Strip linebreaks in uploaded ssh keys from Krenair to yuvipanda.

Yuvi fixed it, I just added the task to the commit message and approved it :)

Sun, Nov 10, 3:50 AM · MediaWiki-extensions-OpenStackManager

Sat, Nov 9

Krenair added a comment to T235218: Catch cloud-puppetmasters up with production puppetmaster versions.

rsync -ar'd the files from the old puppetmaster, to my machine, then to the new puppetmaster frontend (-03), then shredded my copy of the files. Added the new puppetmasters to the puppetmaster security group so they can actually receive traffic.
Ran echo '172.16.0.38 puppetmaster.cloudinfra.wmflabs.org' >> /etc/hosts on my test instance (krenair-t235218-test.testlabs.eqiad.wmflabs) to get it talking to the new frontend (-03).

Sat, Nov 9, 7:24 PM · Patch-For-Review, User-jbond, cloud-services-team (Kanban)

Fri, Nov 8

Krenair added a comment to T237768: Could not find dependency Package[python-yaml] error in profile::toolforge::grid::node::web .

May have been implicitly depending on the python-yaml package from profile::wmcs::instance -> diamond::collector::minimalpuppetagent that got removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/549241 ?

Fri, Nov 8, 9:36 PM · cloud-services-team
Krenair added a comment to T179816: Cumin: create external backend for WMCS Puppet API.

People working on this task will probably want to add a GET route to modules/openstack/files/puppet/master/encapi/labspuppetbackend.py in operations/puppet.git (to make it quick to get data about all instances and their puppet classes in one query), and it will be exposed within the labs network at http://puppetmaster.cloudinfra.wmflabs.org:8100 where a cumin backend can pick it up without needing to send hundreds of requests

Fri, Nov 8, 7:45 PM · cloud-services-team (Kanban), SRE-tools

Nov 7 2019

Krenair added a comment to T237691: cloud-cumin-01: HTTPSConnectionPool - Max retries exceeded with url.

Cloud-cumin's domain is the whole of labs, it doesn't have a PuppetDB. I
wonder what it's trying to connect to...

Nov 7 2019, 9:19 PM · SRE-tools, Cloud-VPS
Krenair created P9546 Cloud VPS instance flavours.
Nov 7 2019, 2:44 AM

Nov 5 2019

bd808 awarded T236952: Move tools-static.wmflabs.org behind project-proxy a Meh! token.
Nov 5 2019, 10:51 PM · cloud-services-team (Kanban), Toolforge
Krenair closed T236952: Move tools-static.wmflabs.org behind project-proxy as Resolved.
Nov 5 2019, 10:51 PM · cloud-services-team (Kanban), Toolforge
Krenair added a comment to T237468: tools-k8s-master-01 (Kubernetes API server for toolforge) has failing puppet staleness cron.

For the record what I found (and then redacted pending upgrade) was this:

krenair@tools-k8s-master-01:~$ dpkg -S /usr/lib/python3.4/http/client.py
libpython3.4-stdlib:amd64: /usr/lib/python3.4/http/client.py
krenair@tools-k8s-master-01:~$ apt-cache policy libpython3.4-stdlib
libpython3.4-stdlib:
  Installed: 3.4.2-1+deb8u3
  Candidate: 3.4.2-1+deb8u7
  Version table:
     3.4.2-1+deb8u7 0
        500 http://security.debian.org/ jessie/updates/main amd64 Packages
 *** 3.4.2-1+deb8u3 0
        100 /var/lib/dpkg/status
     3.4.2-1 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
Nov 5 2019, 10:38 PM · Toolforge, cloud-services-team (Kanban)
Krenair added a comment to T237468: tools-k8s-master-01 (Kubernetes API server for toolforge) has failing puppet staleness cron.
Nov 5 2019, 10:28 PM · Toolforge, cloud-services-team (Kanban)
Krenair added a comment to T237468: tools-k8s-master-01 (Kubernetes API server for toolforge) has failing puppet staleness cron.

The weird thing is that it should crash a lot more if that's happening here...but other python3 things appear to be working?

Nov 5 2019, 10:26 PM · Toolforge, cloud-services-team (Kanban)
Krenair closed T235674: Beta cluster doesn’t update since ca. 2019-10-15 21:00 UTC as Resolved.

I don't think we figured out *why* exactly the unencrypted snakeoil keys got deployed everywhere but we sorted out the problem that this task was for.

Nov 5 2019, 9:46 PM · Patch-For-Review, Release-Engineering-Team-TODO (201910), Beta-Cluster-Infrastructure
Krenair awarded T237470: Create and maintain somehow a list of repos mastered in GitHub (and in Differential) a Love token.
Nov 5 2019, 9:44 PM · Release-Engineering-Team (Development services), Gerrit, Differential, GitHub-Mirrors, Release-Engineering-Team-TODO
Krenair added a comment to T237468: tools-k8s-master-01 (Kubernetes API server for toolforge) has failing puppet staleness cron.

f-strings are 3.6+, wonder what that's doing lurking under /usr/lib/python3.4

Nov 5 2019, 9:43 PM · Toolforge, cloud-services-team (Kanban)
Krenair added a comment to T237066: Push renewed *.wmflabs.org certificate and new private key to cluster (expires 2019-11-16).

(FYI docs for this now live at https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/SSL_certificate)

Nov 5 2019, 8:07 PM · cloud-services-team (Kanban), Cloud-Services

Nov 4 2019

Krenair added a comment to T235218: Catch cloud-puppetmasters up with production puppetmaster versions.

So I've started making cloud-puppetmaster-04 ready to be a backend, it's got puppet errors about not being able to find the geoipupdate package. The only buster instance in labs with it installed is jbond-pmaster-buster.puppet.eqiad.wmflabs, which gets it from 500 http://deb.debian.org/debian buster/contrib amd64 Packages. Looking at that instance a little more closely that will be coming from this first line of /etc/apt/sources.list: deb http://deb.debian.org/debian/ buster main non-free contrib, however the file appears unpuppetised and on cloud-puppetmaster-04 the line is just deb http://deb.debian.org/debian/ buster main. @jbond, did you add that contrib component yourself or are we missing something? Where do production puppetmasters get it from?

@Krenair The production host have non-free contrib configured via d-i during installation*. This is not currently managed via puppet but there are plans to do so
*Contrib is implicit with non-free

Nov 4 2019, 11:25 PM · Patch-For-Review, User-jbond, cloud-services-team (Kanban)
Krenair renamed T237290: Disable mobile beta mode (for now) from Disable beta (for now) to Disable mobile beta mode (for now).
Nov 4 2019, 7:28 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), User-Jdlrobson, User-notice, Patch-For-Review, Readers-Web-Backlog (Kanbanana-2019-20-Q2), Wikimedia-Site-requests
Krenair added a comment to T149589: Puppet tab in Horizon unusably slow.

I've added an alternative editor that loads much faster. To switch modes, scroll to the bottom of the puppet tab and click on "Switch to guided mode". The change will last for the life of your Horizon session.
I'm reluctant to make this editor the default as it seems hostile to beginners and volunteers who are only switching on a single class. I'm not sure how to balance between the use cases really.

Nov 4 2019, 7:13 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Puppet, Cloud-Services
Krenair added a comment to T237259: Document all uses of the puppetCA certificate.

Acme-chief nginx config probably?

Nov 4 2019, 6:30 PM · Patch-For-Review, DBA, User-jbond, Puppet, Operations

Nov 3 2019

Krenair added a comment to T235218: Catch cloud-puppetmasters up with production puppetmaster versions.

Anyone have any preferences for how we might transfer /var/lib/puppet/server/ssl/private_keys/*.pem from cloud-puppetmaster-01 to cloud-puppetmaster-03?

Nov 3 2019, 11:58 PM · Patch-For-Review, User-jbond, cloud-services-team (Kanban)
Krenair triaged T237208: Deprecated Stretch image from February has been reactivated as Lowest priority.
Nov 3 2019, 11:42 PM · Cloud-VPS
Krenair created T237208: Deprecated Stretch image from February has been reactivated.
Nov 3 2019, 11:42 PM · Cloud-VPS
Krenair added a comment to T235218: Catch cloud-puppetmasters up with production puppetmaster versions.

I also see mwv-puppetmaster.mediawiki-vagrant.eqiad.wmflabs gets this component in a /etc/apt/sources.list.d/debian-contrib.list file but I don't see any apt::repository resources in puppet.git that could create such a file.

I live hacked the package there recently before removing the provisioning of geoipupdate for ::role::puppetmaster::standalone users through T236487: geoipupdate missing on buster on Cloud VPS. Setting puppetmaster::enable_geoip: false in hiera for cloud-puppetmaster-* should do the same thing.

Nov 3 2019, 9:28 PM · Patch-For-Review, User-jbond, cloud-services-team (Kanban)
Krenair added projects to T237173: InvalidArgumentException breaking Commons "undo" functionality: SDC General, Wikimedia-production-error.

I've managed to deal with this individual edit by editing the caption myself to blank it.

Nov 3 2019, 2:15 AM · Wikidata, Wikimedia-production-error, SDC General, Commons

Nov 2 2019

Krenair added a comment to T235218: Catch cloud-puppetmasters up with production puppetmaster versions.

So I've started making cloud-puppetmaster-04 ready to be a backend, it's got puppet errors about not being able to find the geoipupdate package. The only buster instance in labs with it installed is jbond-pmaster-buster.puppet.eqiad.wmflabs, which gets it from 500 http://deb.debian.org/debian buster/contrib amd64 Packages. Looking at that instance a little more closely that will be coming from this first line of /etc/apt/sources.list: deb http://deb.debian.org/debian/ buster main non-free contrib, however the file appears unpuppetised and on cloud-puppetmaster-04 the line is just deb http://deb.debian.org/debian/ buster main. @jbond, did you add that contrib component yourself or are we missing something? Where do production puppetmasters get it from?

Nov 2 2019, 1:24 PM · Patch-For-Review, User-jbond, cloud-services-team (Kanban)
Krenair closed T237150: paws-public.wmflabs.org returns 502 Bad Gateway as Resolved.

I sorted this and cleaned up two other cases of hardcoded IPs in the config:

root@paws-proxy-02:~# grep proxy_pass /etc/nginx/sites-enabled/default 
#		proxy_pass https://172.16.6.39;
		proxy_pass https://tools.wmflabs.org;
#		proxy_pass http://172.16.5.206:32611;
		proxy_pass http://tools-paws-worker-1006.tools.eqiad.wmflabs:32611;
#		proxy_pass http://172.16.5.206:32612;
		proxy_pass http://tools-paws-worker-1006.tools.eqiad.wmflabs:32612;
root@paws-proxy-02:~#
Nov 2 2019, 12:14 PM · PAWS
Krenair added a comment to T237150: paws-public.wmflabs.org returns 502 Bad Gateway.

from paws-proxy-02.paws.eqiad.wmflabs:/var/log/nginx/error.log:
upstream: "https://172.16.6.39:443/paws-public/User:Lucas_Werkmeister_(WMDE)/"

Nov 2 2019, 12:11 PM · PAWS

Nov 1 2019

Krenair closed T236103: Beta Cluster deployment failed: sign_and_send_pubkey: signing failed: agent refused operation as Resolved.
Nov 1 2019, 9:18 PM · Product-Infrastructure-Team-Backlog, Page Content Service, Beta-Cluster-Infrastructure
Krenair closed T236962: Migrate away from legacy star.tools.wmflabs.org certificate as Resolved.
Nov 1 2019, 9:18 PM · Toolforge
Krenair added a comment to T237132: Request IP address quota of 1 for instance 'gratitude'.

I'm a little concerned by the idea of running a system within cloud VPS
where the uptime is considered very important.

Nov 1 2019, 7:22 PM · Cloud-VPS (Quota-requests)

Oct 31 2019

Krenair updated the task description for T236962: Migrate away from legacy star.tools.wmflabs.org certificate.
Oct 31 2019, 11:33 PM · Toolforge
Krenair added a comment to T236952: Move tools-static.wmflabs.org behind project-proxy.

It's accepting traffic with X-Forwarded-Proto correctly, confirmed we have to do the proxy config manually (can't just add a new proxy through the horizon UI and have it take over the existing DNS A record), so I've configured the proxy with this on proxy-01.project-proxy.eqiad.wmflabs:

>>> import requests
>>> resp = requests.put(
...     'http://proxy-01.project-proxy.eqiad.wmflabs:5668/v1/tools/mapping',
...     json={
...         "backends": ['http://172.16.0.186:80'],
...         "domain": 'tools-static.wmflabs.org'
...     }
... )
>>> resp.content
b''
>>> resp.status_code
200
Oct 31 2019, 11:19 PM · cloud-services-team (Kanban), Toolforge
Krenair claimed T235218: Catch cloud-puppetmasters up with production puppetmaster versions.
Oct 31 2019, 7:34 PM · Patch-For-Review, User-jbond, cloud-services-team (Kanban)
Krenair added a comment to T236952: Move tools-static.wmflabs.org behind project-proxy.

Proposed plan here:

  • https://gerrit.wikimedia.org/r/547360
  • Check it accepts traffic properly with something like curl -H 'Host: tools-static.wmflabs.org' -H 'X-Forwarded-Proto: https' http://tools-static-12/pagecounts/pagecounts.json
  • Create proxy entry for tools-static.wmflabs.org pointing at this instance (manually if we have to, don't want to interrupt existing traffic)
  • If we had to do the previous step manually, update A record for tools-static to point at project-proxy. Wait for DNS TTL.
  • Traffic should still be working externally, test that with something like https://tools-static.wmflabs.org/pagecounts/pagecounts.json - logs should appear on the tools-static instance appearing from the proxy-01.project-proxy.eqiad.wmflabs internal IP
  • https://gerrit.wikimedia.org/r/547363
  • Wait for a puppet run
  • Confirm everything is fine
  • https://gerrit.wikimedia.org/r/547364
  • Disassociate floating IP
Oct 31 2019, 12:40 AM · cloud-services-team (Kanban), Toolforge
Krenair claimed T236952: Move tools-static.wmflabs.org behind project-proxy.
Oct 31 2019, 12:30 AM · cloud-services-team (Kanban), Toolforge
Krenair claimed T236962: Migrate away from legacy star.tools.wmflabs.org certificate.
Oct 31 2019, 12:05 AM · Toolforge

Oct 30 2019

Krenair removed projects from T236952: Move tools-static.wmflabs.org behind project-proxy: HTTPS, Cloud-VPS, Traffic.
Oct 30 2019, 11:32 PM · cloud-services-team (Kanban), Toolforge
Krenair added a comment to T236876: git-sync on deployment-puppetmaster03 fails to rebase since days ago.

Sure, even if I might not resolve the conflict in the best way and break Beta :)

Oct 30 2019, 10:33 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure
Krenair created T236962: Migrate away from legacy star.tools.wmflabs.org certificate.
Oct 30 2019, 9:48 PM · Toolforge
Krenair added a comment to T235218: Catch cloud-puppetmasters up with production puppetmaster versions.

Phabricator is being weird and hasn't shown it yet but I have made a subtask for quota: T236961: Requesting more quota space in cloudinfra for 2x m1.xlarge (or custom) instances for buster cloud-puppetmasters

Oct 30 2019, 9:42 PM · Patch-For-Review, User-jbond, cloud-services-team (Kanban)
Krenair created T236961: Requesting more quota space in cloudinfra for 2x m1.xlarge (or custom) instances for buster cloud-puppetmasters.
Oct 30 2019, 9:40 PM · Cloud-VPS (Quota-requests), cloud-services-team (Kanban)
Krenair added a parent task for T235252: Toolforge: SSL support for new domain toolforge.org: Unknown Object (Task).
Oct 30 2019, 8:17 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
Krenair created T236952: Move tools-static.wmflabs.org behind project-proxy.
Oct 30 2019, 8:17 PM · cloud-services-team (Kanban), Toolforge
Krenair added a comment to T235252: Toolforge: SSL support for new domain toolforge.org.

Yay. https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/547293/ to clean up the old absenting resource.

Oct 30 2019, 8:14 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban), Kubernetes
Krenair added a comment to T235218: Catch cloud-puppetmasters up with production puppetmaster versions.

We do have a front-end and a backend-only host in the new setup. Obviously
it's all one site.

Oct 30 2019, 6:46 PM · Patch-For-Review, User-jbond, cloud-services-team (Kanban)
Krenair added a comment to T236876: git-sync on deployment-puppetmaster03 fails to rebase since days ago.

You may remove this particular commit.

Oct 30 2019, 8:52 AM · Release-Engineering-Team, Beta-Cluster-Infrastructure

Oct 29 2019

Krenair added a comment to T236703: OpenStack server list shows outdated floating ip information.

Don't those commands pull directly from the nova/neutron API(s)? This is feeling like it may be an upstream thing.

Oct 29 2019, 9:53 PM · cloud-services-team, Cloud-VPS

Oct 28 2019

Krenair added a comment to T235627: Toolforge: upgrade main proxy servers to Debian Buster.

I don't know about your local network but it breaking specifically within cloud may indicate a labsaliaser problem - resolving tools.wmflabs.org should give the internal IP of tools-proxy-05?

Oct 28 2019, 10:09 PM · Toolforge, cloud-services-team (Kanban), Kubernetes