Page MenuHomePhabricator

Krenair (Alex Monk)
Wikimedia volunteer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:34 PM (246 w, 4 d)
Availability
Available
IRC Nick
Krenair
LDAP User
Alex Monk
MediaWiki User
Krenair [ Global Accounts ]

I am a Wikimedia volunteer helping in various technical ways. These days it's usually Beta Cluster, Cloud VPS, or Operations related. Since 2012 I've spent significant amounts of time involved in MediaWiki development, software deployments to the Wikimedia cluster, OTRS (email response to e.g. info-en@wikimedia.org addresses), and various other things.

Some of my old VisualEditor and other work (2014-2016) can be found under @AlexMonk-WMF instead.

I have opinions on things, which do not necessarily represent those of any organisation I am, have previously been, or will in the future be affiliated with.

Recent Activity

Sat, Jun 22

Krenair added a comment to T224561: Migrate remaining cloudvirt hosts to Stretch/Mitaka.

the first of those is basically empty but the rest are busy machines:

cloudvirt1014.eqiad.wmnet:
    canary-1014-01.testlabs.eqiad.wmflabs
cloudvirt1016.eqiad.wmnet:
    accounts-appserver4.account-creation-assistance.eqiad.wmflabs
    butterfly-m4m2.butterfly.eqiad.wmflabs
    Radon.codereview.eqiad.wmflabs
    Krypton.codereview.eqiad.wmflabs
    clm-web-01.community-labs-monitoring.eqiad.wmflabs
    clm-worker-01.community-labs-monitoring.eqiad.wmflabs
    dashiki-01.dashiki.eqiad.wmflabs
    deployment-restbase01.deployment-prep.eqiad.wmflabs
    deployment-logstash2.deployment-prep.eqiad.wmflabs
    deployment-changeprop.deployment-prep.eqiad.wmflabs
    deployment-sentry01.deployment-prep.eqiad.wmflabs
    deployment-memc06.deployment-prep.eqiad.wmflabs
    deployment-restbase02.deployment-prep.eqiad.wmflabs
    deployment-imagescaler01.deployment-prep.eqiad.wmflabs
    deployment-ircd.deployment-prep.eqiad.wmflabs
    deployment-zookeeper02.deployment-prep.eqiad.wmflabs
    deployment-etcd-01.deployment-prep.eqiad.wmflabs
    deployment-imagescaler02.deployment-prep.eqiad.wmflabs
    deployment-kafka-jumbo-1.deployment-prep.eqiad.wmflabs
    deployment-memc07.deployment-prep.eqiad.wmflabs
    deployment-snapshot01.deployment-prep.eqiad.wmflabs
    deployment-cpjobqueue.deployment-prep.eqiad.wmflabs
    deployment-mediawiki-07.deployment-prep.eqiad.wmflabs
    deployment-chromium01.deployment-prep.eqiad.wmflabs
    deployment-puppetdb02.deployment-prep.eqiad.wmflabs
    deployment-dumps-puppetmaster02.deployment-prep.eqiad.wmflabs
    deployment-jobrunner03.deployment-prep.eqiad.wmflabs
    deployment-ores01.deployment-prep.eqiad.wmflabs
    deployment-puppetmaster03.deployment-prep.eqiad.wmflabs
    deployment-cache-text05.deployment-prep.eqiad.wmflabs
    deployment-mwmaint01.deployment-prep.eqiad.wmflabs
    deployment-chromium02.deployment-prep.eqiad.wmflabs
    deployment-urldownloader02.deployment-prep.eqiad.wmflabs
    dwl.dwl.eqiad.wmflabs
    taxonbota.dwl.eqiad.wmflabs
    tofawiki02.fa-wp.eqiad.wmflabs
    integration-cumin.integration.eqiad.wmflabs
    integration-slave-jessie-1002.integration.eqiad.wmflabs
    integration-slave-docker-1041.integration.eqiad.wmflabs
    lizenzhinweisgenerator.lizenzhinweisgenerator.eqiad.wmflabs
    openrefine01.openrefine.eqiad.wmflabs
    cloud-bootstrapvz-stretch.openstack.eqiad.wmflabs
    otrs-oneclickspam-test.otrs.eqiad.wmflabs
    packagist-mirror1.packagist-mirror.eqiad.wmflabs
    partnermetrics-redis-01.partnermetrics.eqiad.wmflabs
    novaadminmadethis6.quotatest.eqiad.wmflabs
    logparse01.security-tools.eqiad.wmflabs
    kask-client.services.eqiad.wmflabs
    discovery-production-02.shiny-r.eqiad.wmflabs
    canary1016-01.testlabs.eqiad.wmflabs
    toolsbeta-sgegrid-shadow.toolsbeta.eqiad.wmflabs
    toolsbeta-sgecron-01.toolsbeta.eqiad.wmflabs
    toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad.wmflabs
    wmil.twl.eqiad.wmflabs
    video-redis.video.eqiad.wmflabs
    gfg01.video.eqiad.wmflabs
    videodev.video.eqiad.wmflabs
    encoding01.video.eqiad.wmflabs
    disposable.webperf.eqiad.wmflabs
    wikidata-constraints.wikidata-dev.eqiad.wmflabs
    whgi.wikidumpparse.eqiad.wmflabs
    elasticsearch-20.wikifactmine.eqiad.wmflabs
    elasticsearch-21.wikifactmine.eqiad.wmflabs
    wikitextexp-base-1002.wikitextexp.eqiad.wmflabs
    wikitextexp-expt-1002.wikitextexp.eqiad.wmflabs
    wm-bot-pg.wm-bot.eqiad.wmflabs
    wm-bot2.wm-bot.eqiad.wmflabs
    wikilabels.wmf-research-tools.eqiad.wmflabs
    diegoTest.wmf-research-tools.eqiad.wmflabs
cloudvirt1017.eqiad.wmnet:
    accounts-mwoauth.account-creation-assistance.eqiad.wmflabs
    af-puppetdb02.automation-framework.eqiad.wmflabs
    cloudinfra-db02.cloudinfra.eqiad.wmflabs
    commtech-2.commtech.eqiad.wmflabs
    dashiki-staging-01.dashiki.eqiad.wmflabs
    deployment-elastic07.deployment-prep.eqiad.wmflabs
    deployment-elastic06.deployment-prep.eqiad.wmflabs
    deployment-eventlog05.deployment-prep.eqiad.wmflabs
    design-research-methods.design.eqiad.wmflabs
    dumps-0.dumps.eqiad.wmflabs
    webservices.getstarted.eqiad.wmflabs
    gitservices.getstarted.eqiad.wmflabs
    Glampipe.glampipe.eqiad.wmflabs
    hound-puppet-02.hound.eqiad.wmflabs
    integration-r-lang-01.integration.eqiad.wmflabs
    integration-slave-docker-1040.integration.eqiad.wmflabs
    k8s-dzahn.k8splay.eqiad.wmflabs
    maps-tiles1.maps.eqiad.wmflabs
    maps-warper3.maps.eqiad.wmflabs
    compiler1001.puppet-diffs.eqiad.wmflabs
    meza-new2.qna.eqiad.wmflabs
    readers-web-master.reading-web-staging.eqiad.wmflabs
    tool.recommendation-api.eqiad.wmflabs
    missing-sections.recommendation-api.eqiad.wmflabs
    rec-wiki.recommendation-api.eqiad.wmflabs
    related-articles.recommendation-api.eqiad.wmflabs
    frama-test5.sentry.eqiad.wmflabs
    frama-test6-sb.sentry.eqiad.wmflabs
    kask.services.eqiad.wmflabs
    shinken-02.shinken.eqiad.wmflabs
    abogott-puppetmaster.testlabs.eqiad.wmflabs
    tools-sgeexec-0905.tools.eqiad.wmflabs
    tools-sgegrid-shadow.tools.eqiad.wmflabs
    tools-sgecron-01.tools.eqiad.wmflabs
    app-instance.videowiki.eqiad.wmflabs
    dumpgrepper.visualeditor.eqiad.wmflabs
    federated-commons.wikidata-federation.eqiad.wmflabs
    federated-wikidata.wikidata-federation.eqiad.wmflabs
    wmde-wikidiff2-jacnth.wikidiff2-wmde-dev.eqiad.wmflabs
    roope.wikidocumentaries.eqiad.wmflabs
    hupu.wikidocumentaries.eqiad.wmflabs
    puppetmaster-01.wikifactmine.eqiad.wmflabs
    wikilabels-experiment.wikilabels.eqiad.wmflabs
    wikilabels-02.wikilabels.eqiad.wmflabs
    wikimetrics-01.wikimetrics.eqiad.wmflabs
    ws-web.wikistream.eqiad.wmflabs
    wpx-redirects-01.wpx.eqiad.wmflabs
cloudvirt1021.eqiad.wmnet:
    zk1-1.analytics.eqiad.wmflabs
    zk1-3.analytics.eqiad.wmflabs
    antiharassment-web1.antiharassment.eqiad.wmflabs
    af-puppetmaster02.automation-framework.eqiad.wmflabs
    bastion-eqiad1-01.bastion.eqiad.wmflabs
    sylvester.catgraph.eqiad.wmflabs
    mx-out02.cloudinfra.eqiad.wmflabs
    cyberbot-exec-01.cyberbot.eqiad.wmflabs
    taxonbot.dwl.eqiad.wmflabs
    etcd05.etcd.eqiad.wmflabs
    etcd04.etcd.eqiad.wmflabs
    etcd06.etcd.eqiad.wmflabs
    fastcci-worker1.fastcci.eqiad.wmflabs
    fastcci-worker2.fastcci.eqiad.wmflabs
    google-api-proxy-02.google-api-proxy.eqiad.wmflabs
    hashtags-prod.hashtags.eqiad.wmflabs
    k8s-test-builder.hat-imagescalers.eqiad.wmflabs
    huggle-wl.huggle.eqiad.wmflabs
    xmlrcs.huggle.eqiad.wmflabs
    qube-master.k8splay.eqiad.wmflabs
    language-mleb-master.language.eqiad.wmflabs
    language-mleb-stable.language.eqiad.wmflabs
    filippo-log-stretch01.logging.eqiad.wmflabs
    filippo-log-jessie01.logging.eqiad.wmflabs
    mwv-stretch-migration.mediawiki-vagrant.eqiad.wmflabs
    T183456-stretch.mediawiki-vagrant.eqiad.wmflabs
    mixnmatch.mix-n-match.eqiad.wmflabs
    emoji.mobile.eqiad.wmflabs
    apps-team-tools.mobile.eqiad.wmflabs
    mwstake.mwstake.eqiad.wmflabs
    mwv-apt-01.mwv-apt.eqiad.wmflabs
    gnd-01.orig.eqiad.wmflabs
    calico-builder01.packaging.eqiad.wmflabs
    builder01.packaging.eqiad.wmflabs
    petscan-dev3.petscan.eqiad.wmflabs
    petscan3.petscan.eqiad.wmflabs
    puppet-phabricator.phabricator.eqiad.wmflabs
    pluggableauth-server.pluggableauth.eqiad.wmflabs
    quarry-web-01.quarry.eqiad.wmflabs
    experiments.reading-web-staging.eqiad.wmflabs
    relforge-search.search.eqiad.wmflabs
    wdsearch2.search.eqiad.wmflabs
    striker-uwsgi03.striker.eqiad.wmflabs
    diffscan.traffic.eqiad.wmflabs
    traffic-text-stretch.traffic.eqiad.wmflabs
    federated-wikis.wikidata-dev.eqiad.wmflabs
    xtools-dev04.xtools.eqiad.wmflabs
cloudvirt1022.eqiad.wmnet:
    accounts-db3.account-creation-assistance.eqiad.wmflabs
    deployment-server.analytics.eqiad.wmflabs
    af-lg.automation-framework.eqiad.wmflabs
    af-debmonitor.automation-framework.eqiad.wmflabs
    bastion-restricted-eqiad1-01.bastion.eqiad.wmflabs
    mx-out01.cloudinfra.eqiad.wmflabs
    deployment-elastic05.deployment-prep.eqiad.wmflabs
    deployment-mx02.deployment-prep.eqiad.wmflabs
    deployment-kafka-main-2.deployment-prep.eqiad.wmflabs
    deployment-webperf11.deployment-prep.eqiad.wmflabs
    wikimedia-ui.design.eqiad.wmflabs
    discourse1002.discourse.eqiad.wmflabs
    download-01.download.eqiad.wmflabs
    dumps-2.dumps.eqiad.wmflabs
    dumps-1.dumps.eqiad.wmflabs
    extdist-01.extdist.eqiad.wmflabs
    gerrit-test.git.eqiad.wmflabs
    gerrit-mysql.git.eqiad.wmflabs
    puppet-paladox.git.eqiad.wmflabs
    grantreview-03.grantreview.eqiad.wmflabs
    hhvm-stretch-jmm.hhvm.eqiad.wmflabs
    hhvm-jmm-vp9.hhvm.eqiad.wmflabs
    medbox3-iiab.iiab.eqiad.wmflabs
    integration-slave-docker-1043.integration.eqiad.wmflabs
    language-eg.language.eqiad.wmflabs
    language-cx.language.eqiad.wmflabs
    moses01-small.language.eqiad.wmflabs
    language-apertium2.language.eqiad.wmflabs
    mcr-full.mcr-dev.eqiad.wmflabs
    mwv-builder-02.mediawiki-vagrant.eqiad.wmflabs
    T183456-jessie.mediawiki-vagrant.eqiad.wmflabs
    mwoffliner4.mwoffliner.eqiad.wmflabs
    mwoffliner2.mwoffliner.eqiad.wmflabs
    newsletter-test.newsletter.eqiad.wmflabs
    labs-bootstrapvz-jessie.openstack.eqiad.wmflabs
    vmbuilder-trusty.openstack.eqiad.wmflabs
    ores-web-01.ores.eqiad.wmflabs
    ores-web-03.ores.eqiad.wmflabs
    phabricator.phabricator.eqiad.wmflabs
    phab-tin.phabricator.eqiad.wmflabs
    puppet-jmm-pmaster.puppet.eqiad.wmflabs
    puppet-jmm-pmaster-client.puppet.eqiad.wmflabs
    meza-new4.qna.eqiad.wmflabs
    quarry-db-01.quarry.eqiad.wmflabs
    quarry-worker-02.quarry.eqiad.wmflabs
    marvin-staging.reading-web-staging.eqiad.wmflabs
    readingwebstaging.reading-web-staging.eqiad.wmflabs
    tool-alpha.recommendation-api.eqiad.wmflabs
    gapfinder-tools.recommendation-api.eqiad.wmflabs
    cirrus-browser-bot.search.eqiad.wmflabs
    snuggle-wikidatawiki-01.snuggle.eqiad.wmflabs
    timeless2.social-tools.eqiad.wmflabs
    mc-clusterB-1.test-twemproxy.eqiad.wmflabs
    mc-clusterB-2.test-twemproxy.eqiad.wmflabs
    toolsbeta-sgebastion-04.toolsbeta.eqiad.wmflabs
    toolsbeta-sgeexec-0901.toolsbeta.eqiad.wmflabs
    toolsbeta-sgegrid-master.toolsbeta.eqiad.wmflabs
    relic-stretch.toolserver-legacy.eqiad.wmflabs
    traffic-puppetmaster.traffic.eqiad.wmflabs
    utrs-beta-production.utrs.eqiad.wmflabs
    utrs-production2.utrs.eqiad.wmflabs
    utrs-database2.utrs.eqiad.wmflabs
    wcdo.wcdo.eqiad.wmflabs
    wbregistry-01.wikibase-registry.eqiad.wmflabs
    orig-01.wikibase-registry.eqiad.wmflabs
    federated-wikis2.wikidata-dev.eqiad.wmflabs
    pst.wikidata-primary-sources-tool.eqiad.wmflabs
    wmde-wikidiff2-patched.wikidiff2-wmde-dev.eqiad.wmflabs
    wmde-wikidiff2-unpatched.wikidiff2-wmde-dev.eqiad.wmflabs
    sqltest02.wikidiff2-wmde-dev.eqiad.wmflabs
    wikilabels-staging-01.wikilabels.eqiad.wmflabs
    scholarships-02.wikimania-support.eqiad.wmflabs
    wikispeech-tts-release.wikispeech.eqiad.wmflabs
    parsing-qa-01.wikitextexp.eqiad.wmflabs
    dannyb.wildcat.eqiad.wmflabs
    xtools-prod05.xtools.eqiad.wmflabs
    xtools-prod03.xtools.eqiad.wmflabs
cloudvirt1023.eqiad.wmnet:
    k4-1.analytics.eqiad.wmflabs
    k4-2.analytics.eqiad.wmflabs
    zk1-2.analytics.eqiad.wmflabs
    fishbone.catgraph.eqiad.wmflabs
    logintest2.catgraph.eqiad.wmflabs
    codesearch4.codesearch.eqiad.wmflabs
    cvn-app8.cvn.eqiad.wmflabs
    cvn-app9.cvn.eqiad.wmflabs
    cvn-apache9.cvn.eqiad.wmflabs
    cyberbot-db-01.cyberbot.eqiad.wmflabs
    deployment-sca02.deployment-prep.eqiad.wmflabs
    discourse-mw.discourse.eqiad.wmflabs
    dumps-3.dumps.eqiad.wmflabs
    eventmetrics-prod01.eventmetrics.eqiad.wmflabs
    fastcci-new-master.fastcci.eqiad.wmflabs
    wikiedubackups.globaleducation.eqiad.wmflabs
    huggle-deb-builder.huggle.eqiad.wmflabs
    saucelabs-02.integration.eqiad.wmflabs
    integration-slave-jessie-1001.integration.eqiad.wmflabs
    integration-slave-jessie-1004.integration.eqiad.wmflabs
    qube-node2.k8splay.eqiad.wmflabs
    qube-node1.k8splay.eqiad.wmflabs
    language-cx1.language.eqiad.wmflabs
    cxserver.language.eqiad.wmflabs
    upgrader-04.library-upgrader.eqiad.wmflabs
    matrix-synapse-01.matrix.eqiad.wmflabs
    mcr-base.mcr-dev.eqiad.wmflabs
    mcr-sdc.mcr-dev.eqiad.wmflabs
    ores-worker-01.ores.eqiad.wmflabs
    ores-worker-02.ores.eqiad.wmflabs
    ores-lb-03.ores.eqiad.wmflabs
    ores-redis-02.ores.eqiad.wmflabs
    ores-staging-01.ores-staging.eqiad.wmflabs
    ores-misc-01.ores-staging.eqiad.wmflabs
    phlogiston-4.phlogiston.eqiad.wmflabs
    cindy.pluggableauth.eqiad.wmflabs
    captcha-tf-43.privpol-captcha.eqiad.wmflabs
    captcha-imageprocessing-11.privpol-captcha.eqiad.wmflabs
    puppet-ema-2.puppet.eqiad.wmflabs
    keith-puppetmaster.puppet.eqiad.wmflabs
    puppet-jmm-kernel-stretch2.puppet.eqiad.wmflabs
    puppet-jmm-kernel-jessie.puppet.eqiad.wmflabs
    quarry-worker-01.quarry.eqiad.wmflabs
    trending.reading-web-staging.eqiad.wmflabs
    recommendation-api-build.recommendation-api.eqiad.wmflabs
    clickmodel.search.eqiad.wmflabs
    rel2.search.eqiad.wmflabs
    sentry-builder.sentry.eqiad.wmflabs
    appservice.services.eqiad.wmflabs
    suggestbot-01.suggestbot.eqiad.wmflabs
    mc-clusterA-2.test-twemproxy.eqiad.wmflabs
    canary1023-01.testlabs.eqiad.wmflabs
    puppetmaster.thumbor.eqiad.wmflabs
    tools-static-12.tools.eqiad.wmflabs
    tools-sgegrid-master.tools.eqiad.wmflabs
    toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad.wmflabs
    visualeditor-test2.visualeditor.eqiad.wmflabs
    misc-01.wikibase-registry.eqiad.wmflabs
    libcanada-01.wikibase-registry.eqiad.wmflabs
    wikibrain-embeddings-02.wikibrain.eqiad.wmflabs
    wikibrain-embeddings-01.wikibrain.eqiad.wmflabs
    wikibase-stretch.wikidata-dev.eqiad.wmflabs
    wmde-wikidiff2-debug.wikidiff2-wmde-dev.eqiad.wmflabs
    elasticsearch-01.wikifactmine.eqiad.wmflabs
    wikilabels-backups.wikilabels.eqiad.wmflabs
    wikispeech-wiki-stretch.wikispeech.eqiad.wmflabs
    wikispeech-tts-dev.wikispeech.eqiad.wmflabs
    wikistats-greyhound.wikistats.eqiad.wmflabs
    wpx-mediawiki-02.wpx.eqiad.wmflabs
Sat, Jun 22, 8:18 PM · cloud-services-team, Operations
Krenair added a comment to T215888: openstack-browser: include deployment view support.

If it has to be in the admin project that may be problematic - IIRC that is the one project that novaobserver cannot observe, can't remember if there was a good reason for that.

Sat, Jun 22, 8:16 PM · Tools, cloud-services-team (Kanban)
Krenair closed T219424: Decide how we're going to handle certificates for the puppetmaster migration as Resolved.
Sat, Jun 22, 4:36 PM · cloud-services-team (Kanban), Cloud-Services
Krenair closed T219424: Decide how we're going to handle certificates for the puppetmaster migration, a subtask of T171188: Move the main WMCS puppetmaster into the Labs realm, as Resolved.
Sat, Jun 22, 4:36 PM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Krenair closed T220268: Consider ways to make puppetmaster CA changes smoother on the puppet client end as Resolved.

Wish I'd done this years ago. It seems to have worked and allows us to effortlessly move instances between puppetmasters - which should prove very helpful to the puppetmaster realm migration.

Sat, Jun 22, 4:31 PM · Puppet, cloud-services-team (Kanban), Cloud-Services
Krenair closed T220268: Consider ways to make puppetmaster CA changes smoother on the puppet client end, a subtask of T219424: Decide how we're going to handle certificates for the puppetmaster migration, as Resolved.
Sat, Jun 22, 4:31 PM · cloud-services-team (Kanban), Cloud-Services

Thu, Jun 20

Krenair added a comment to T225253: Access Q re maint1002.

enwiki would be s1, eswiki s7, and cawiki s7. Not sure what the problem with the defaults file is - are you able to open the file? It will contain a password so do not paste it here.

Thu, Jun 20, 10:08 PM · SRE-Access-Requests, Operations
Krenair added a comment to T225253: Access Q re maint1002.

Ah, I see the old analytics-store got removed. Baring in mind eswiki would live on s7, and looking at the docs you linked, your last two attempts should probably be pretty close. Try mysql --defaults-file=/etc/mysql/conf.d/research-client.cnf -h s7-analytics-replica.eqiad.wmnet -P 3317 eswiki -e 'describe cx_corpora;' ?

Thu, Jun 20, 9:52 PM · SRE-Access-Requests, Operations

Tue, Jun 11

Krenair added a comment to T225557: Puppet fails on newly created deployment-wikifeeds01.

Ok. I'm kind of hoping that with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/506873/ and its parent we can make the puppetmaster switch more or less seamless with just the signing step to perform.

Tue, Jun 11, 10:04 PM · Reading-Infrastructure-Team-Backlog, Cloud-VPS, Beta-Cluster-Infrastructure
Krenair closed T225557: Puppet fails on newly created deployment-wikifeeds01 as Resolved.

Ran cd /var/lib/puppet; mv ssl ssl_old; rm /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; nano /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; update-ca-certificates --fresh; puppet agent -tv, gave it a copy of the CA cert from the puppetmaster, signed the new client cert on the puppetmaster. Puppet cert error is gone, there's a new error which would be a different task.

Tue, Jun 11, 9:51 PM · Reading-Infrastructure-Team-Backlog, Cloud-VPS, Beta-Cluster-Infrastructure
Krenair closed T225557: Puppet fails on newly created deployment-wikifeeds01, a subtask of T170455: Extract the feed endpoints from PCS into a new wikifeeds service, as Resolved.
Tue, Jun 11, 9:51 PM · Wikifeeds, Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service
Krenair added a comment to T225557: Puppet fails on newly created deployment-wikifeeds01.

That's what always happens, you still have to go through and manually sort out the situation with certs.

Tue, Jun 11, 9:48 PM · Reading-Infrastructure-Team-Backlog, Cloud-VPS, Beta-Cluster-Infrastructure
Krenair awarded T225553: gmail users being suspended from mediawiki-l due to excessive bounces due to DMARC a Heartbreak token.
Tue, Jun 11, 9:38 PM · Operations, Wikimedia-Mailing-lists

Mon, Jun 10

Krenair renamed T225261: CentralNotice setting a surprising content security policy in production when using &banner= URL parameter from CentralNotice setting a surprising content security policy in production to CentralNotice setting a surprising content security policy in production when using &banner= URL parameter.
Mon, Jun 10, 8:15 PM · Release-Engineering-Team-TODO, Fundraising-Backlog, Release-Engineering-Team, Security, MediaWiki-extensions-CentralNotice

Fri, Jun 7

Krenair added a comment to T225269: Verify that all mailman mailing lists have private_roster=2.

Don't most of our lists require people to have a list admin password to read the subscriber list? Do we have any that don't?

Fri, Jun 7, 1:33 AM · Operations, Wikimedia-Mailing-lists
Krenair renamed T225269: Verify that all mailman mailing lists have private_roster=2 from Mailman: Consider restricting access to members to Consider restricting access to list subscriber list.
Fri, Jun 7, 1:31 AM · Operations, Wikimedia-Mailing-lists
Krenair added a comment to T225253: Access Q re maint1002.

Okay. You might just want to check that something like mysql --defaults-file=/etc/mysql/conf.d/research-client.cnf -h analytics-store.eqiad.wmnet <insert wiki ID> -e 'describe cx_translations;' works as expected on stat1006. Not able to test it myself.

Fri, Jun 7, 12:21 AM · SRE-Access-Requests, Operations

Thu, Jun 6

Krenair updated subscribers of T225261: CentralNotice setting a surprising content security policy in production when using &banner= URL parameter.
Thu, Jun 6, 10:58 PM · Release-Engineering-Team-TODO, Fundraising-Backlog, Release-Engineering-Team, Security, MediaWiki-extensions-CentralNotice
Krenair added a comment to T225253: Access Q re maint1002.

That's from ContentTranslation right? You should be able to get all this stuff already by being in the researcher group. I don't think the maintenance hosts store this either, they just have access to it by virtue of wikiuser/wikiadmin MySQL credentials. Restricted/deployment users would be expected to get to it from a MW machine, but analytics/research users should probably be using a stat box's access to analytics replicas of the MW databases.

Thu, Jun 6, 10:16 PM · SRE-Access-Requests, Operations
Krenair edited projects for T225253: Access Q re maint1002, added: SRE-Access-Requests; removed Access-Policy.

What data are you trying to get to exactly?

Thu, Jun 6, 9:13 PM · SRE-Access-Requests, Operations

Wed, Jun 5

Krenair renamed T224289: "sso" project did not get sso.wmflabs.org zone created automatically from Allow DNS changes for "sso" project to "sso" project did not get sso.wmflabs.org zone created automatically.
Wed, Jun 5, 11:02 PM · Cloud-VPS
Krenair closed T225168: New OpenStack control-plane nodes can't talk to novaproxy, a subtask of T224981: rabbitmq: connectivity issues between cloudservices1004 and rabbitmq, as Resolved.
Wed, Jun 5, 10:52 PM · cloud-services-team (Kanban)
Krenair closed T225168: New OpenStack control-plane nodes can't talk to novaproxy as Resolved.
Wed, Jun 5, 10:52 PM · cloud-services-team (Kanban)
Krenair added a comment to T225168: New OpenStack control-plane nodes can't talk to novaproxy.

Also did other cleanup of that list visible in https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL#2019-06-05
List is now:

Wed, Jun 5, 10:52 PM · cloud-services-team (Kanban)
Krenair added a comment to T225168: New OpenStack control-plane nodes can't talk to novaproxy.

Already did the second service node: https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Project-proxy/SAL&diff=1828608&oldid=1819110

Wed, Jun 5, 10:47 PM · cloud-services-team (Kanban)
Krenair added a comment to T225168: New OpenStack control-plane nodes can't talk to novaproxy.

(@JHedden helped identify this one)

Wed, Jun 5, 10:46 PM · cloud-services-team (Kanban)
Krenair added a comment to T224265: Redirect svgtranslate from toolserver.org.
Wed, Jun 5, 10:25 PM · cloud-services-team (Kanban), Cloud-VPS, Community-Tech (Kanban), Patch-For-Review, SVG Translate Tool
Krenair added a comment to T225048: ipblocks.ipb_by_text database column has gone missing from English Wikipedia database replica.

https://wikitech.wikimedia.org/wiki/News/Actor_storage_changes_on_the_Wiki_Replicas

Wed, Jun 5, 7:42 AM · Data-Services, Documentation

Tue, Jun 4

Krenair awarded T225025: Request new Flavor for integration Cloud VPS project a Like token.
Tue, Jun 4, 9:54 PM · cloud-services-team (Kanban), Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, Cloud-VPS (Quota-requests)

Mon, Jun 3

Krenair added a comment to T224265: Redirect svgtranslate from toolserver.org.

I went to look at https://commons.wikimedia.org/w/index.php?target=https%3A%2F%2Ftoolserver.org%2F%7Enikola%2Fsvgtranslate.php&title=Special%3ALinkSearch but the svg= data I tried on the new svgtranslate did not appear to work. Does it support upload.wm.o URLs?

Mon, Jun 3, 7:57 PM · cloud-services-team (Kanban), Cloud-VPS, Community-Tech (Kanban), Patch-For-Review, SVG Translate Tool
Krenair added a project to T220505: Decommission iron: Cloud-VPS.

'install access for WMCS' struck me as odd so I asked around a bit:

<bstorm_> Iron has been used for cloudvirt installs in the past
<andrewbogott> Normally we access new unpuppetized servers from the puppetmasters.  They aren't allowed to access the cloudvirts though, due to a network rule I don't understand.  So we use iron instead.
<andrewbogott> As far as I know it's still the only way.
<bstorm_> Yup
Mon, Jun 3, 7:18 PM · Cloud-VPS, ops-eqiad, decommission, Operations
Krenair closed T224865: Requesting access to deployment-prep for @Urbanecm as Resolved.
Mon, Jun 3, 3:35 PM · Beta-Cluster-Infrastructure

Sun, Jun 2

Krenair added a comment to T224840: category; British leprologists.
Sun, Jun 2, 8:13 PM

Sat, Jun 1

Krenair added a comment to T221526: Explain on Special:UrlShortener that it can only be used on Meta-Wiki.
krenair@deployment-deploy01:~$ mwscript eval.php metawiki
> echo wfMessage('urlshortener-disabled')->parse();
Creating new short URLs is temporarily disabled.
> ^D
krenair@deployment-deploy01:~$ mwscript eval.php enwiki
> echo wfMessage('urlshortener-disabled')->parse();
Short links can only be created via <a href="https://meta.wikimedia.beta.wmflabs.org/wiki/Special:UrlShortener" class="extiw" title="m:Special:UrlShortener">Special:UrlShortener</a> page on meta.wikimedia.org.
Sat, Jun 1, 1:53 PM · MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), Patch-For-Review, MediaWiki-extensions-UrlShortener, WikimediaMessages
Krenair closed T221526: Explain on Special:UrlShortener that it can only be used on Meta-Wiki as Resolved.

Seems beta is not configured to test this one properly as we set UrlShortenerReadOnly to false everywhere there.

Sat, Jun 1, 1:51 PM · MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), Patch-For-Review, MediaWiki-extensions-UrlShortener, WikimediaMessages
Krenair added a comment to T224000: wmcs-wikireplica-dns: error: circular reference detected.

I've tested and modern Designate appears to do better error handling of this:

krenair@labs-t224000-alex-osdev:~/devstack$ cat test_mwoc.py
import mwopenstackclients
dns_mgr = mwopenstackclients.DnsManager(mwopenstackclients.Clients(
    username="admin",
    password="password",
    url="http://labs-t224000-alex-osdev/identity/v3",
    project="demo"
), None)
dns_mgr.create_recordset('2bc35734-0fd6-4b04-83dd-88e1672dbfb6', "asd.{u'status': u'ACTIVE'}.alextest.wmflabs.org.", 'A', ['127.0.0.1'])
#dns_mgr.create_recordset('2bc35734-0fd6-4b04-83dd-88e1672dbfb6', "asd.abcstatusdefugACTIVEhi.alextest.wmflabs.org.", 'A', ['127.0.0.1'])
Sat, Jun 1, 1:28 PM · Patch-For-Review, cloud-services-team (Kanban)
Krenair closed T186993: Beta Cluster search box displays unexisting pages as results as Resolved.

It looks like this got resolved by Joe in the child ticket

Sat, Jun 1, 1:14 PM · Discovery-Search, Services (next), MediaWiki-Search, Beta-Cluster-Infrastructure

Fri, May 31

Krenair renamed T224708: Drop most of mwopenstackclients.DnsManager in favour of designateclient from Once all our stuff runs stretch, drop mwopenstackclients.DnsManager in favour of designateclient to Drop most of mwopenstackclients.DnsManager in favour of designateclient.
Fri, May 31, 8:57 AM · Patch-For-Review, Cloud-VPS
Krenair added a comment to T224708: Drop most of mwopenstackclients.DnsManager in favour of designateclient.

All of this is replaceable with the stuff under designateclient.zones and designateclient.recordsets, except the ensure stuff which probably belongs in applications rather than the library.

The ensure stuff is definitely what I would want to keep. I don't understand the "probably belongs in applications" part of your argument. Libraries exist to reduce code duplication and make proper implementation of algorithms easier. ensure_recordset is used in 3 utility tools. I don't see why that code should be copied rather than living in our shim library that makes using the upstream libraries easier.

Fri, May 31, 8:55 AM · Patch-For-Review, Cloud-VPS
Krenair added a comment to T224708: Drop most of mwopenstackclients.DnsManager in favour of designateclient.

Looked at which stuff runs jessie - based on parsing the dhcpd files it's, cloudservices1003, and a bunch of cloudvirt/labpuppetmaster/labmon/labstore hosts
Users of mwopenstackclients.DnsManager are wmcs-dns-floating-ip-updater.py which runs on control hosts, and the util::admin_scripts scripts which all seem to run on control too.

Fri, May 31, 2:40 AM · Patch-For-Review, Cloud-VPS
Krenair added a comment to T224708: Drop most of mwopenstackclients.DnsManager in favour of designateclient.

Well that would leave these:

def zones(self, name=None, params=None):
def create_zone(
def ensure_zone(
def recordsets(self, uuid, name=None, params=None):
def create_recordset(
def update_recordset(
def ensure_recordset(
def delete_recordset(self, uuid, rs):
Fri, May 31, 2:31 AM · Patch-For-Review, Cloud-VPS
Krenair created T224708: Drop most of mwopenstackclients.DnsManager in favour of designateclient.
Fri, May 31, 2:13 AM · Patch-For-Review, Cloud-VPS

Thu, May 30

Krenair added a comment to T222089: Allow URL shortening for wikimediafoundation.org domain.

wmflabs.org should probably be a whole separate ticket.

Thu, May 30, 10:57 PM · wikimediafoundation.org, Wikimedia-Site-requests, MediaWiki-extensions-UrlShortener
Krenair added a comment to T198901: Migrate production services to kubernetes using the pipeline.

Okay. Is it potentially in-scope i.e. should it appear on one of the lists in this task?

Thu, May 30, 7:13 PM · Release-Engineering-Team, Release-Engineering-Team-TODO, Core Platform Team Backlog (Watching / External), Epic, Services (watching), Operations, Release Pipeline

Wed, May 29

Krenair updated the task description for T218729: Migrate away from Debian Jessie to Debian Stretch.
Wed, May 29, 4:26 PM · Beta-Cluster-Infrastructure
Krenair added a comment to T218609: Figure out future for newly created deployment-prep jessie instances.

I pressed some buttons and it started. Success?

Wed, May 29, 3:40 PM · Beta-Cluster-Infrastructure
Krenair added a comment to T218609: Figure out future for newly created deployment-prep jessie instances.

I made deployment-sessionstore02 for it but couldn't get cassandra to work yet.

Wed, May 29, 3:35 PM · Beta-Cluster-Infrastructure
Krenair added a comment to T198901: Migrate production services to kubernetes using the pipeline.

Is apertium part of the cxserver migration?

Wed, May 29, 1:58 PM · Release-Engineering-Team, Release-Engineering-Team-TODO, Core Platform Team Backlog (Watching / External), Epic, Services (watching), Operations, Release Pipeline
Krenair added a comment to T185319: IRC RecentChanges feed: code stewardship request.

One of the analytics engineers.

Wed, May 29, 1:56 PM · Tools, Operations, Analytics, Wikimedia-IRC-RC-Server, Code-Stewardship-Reviews
Krenair updated the task description for T218729: Migrate away from Debian Jessie to Debian Stretch.
Wed, May 29, 1:54 PM · Beta-Cluster-Infrastructure
Krenair added a comment to T220235: Migrate Beta cluster services to use Kubernetes .

If we're content to stick with simple Docker instances due to beta's relatively small scale, then I suggest we close this and have individual tasks for services needing to be migrated in future?

Wed, May 29, 1:39 PM · Editing-team, Core Platform Team Backlog (Next), Services (next), Kubernetes, Release Pipeline, serviceops, Beta-Cluster-Infrastructure
Krenair added a comment to T223345: Zotero container: Production is running candidate version, last production version is broken due to lack of ca-certificates package.

I'm removing the parent task, I guess I'll leave this open to track the fact that production is running a non-production release, unless there's an existing Phabricator task about it.

Wed, May 29, 1:36 PM · Core Platform Team Backlog (Watching / External), Beta-Cluster-reproducible, Editing-team, Services (next), serviceops
Krenair removed a parent task for T223345: Zotero container: Production is running candidate version, last production version is broken due to lack of ca-certificates package: T220235: Migrate Beta cluster services to use Kubernetes .
Wed, May 29, 1:35 PM · Core Platform Team Backlog (Watching / External), Beta-Cluster-reproducible, Editing-team, Services (next), serviceops
Krenair removed a subtask for T220235: Migrate Beta cluster services to use Kubernetes : T223345: Zotero container: Production is running candidate version, last production version is broken due to lack of ca-certificates package.
Wed, May 29, 1:35 PM · Editing-team, Core Platform Team Backlog (Next), Services (next), Kubernetes, Release Pipeline, serviceops, Beta-Cluster-Infrastructure
Krenair closed T223344: Citoid container: Our config.yaml provided via Docker is unused? as Invalid.

Now that I think more about it, that's probably not an entirely reasonable expectation. Different services might decide to accept config paths in different ways etc.

Wed, May 29, 1:34 PM · Beta-Cluster-reproducible, Editing-team, Core Platform Team Backlog (Next), Services (next), serviceops
Krenair closed T223344: Citoid container: Our config.yaml provided via Docker is unused?, a subtask of T220235: Migrate Beta cluster services to use Kubernetes , as Invalid.
Wed, May 29, 1:34 PM · Editing-team, Core Platform Team Backlog (Next), Services (next), Kubernetes, Release Pipeline, serviceops, Beta-Cluster-Infrastructure
Krenair updated the task description for T218729: Migrate away from Debian Jessie to Debian Stretch.
Wed, May 29, 12:32 PM · Beta-Cluster-Infrastructure
Krenair updated the task description for T218729: Migrate away from Debian Jessie to Debian Stretch.
Wed, May 29, 10:33 AM · Beta-Cluster-Infrastructure
Krenair added a comment to T168494: tracking task: jessie -> stretch.

T224549: Track remaining jessie systems in production

Wed, May 29, 10:24 AM · Operations

Tue, May 28

Framawiki awarded T198813: Integrate MFA into Gerrit a Burninate token.
Tue, May 28, 6:07 PM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Security, Upstream, Gerrit

Mon, May 27

Krenair created T224447: deployment-sentry01 puppet has a broken exec.
Mon, May 27, 10:04 PM · Sentry, Beta-Cluster-Infrastructure
Krenair added a comment to T224442: I can't load tools.wmflabs.org.

Do you have dig? If so can you dig wmflabs.org NS @cloud-ns0.wikimedia.org and dig wmflabs.org NS?

Mon, May 27, 7:29 PM · Toolforge, cloud-services-team (Kanban)

May 24 2019

Krenair added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

If we're going to divide things up in that manner it would strike me as a bit weird to have the full purposes of the different domains be indistinguishable from the outside, covered only deep in some docs/comments somewhere.

May 24 2019, 10:53 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair updated the task description for T220205: Define constraints for cloudelastic use cases.
May 24 2019, 10:17 PM · Discovery-Search
Krenair added a comment to T224272: Request increased quota for sso Cloud VPS project.

Yeah, clearly something went wrong (or my expectation that newly created projects get themselves a zone created was wrong) as it is definitely missing.

May 24 2019, 6:02 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Krenair added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.
  • we could use $subdomain.wmcloud.org if this subdomain is not hosted by desginate (to avoid chicken-egg problems)
May 24 2019, 5:13 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair added a comment to T224289: "sso" project did not get sso.wmflabs.org zone created automatically.

(And even if we did it probably would not work anyway due to the wmflabs.org zone existing in a tenant normal users cannot control)

May 24 2019, 4:26 PM · Cloud-VPS
Krenair added a comment to T224289: "sso" project did not get sso.wmflabs.org zone created automatically.

This zone should already have been created during project creation, I do not think we need to hand out zone creation permissions to users.

May 24 2019, 4:25 PM · Cloud-VPS
Krenair awarded T224272: Request increased quota for sso Cloud VPS project a Like token.
May 24 2019, 4:21 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Krenair added a comment to T224272: Request increased quota for sso Cloud VPS project.

I think one is fine to start, if we need more, I'll make a new task. I also seem to be unable to crate a zone for sso.wmflabs.org, getting "Fehler: Unable to create the zone.", is that something that also needs to be granted?

May 24 2019, 4:21 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Krenair added a comment to T219639: project-local puppetmasters getting reset to labs-puppetmaster.

This has happened again:

(24) deployment-acme-chief[03-04].deployment-prep.eqiad.wmflabs,deployment-aqs[01-03].deployment-prep.eqiad.wmflabs,deployment-cache-text05.deployment-prep.eqiad.wmflabs,deployment-cache-upload05.deployment-prep.eqiad.wmflabs,deployment-chromium02.deployment-prep.eqiad.wmflabs,deployment-cumin02.deployment-prep.eqiad.wmflabs,deployment-db06.deployment-prep.eqiad.wmflabs,deployment-docker-citoid01.deployment-prep.eqiad.wmflabs,deployment-docker-cxserver01.deployment-prep.eqiad.wmflabs,deployment-docker-mathoid01.deployment-prep.eqiad.wmflabs,deployment-hadoop-test-[1-3].deployment-prep.eqiad.wmflabs,deployment-imagescaler03.deployment-prep.eqiad.wmflabs,deployment-maps05.deployment-prep.eqiad.wmflabs,deployment-ms-be[05-06].deployment-prep.eqiad.wmflabs,deployment-ms-fe03.deployment-prep.eqiad.wmflabs,deployment-poolcounter05.deployment-prep.eqiad.wmflabs,deployment-prometheus02.deployment-prep.eqiad.wmflabs,deployment-urldownloader02.deployment-prep.eqiad.wmflabs
----- OUTPUT of 'grep labs-puppet...ppet/puppet.conf' -----                                                                                                                                                 
server = labs-puppetmaster.wikimedia.org
May 22 13:39:01 deployment-acme-chief03 CRON[27438]: (root) CMD (/usr/local/sbin/puppet-run > /dev/null 2>&1)
May 22 13:39:02 deployment-acme-chief03 puppet-agent-cronjob: Sleeping 20 for random splay
May 22 13:39:26 deployment-acme-chief03 puppet-agent[27821]: Downgrading to PSON for future requests
May 22 13:39:26 deployment-acme-chief03 puppet-agent[27821]: Using configured environment 'production'
May 22 13:39:26 deployment-acme-chief03 puppet-agent[27821]: Retrieving pluginfacts
May 22 13:39:26 deployment-acme-chief03 puppet-agent[27821]: Retrieving plugin
May 22 13:39:26 deployment-acme-chief03 puppet-agent[27821]: Loading facts
May 22 13:39:36 deployment-acme-chief03 puppet-agent[27821]: Caching catalog for deployment-acme-chief03.deployment-prep.eqiad.wmflabs
May 22 13:39:36 deployment-acme-chief03 puppet-agent[27821]: Applying configuration version '1558532370'
May 22 13:39:37 deployment-acme-chief03 puppet-agent[27821]: Computing checksum on file /etc/apt/sources.list.d/project-aptly.list
May 22 13:39:37 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Apt/File[/etc/apt/sources.list.d/project-aptly.list]) Filebucketed /etc/apt/sources.list.d/project-aptly.list to puppet with sum 359c83a1139d09149269ef9819af28cf
May 22 13:39:37 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Apt/File[/etc/apt/sources.list.d/project-aptly.list]/ensure) removed
May 22 13:39:37 deployment-acme-chief03 crontab[27908]: (root) LIST (root)
May 22 13:39:37 deployment-acme-chief03 crontab[27911]: (root) LIST (prometheus)
May 22 13:39:37 deployment-acme-chief03 crontab[27914]: (root) LIST (acme-chief)
May 22 13:39:39 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Nrpe/Package[nagios-plugins]/ensure) created
May 22 13:39:39 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Nrpe/Package[nagios-plugins-basic]/ensure) created
May 22 13:39:39 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Nrpe/Package[nagios-plugins-standard]/ensure) created
May 22 13:39:40 deployment-acme-chief03 puppet-agent[27821]: openstack::clientpackages::vms::mitaka::buster: no special configuration yet
May 22 13:39:40 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Openstack::Clientpackages::Vms::Mitaka::Buster/Notify[openstack::clientpackages::vms::mitaka::buster: no special configuration yet]/message) defined 'message' as 'openstack::clientpackages::vms::mitaka::buster: no special configuration yet'
May 22 13:39:40 deployment-acme-chief03 puppet-agent[27821]: The LDAP client stack for this host is: classic
May 22 13:39:40 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message) defined 'message' as 'The LDAP client stack for this host is: classic'
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content) 
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content) --- /etc/puppet/puppet.conf.d/10-main.conf#0112019-04-16 18:09:54.137878345 +0000
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content) +++ /tmp/puppet-file20190522-27821-dcuw4f#0112019-05-22 13:39:41.326089802 +0000
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content) @@ -11,7 +11,7 @@
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content)  factpath = $vardir/lib/facter
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content)  
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content)  [agent]
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content) -server = deployment-puppetmaster03.deployment-prep.eqiad.wmflabs
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content) +server = labs-puppetmaster.wikimedia.org
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content)  
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content)  
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content)  daemonize = false
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: Computing checksum on file /etc/puppet/puppet.conf.d/10-main.conf
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]) Filebucketed /etc/puppet/puppet.conf.d/10-main.conf to puppet with sum 73ac1d853f8c1aeb622e53a03efe0b07
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content) content changed '{md5}73ac1d853f8c1aeb622e53a03efe0b07' to '{md5}a782d5f4005e93bc461131c7e16def71'
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]) Scheduling refresh of Exec[delete master certs]
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]) Scheduling refresh of Exec[compile puppet.conf]
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Exec[delete master certs]) Triggered 'refresh' from 1 event
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: Computing checksum on file /etc/ssh/userkeys/root
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Passwords::Root/Ssh::Userkey[root]/File[/etc/ssh/userkeys/root]) Filebucketed /etc/ssh/userkeys/root to puppet with sum 701b5950ec0373eb918970a97aa64605
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Passwords::Root/Ssh::Userkey[root]/File[/etc/ssh/userkeys/root]/content) content changed '{md5}701b5950ec0373eb918970a97aa64605' to '{md5}af9fc71bb296ceacbc6ad11ff022a3ef'
May 22 13:39:41 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Base::Puppet/Exec[compile puppet.conf]) Triggered 'refresh' from 1 event
May 22 13:39:42 deployment-acme-chief03 ssh-agent[23648]: debug2: fd 4 setting O_NONBLOCK
May 22 13:39:42 deployment-acme-chief03 ssh-agent[23648]: debug1: process_message: socket 1 (fd=4) type 11
May 22 13:39:42 deployment-acme-chief03 ssh-agent[23648]: debug1: process_message: socket 1 (fd=4) type 13
May 22 13:39:43 deployment-acme-chief03 puppet-agent[27821]: (/Stage[main]/Acme_chief::Server/Exec[/usr/local/bin/acme-chief-certs-sync]/returns) executed successfully
May 22 13:39:44 deployment-acme-chief03 puppet-agent[27821]: Applied catalog in 7.93 seconds
May 24 2019, 3:56 PM · cloud-services-team (Kanban)
Krenair added a comment to T220069: Build authenticating reverse proxy for Cloud CirrusSearch replicas.

I'm considering having a go at this. I might use TLS client certificates rather than HTTP Basic auth + lua + redis.
@EBernhardson/@Mathew.onipe (or anyone else involved in cloudelastic), are you likely to get to this before mid-August?

May 24 2019, 1:41 AM · Data-Services, Discovery-Search, Elasticsearch, Discovery

May 23 2019

Krenair added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

Another thing to consider if we're really talking about using the prod caches is that currently those endpoints are not exposed to the world, only some Wikimedia hosts (certain prod hosts + all of labs). I'm not sure if such restrictions are actually necessary though.

May 23 2019, 9:44 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

so, after a quick check you should consider several things:

  • wikimedia.org is a canonical domain for WMF, everything is expected to use secure TLS settings.
  • if you aim to use the production caching layer, the hostnames must match *.wikimedia.org

Does this mean that ldap-ro.eqiad.wikimedia.org is already violating the TLS policy? It supports both tls and unecrypted ldap, and also doesn't match *.wikimedia.org

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/496858/

May 23 2019, 8:02 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair closed T224204: Request to be a interface-admin on zh Beta Cluster as Resolved.
May 23 2019, 5:36 PM · Beta-Cluster-Infrastructure
Krenair added projects to T224180: Send some LibreNMS alerts to dcops and netops only: observability, netops, DC-Ops, Operations.
May 23 2019, 5:34 PM · Operations, DC-Ops, netops, observability
Restricted Application added a project to T223902: cloudcontrol: decide on FQDN for service endpoints: Operations.
May 23 2019, 5:17 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

I do think TLS should be on OpenStack service endpoints in general for a lot of reasons. Independent of the FQDN considerations, I strongly think that should factor in, if we can do it. A caching layer would benefit some read-only stuff, but I tend to imagine we'd want openstack stuff to dodge caching anyway since making that kind of API cache-friendly required quite a bit of tweaking and cache tuning the last time I did it elsewhere (and required me compiling in some varnish stuff to make auth work better through it). I generally have to imagine that OpenStack api caching won't look quite like MediaWiki api caching needs--but you never know. This all makes me think avoiding the prod caching layer might save us trouble at the outset.

May 23 2019, 5:14 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

acme-chief seems like a great idea to me if it is compatible with how we are sharing out the endpoints. I recall last I checked we were doing some wsgi workers or something vs running behind apache, if that's correct.

May 23 2019, 5:08 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair added a comment to T102576: Public log for Phabricator user account role change (e.g. deactivation, admin rights).

These logs should be fully visible to all users (public, if the instance supports that) and impossible to hide.

May 23 2019, 12:23 AM · Upstream, Phabricator (Upstream)

May 22 2019

Krenair added a comment to T223971: Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake).

I wonder if there should be two separate sets of flavours, one for each type of host. Probably wouldn't want an instance set up on one type migrated to the other. It sounds like right now if you see docs/examples that say a particular flavour should be used (perhaps on the basis of VCPUs), it's useless due to it actually coming down to the luck of what host you get scheduled on?

May 22 2019, 11:39 PM · cloud-services-team (Kanban), Continuous-Integration-Infrastructure
Krenair added a comment to T171188: Move the main WMCS puppetmaster into the Labs realm.

Looks like we regressed here while I was busy - logged onto the new puppetmasters to find puppet has been broken for weeks. Seems to be related to clientpackages changes

May 22 2019, 1:25 PM · cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Krenair added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

I just noticed the proxy endpoint, that obviously cannot be under wikimedia.org, but the rest should be there.
If you're wondering about SSL certs, all the wikimedia.org ones should be fine. Though interestingly right now those are all hard-coded http:// for some reason, we should probably fix that.

May 22 2019, 1:05 AM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair added a project to T224059: Should https://meta.wikimedia.org/wiki/Special:Contact/Stewards require a login: MediaWiki-extensions-ContactPage.

Its interesting that the legitimate party got the response - do we set reply-to to the user-provided email?
Even as a logged in user I appear to be able to go to that form and set a different email address. Maybe it should be made clear that the email provided is unverified.

May 22 2019, 12:50 AM · MediaWiki-extensions-ContactPage, Stewards-and-global-tools, Security
Krenair added a comment to T224000: wmcs-wikireplica-dns: error: circular reference detected.

HTTP 5xx from the Designate API... Can we get the full error message from the logs on the server?

May 22 2019, 12:40 AM · Patch-For-Review, cloud-services-team (Kanban)
Krenair added a comment to T214907: Request increased quota for Wikidocumentaries Cloud VPS project.

It's not possible to resize an existing VM.

May 22 2019, 12:12 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

May 21 2019

Krenair added a comment to T223835: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs.

Does the Foundation have an NDA with modular.im?

May 21 2019, 10:25 PM · serviceops, Patch-For-Review, Traffic, DNS, Operations, Wikimedia-Apache-configuration, Matrix
Krenair added projects to T223840: Can/should *.wmflabs.org be added to the default-src Content Security Policy?: Wikimedia-Site-requests, Security.
May 21 2019, 9:16 PM · Security, Wikimedia-Site-requests
Krenair added a comment to T223840: Can/should *.wmflabs.org be added to the default-src Content Security Policy?.

*.wmflabs.org should certainly not be added, not sure about tools.wmflabs.org.

May 21 2019, 9:15 PM · Security, Wikimedia-Site-requests
Krenair awarded T224057: Request increased quota for Automation Framework Cloud VPS project a Like token.
May 21 2019, 7:20 PM · Cloud-VPS (Quota-requests)
Krenair awarded T223902: cloudcontrol: decide on FQDN for service endpoints a Like token.
May 21 2019, 5:18 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair claimed T223734: nginx is failing to restart on cloudelastic100[1-2].wikimedia.org. Will also fail on cloudelastic100[3-4] when restart is attempted..
May 21 2019, 4:38 AM · Patch-For-Review, Operations, Discovery-Search (Current work), Traffic
Krenair added a comment to T223734: nginx is failing to restart on cloudelastic100[1-2].wikimedia.org. Will also fail on cloudelastic100[3-4] when restart is attempted..

@Mathew.onipe, how are things on cloudelastic now?

May 21 2019, 3:23 AM · Patch-For-Review, Operations, Discovery-Search (Current work), Traffic

May 20 2019

Krenair added a comment to T223920: Ensure/confirm a way to shell into unpuppetized VMs.

(it may be worth noting that while this is not directly necessary for the migration in the parent ticket, it is important to maintain the ability to bootstrap a realm that has no puppetmasters - either for a brand new realm or in the event of disaster wiping out all the existing puppetmasters. previously this was done by just making a new production host that the realm is allowed access to, with a move away from the model of puppetmasters for other realms sitting in production this becomes important)

May 20 2019, 10:45 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services, Puppet, Operations
Krenair added a comment to T223934: Add annotations from ops vendor maintenance calendar to Grafana.

it sounds like a nice idea, is the calendar publicly accessible?

May 20 2019, 9:03 PM · Operations
Krenair added a comment to T223902: cloudcontrol: decide on FQDN for service endpoints.

These endpoints should be .wikimedia.org.
Edit: Except proxy

May 20 2019, 2:26 PM · Traffic, Operations, Cloud-VPS, cloud-services-team (Kanban)
Krenair added a comment to T178520: Find somewhere else (not NFS) to store Quarry's resultsets.

Is the NFS server you're using being shared with other projects e.g. tools?
If you're simply treating the sqlite DB files as objects, downloading them to the local instance and then processing locally then you'll probably be okay for smaller datasets. If you're actually trying to run databases on top of NFS then things are not likely to be particularly fast.
Bare in mind Swift has a default 5GB object size limit. I don't think any existing Wikimedia deployment of it changes that.

May 20 2019, 12:23 AM · Quarry
Krenair added a comment to T223406: Remove reference to fields replaced by the actor table from WMCS views.

ahhh, that being in-progress might explain the difference observed. good catch

May 20 2019, 12:04 AM · Data-Services, Core Platform Team Backlog (Watching / External)

May 19 2019

Krenair added a comment to T223406: Remove reference to fields replaced by the actor table from WMCS views.

This seems to return quickly, on itwiki_p:

SELECT rev_page, COUNT(rev_id) AS edit_count
FROM revision_userindex
JOIN actor ON actor_id = rev_actor
WHERE actor_name IN ('Pietrodn', 'Frieda')
GROUP BY rev_page
HAVING COUNT(DISTINCT actor_id)=2
838 rows in set (6.09 sec)
May 19 2019, 9:59 PM · Data-Services, Core Platform Team Backlog (Watching / External)
Krenair added a comment to T223406: Remove reference to fields replaced by the actor table from WMCS views.

Use FROM revision_userindex instead of FROM revision.

May 19 2019, 8:20 PM · Data-Services, Core Platform Team Backlog (Watching / External)