Dzahn (Daniel Zahn)
Operations Engineer

Projects (18)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (167 w, 20 h)
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Unknown

Recent Activity

Today

Dzahn added a comment to T179565: Port exim statistics to Prometheus.

removed ganglia from mx1001/mx2001

Wed, Dec 13, 12:34 AM · Patch-For-Review, User-fgiunchedi, Goal, Operations

Yesterday

Dzahn added a comment to T182705: gerrit's ipv6 param is failing on labs.

We talked about this. The problem here is manifold. Some facts:

Tue, Dec 12, 7:22 PM · Patch-For-Review, Operations, Gerrit

Mon, Dec 11

Dzahn added a comment to T40860: security@mediawiki.org : Create a public key and publish it on the public key servers.

Adding Security-Team. What do you guys think about such a key nowadays? Ticket is from 2012 after all.

Mon, Dec 11, 4:54 PM · Security-Team, WorkType-NewFunctionality, Operations, Wikimedia-General-or-Unknown
Dzahn added a project to T40860: security@mediawiki.org : Create a public key and publish it on the public key servers: Security-Team.
Mon, Dec 11, 4:54 PM · Security-Team, WorkType-NewFunctionality, Operations, Wikimedia-General-or-Unknown
Dzahn added a comment to T169450: Redirect several wikis.

@Dzahn (not sure if you handle this stuff):

Mon, Dec 11, 4:51 PM · Patch-For-Review, Operations, Wikimedia-Apache-configuration, Puppet, Wiki-Setup (Close), Wikimedia-Language-setup

Fri, Dec 8

Dzahn updated subscribers of T99531: [Task] move wikiba.se webhosting to wikimedia misc-cluster.

@BBlack ^ So i think it's basically a request to add a new domain to the list of canonical domains. Any thoughts how we should proceed on this one?

Fri, Dec 8, 12:59 AM · Patch-For-Review, Traffic, wikiba.se, Operations, Wikidata-Sprint-2016-11-08, Wikidata

Thu, Dec 7

Dzahn added a comment to T119930: eventlog2001 - CRITICAL status of defined EventLogging jobs.

So it should be decom'ed?

Thu, Dec 7, 8:28 PM · Patch-For-Review, Analytics, Icinga, Analytics-EventLogging, Operations
Restricted Application added a project to T119930: eventlog2001 - CRITICAL status of defined EventLogging jobs: Analytics.

Is there a ticket to get eventlog2001 back into production? It is in site.pp but doesn't have any roles. Adding it with role spare::system for now.

Thu, Dec 7, 8:26 PM · Patch-For-Review, Analytics, Icinga, Analytics-EventLogging, Operations
Dzahn added a comment to T169246: Stress/capacity test new ores* cluster.

Is the stress test over? Then T165170 is probably unstalled now. Is it not over yet? Then maybe this ticket should be reopened?

Thu, Dec 7, 7:41 PM · User-Ladsgroup, Operations, User-Joe, Scoring-platform-team (Current), Patch-For-Review, ORES
Dzahn added a comment to T165170: rack/setup/install ores2001-2009.

Is this unstalled now? The reason was while T168246 is ongoing but that ticket is resolved. Is it really resolved though?

Thu, Dec 7, 7:40 PM · Patch-For-Review, Scoring-platform-team, ORES, Operations
Dzahn added a comment to T114801: operations-apache-config-lint replacement doesn't check syntax.

This is correct, i believe it's still a valid ToDo for Contint in general.

Thu, Dec 7, 6:35 PM · Wikimedia-Incident, Incident-20160126-WikimediaDomainRedirection, Operations, Continuous-Integration-Config, Regression
Dzahn created T182342: stat1004 - /mnt/hdfs is not accessible.
Thu, Dec 7, 6:32 PM · Analytics-Cluster, Operations, Analytics
Dzahn closed T179317: Varnish and Apache debug tools and logs for hoo as Resolved.

Let us know if something isn't working.

Thu, Dec 7, 1:43 AM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
Dzahn added a comment to T179317: Varnish and Apache debug tools and logs for hoo.

Alright, this should be resolved now. As requested and approved, using 3 new groups as such:

Thu, Dec 7, 1:41 AM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
Dzahn added a comment to T177843: Set up octocatalog-diff on host with access to puppetmasters and puppetdb.

Hi, is puppetcompiler1001 going to stay in site.pp permanently or was it a temporary thing?

Thu, Dec 7, 12:09 AM · Patch-For-Review, Puppet, User-Joe, Operations

Wed, Dec 6

Dzahn added a comment to T175150: Decommission stat1003.eqiad.wmnet.

I noticed this host was up and running but not in site.pp / no roles as part of decoming Ganglia from everything (T177225). It's gone from Icinga but running. I am adding it back to site.pp with role(spare) per decom workflow because this prevents a couple issues, like Ganglia will be removed and the host will keep getting security upgrades.

Wed, Dec 6, 10:34 PM · Patch-For-Review, ops-eqiad, Operations
Dzahn added a comment to T179974: Add Annual Report 2017 to git repo.

There was a meeting about this today, involving the actual developer and security. He confirmed he analyzed our pages of the previous years and it will be just like that, only static HTML/CSS/JS, no tracking, no loading from external sites etc. We don't foresee any issues here. It will be released some time in January.

Wed, Dec 6, 5:38 PM · Annual-Report
Dzahn triaged T182215: install_server: switch to stretch as default install image as Normal priority.
Wed, Dec 6, 5:34 PM · Operations
Dzahn updated subscribers of T182215: install_server: switch to stretch as default install image.
Wed, Dec 6, 5:32 PM · Operations
Dzahn created T182215: install_server: switch to stretch as default install image.
Wed, Dec 6, 5:32 PM · Operations
Dzahn added a comment to T182159: dispatchChanges.php still being run from the Wikidata build.

applied on terbium

Wed, Dec 6, 3:11 PM · Puppet, Operations, User-Addshore, Patch-For-Review, Wikidata
Dzahn added a comment to T182159: dispatchChanges.php still being run from the Wikidata build.

cronjobs have been updated on terbium and confirmed there are no remnants, just the 3 updated commands. no crons on wasat

Wed, Dec 6, 12:10 AM · Puppet, Operations, User-Addshore, Patch-For-Review, Wikidata

Tue, Dec 5

Dzahn created P6431 (An Untitled Masterwork).
Tue, Dec 5, 10:05 PM
Dzahn added a comment to T180657: Purchase domains mediawi.ki and media.wiki to use as a url shortener.

This is blocked technically on T133548 (and partially on the other ones linked in my last comment).

Tue, Dec 5, 5:19 PM · Operations, Traffic, Domains
Dzahn changed the status of T180657: Purchase domains mediawi.ki and media.wiki to use as a url shortener from Open to Stalled.
Tue, Dec 5, 5:14 PM · Operations, Traffic, Domains
Liuxinyu970226 awarded T99531: [Task] move wikiba.se webhosting to wikimedia misc-cluster a Orange Medal token.
Tue, Dec 5, 10:43 AM · Patch-For-Review, Traffic, wikiba.se, Operations, Wikidata-Sprint-2016-11-08, Wikidata

Mon, Dec 4

Dzahn renamed T179317: Varnish and Apache debug tools and logs for hoo from Varnish and Apache root for hoo to Varnish and Apache debug tools and logs for hoo.
Mon, Dec 4, 7:16 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
Dzahn added a comment to T181479: Requesting access to terbium/wasat for Trey Jones.

P.S. Like all shell users, this also gives you a home dir on https://wikitech.wikimedia.org/wiki/People.wikimedia.org so you can use https://people.wikimedia.org/~tjones/ if you like.

Mon, Dec 4, 7:13 PM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn added a comment to T181626: Requesting access to terbium.eqiad.wmnet for cparle.

P.S. Like all shell users, this also gives you a home dir on https://wikitech.wikimedia.org/wiki/People.wikimedia.org so you can use https://people.wikimedia.org/~cparle/ if you like.

Mon, Dec 4, 7:12 PM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn closed T181626: Requesting access to terbium.eqiad.wmnet for cparle as Resolved.

your request has been approved and the code change has been merged.

Mon, Dec 4, 7:09 PM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn claimed T181626: Requesting access to terbium.eqiad.wmnet for cparle.
Mon, Dec 4, 6:33 PM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn closed T181540: Enable wdqs-admin's to control nginx as Resolved.

This should be resolved now (https://gerrit.wikimedia.org/r/#/c/393814/4/modules/admin/data/data.yaml)

Mon, Dec 4, 6:11 PM · Patch-For-Review, Discovery, Wikidata, Operations, Ops-Access-Requests, Wikidata-Query-Service
Dzahn closed T181479: Requesting access to terbium/wasat for Trey Jones as Resolved.

your request has been approved and the Gerrit change is merged.

Mon, Dec 4, 6:10 PM · Patch-For-Review, Ops-Access-Requests, Operations
Noella94 awarded T181587: add Outreachy round 15 feeds to Planet a Like token.
Mon, Dec 4, 9:15 AM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet

Sat, Dec 2

Dzahn added a comment to T177225: Uninstall ganglia from the fleet.

db2023 - https://gerrit.wikimedia.org/r/394647
etcd::networking https://gerrit.wikimedia.org/r/394724
db2028,db2029 - https://gerrit.wikimedia.org/r/394725
spare systems - https://gerrit.wikimedia.org/r/394733
test servers - https://gerrit.wikimedia.org/r/394734

Sat, Dec 2, 12:57 AM · Patch-For-Review, Operations, monitoring

Fri, Dec 1

Dzahn added a comment to T177225: Uninstall ganglia from the fleet.

Running the exact same "/usr/bin/apt-get -y -q remove --purge ganglia-monitor" manually on a trusty host (db2029)... works! But when puppet ran it, it failed because there was still a running gmond process...

Fri, Dec 1, 9:58 PM · Patch-For-Review, Operations, monitoring
Dzahn added a comment to T177225: Uninstall ganglia from the fleet.

@jcrespo Did it happen on all hosts or just a few? Do you recall which host the paste above is from? I am surprised since this same patch got applied on a ton of other roles without this issue over the last few days and i don't see an include of ganglia in the mariadb module yet. The difference here that i see so far is just that i apply it via regex and not via individual role classes. Let me try it with just a single host, then based on a role class.

Fri, Dec 1, 4:13 PM · Patch-For-Review, Operations, monitoring

Thu, Nov 30

Dzahn updated subscribers of T181587: add Outreachy round 15 feeds to Planet.

@Nehajha @Meghasharma213 Please see above, i just realized you probably have Phabricator accounts as well and you do. Your blog feeds have been added to Planet.

Thu, Nov 30, 10:07 PM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet
Dzahn added a comment to T180655: Phabricator and Gerrit: Improve the way that maintenance downtime is communicated to users..

Yea, we should have both, it was meant to be additional not replacing the other. But IRC as well because that's where people ask.

Thu, Nov 30, 10:02 PM · Traffic, periodic-update, Gerrit, Phabricator, Operations
Dzahn closed T181587: add Outreachy round 15 feeds to Planet as Resolved.
Thu, Nov 30, 10:01 PM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet
Dzahn added a comment to T181518: kafka1018 fails to boot.

I saw this host as DOWN when looking at Icinga as it was in the unacknowledeged section (though notifications were disabled).

Thu, Nov 30, 9:32 PM · User-Elukey, Patch-For-Review, Analytics-Kanban, Operations, ops-eqiad
Dzahn added a comment to T180998: Switch on http/2 in phabricator apache.

As Moritz said on a Gerrit patch to enable this for Gerrit/Planet. There are security concerns even with the version in stretch and it probably becomes realistic to do in buster.

Thu, Nov 30, 5:11 PM · Traffic, Operations, Phabricator
Dzahn added a comment to T180655: Phabricator and Gerrit: Improve the way that maintenance downtime is communicated to users..

I was thinking about "jouncebot: next" when reading this. I mean it already outputs the deployments at the specific times by itself and reads the info the the deployment calendar, wouldn't it work just the same way to add scheduled downtimes to the calendar on wiki and then have the bot announce it to people on a couple channels like an hour and 10 minutes before?

Thu, Nov 30, 1:33 AM · Traffic, periodic-update, Gerrit, Phabricator, Operations
Dzahn added a project to T181630: Send celery and wsgi service logs to logstash: Wikimedia-Logstash.
Thu, Nov 30, 12:39 AM · Wikimedia-Logstash, monitoring, Wikimedia-Incident, Operations, Scoring-platform-team
Dzahn updated subscribers of T178008: ensure that services on labtest machines never create SMS from Icinga (not send sms pages for labtest* things to non-cloud folks).

Ok, so what we have meanwhile is this in Hiera, which all applies to any host starting with labtest*.

Thu, Nov 30, 12:37 AM · Patch-For-Review, monitoring, Operations

Wed, Nov 29

Dzahn added a comment to T178849: Click on fullImageLink <a> for PDF on File: page no longer rendering in browser.

starting a screen and running the command is very simple.

Wed, Nov 29, 9:35 PM · MediaWiki-File-management, Commons, MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), Patch-For-Review, Regression, media-storage, Multimedia-Team-Working-Board, Multimedia
Dzahn added a comment to T181626: Requesting access to terbium.eqiad.wmnet for cparle.

@Gilles @Cparle Or would it make sense to just add the maintenance command to a cronjob and let it run automatically at fixed intervals? Does it really need to be manual? I'd be happy to help getting another cron on maintenance servers into puppet.

Wed, Nov 29, 8:47 PM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn added a comment to T178849: Click on fullImageLink <a> for PDF on File: page no longer rendering in browser.

Would it make sense to run this script automatically via a puppetized cronjob? Or does it need manual interaction? I mean it would always be good to reduce the need for manual things and just automate it and i'd be happy to help getting another cron on maintenance servers.

Wed, Nov 29, 8:45 PM · MediaWiki-File-management, Commons, MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), Patch-For-Review, Regression, media-storage, Multimedia-Team-Working-Board, Multimedia
Dzahn added a comment to T181626: Requesting access to terbium.eqiad.wmnet for cparle.

Confirmed that L3 has already been signed by cparle. A user exists in admin module but in the "ldap_only_users" section. Adding real shell access means having to move that account, it should not be duplicate.

Wed, Nov 29, 8:43 PM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn claimed T179317: Varnish and Apache debug tools and logs for hoo.
Wed, Nov 29, 7:36 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
Dzahn added a comment to T179317: Varnish and Apache debug tools and logs for hoo.
  • strace and tcpdump

'ALL = NOPASSWD: /usr/bin/strace *',
'ALL = NOPASSWD: /usr/sbin/tcpdump *',
(https://gerrit.wikimedia.org/r/#/c/394102/2/modules/admin/data/data.yaml)

Wed, Nov 29, 7:34 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
Dzahn added a comment to T179317: Varnish and Apache debug tools and logs for hoo.
on the mw canaries

role::mediawiki::canary_appserver (https://gerrit.wikimedia.org/r/#/c/394102/)

Wed, Nov 29, 7:14 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
Dzahn added a comment to T179317: Varnish and Apache debug tools and logs for hoo.
  • varnish log access on the varnish hosts
Wed, Nov 29, 7:09 PM · Patch-For-Review, Performance-Team (Radar), Operations, Ops-Access-Requests
TJones awarded T181479: Requesting access to terbium/wasat for Trey Jones a Doubloon token.
Wed, Nov 29, 6:33 PM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn changed the status of T181479: Requesting access to terbium/wasat for Trey Jones from Open to Stalled.

This ticket is now just pending approval in the weekly ops meeting which is next Monday.

Wed, Nov 29, 6:24 PM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn changed the status of T181540: Enable wdqs-admin's to control nginx from Open to Stalled.
Wed, Nov 29, 6:20 PM · Patch-For-Review, Discovery, Wikidata, Operations, Ops-Access-Requests, Wikidata-Query-Service
Dzahn added a comment to T181540: Enable wdqs-admin's to control nginx.

+1 and patch looks good to me now (I encouraged using systemctl instead/besides the service command).

Wed, Nov 29, 6:19 PM · Patch-For-Review, Discovery, Wikidata, Operations, Ops-Access-Requests, Wikidata-Query-Service
srishakatux awarded T181587: add Outreachy round 15 feeds to Planet a Love token.
Wed, Nov 29, 5:11 AM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet
Dzahn added a comment to T181479: Requesting access to terbium/wasat for Trey Jones.

@TJones please ignore my comments, i just realized you already have previous shell accesss (mw-log-readers, analytics-privatedata-uers). So all i said about SSH key and L3 doesn't apply anymore :) This will be much quicker.

Wed, Nov 29, 4:13 AM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn added a comment to T181479: Requesting access to terbium/wasat for Trey Jones.

We can probably use the existing group called "restricted".

Wed, Nov 29, 4:10 AM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn added a comment to T181479: Requesting access to terbium/wasat for Trey Jones.

Hi @TJones I will handle this access request. We might have to create a new admin group for this type of access, i will look at that. Meanwhile you could start by creating an ssh key and attaching it to this ticket. Also, please take a look at https://phabricator.wikimedia.org/L3 and sign it. Best, Daniel

Wed, Nov 29, 3:58 AM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn added a comment to T181587: add Outreachy round 15 feeds to Planet.

https://www.mediawiki.org/wiki/Talk:Outreachy/Round_15

Wed, Nov 29, 3:51 AM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet
Dzahn claimed T181479: Requesting access to terbium/wasat for Trey Jones.
Wed, Nov 29, 3:50 AM · Patch-For-Review, Ops-Access-Requests, Operations
Dzahn claimed T181540: Enable wdqs-admin's to control nginx.
Wed, Nov 29, 3:50 AM · Patch-For-Review, Discovery, Wikidata, Operations, Ops-Access-Requests, Wikidata-Query-Service
Dzahn added a comment to T181587: add Outreachy round 15 feeds to Planet.

Can you find more working ones than i did? Does "medium" have RSS feeds? Please add more?

Wed, Nov 29, 3:40 AM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet
Dzahn edited projects for T181587: add Outreachy round 15 feeds to Planet, added: Outreachy (Round-15); removed Outreachy.
Wed, Nov 29, 2:57 AM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet
Dzahn added a project to T181587: add Outreachy round 15 feeds to Planet: Outreachy.
Wed, Nov 29, 2:56 AM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet
Dzahn created T181587: add Outreachy round 15 feeds to Planet.
Wed, Nov 29, 1:46 AM · Patch-For-Review, Outreachy (Round-15), Wikimedia-Planet
Dzahn added a comment to T181397: Lost network connectivity on mw1276.

When there are hardware issues, let's close the ticket only after servers actually get repooled. Because otherwise we keep forgetting that. Noticed it from Icinga saying: "Host mw1276 is not in mediawiki-installation dsh group"

Wed, Nov 29, 1:13 AM · ops-eqiad, Operations

Tue, Nov 28

Krinkle awarded T177225: Uninstall ganglia from the fleet a Orange Medal token.
Tue, Nov 28, 4:32 AM · Patch-For-Review, Operations, monitoring
Dzahn added a comment to T181446: Access to logstash (LDAP group 'nda') for Paladox.

Confirmed. This was about Gerrit logs. If there was a way to request "logstash but just Gerrit" then that would have been the request. Paladox is the one who did the majority of the work to move logs into logstash in the first place (to enable volunteers to read them without shell access) and works with Gerrit upstream so i also encouraged him. That said, it doesn't change that the "nda" group does a lot more than that and the permission system is not fine-tuned enough to allow it "per service".

Tue, Nov 28, 4:10 AM · Gerrit, Operations, Ops-Access-Requests

Mon, Nov 27

Dzahn renamed T181446: Access to logstash (LDAP group 'nda') for Paladox from Requesting access to the ldap nda group to Access to logstash (LDAP group 'nda') for Paladox.
Mon, Nov 27, 10:38 PM · Gerrit, Operations, Ops-Access-Requests
Dzahn added a project to T181446: Access to logstash (LDAP group 'nda') for Paladox: Gerrit.
Mon, Nov 27, 10:33 PM · Gerrit, Operations, Ops-Access-Requests
Dzahn added a comment to T181446: Access to logstash (LDAP group 'nda') for Paladox.

Hi @Paladox I know it since we already talked on IRC, but you should add that you already mailed legal and your question if the volunteer NDA is right and all that.

Mon, Nov 27, 10:32 PM · Gerrit, Operations, Ops-Access-Requests
Dzahn added a comment to T174342: Missing IP addresses for Maroc Telecom .

Sounds like it would be useful if Partnerships could be added in the Phabricator-based workflow for future updates.

Mon, Nov 27, 9:26 PM · Operations, Traffic, Phabricator, Zero
Dzahn added a comment to T127771: Make labs wikitech role aware.

I'm not entirely sure if role() is meant to work the same way going forward with the new role/profile structure that is being promoted in ops/puppet.git.

Mon, Nov 27, 6:15 PM · Cloud-VPS, Cloud-Services
Dzahn added a comment to T178454: Icinga disk space alert when a Docker container is running on an host.

Is https://gerrit.wikimedia.org/r/#/c/393215/ also this ticket? It links to T177052 which seems somewhat related but not about Icinga, just Grafana itself.

Mon, Nov 27, 4:44 PM · Patch-For-Review, Release Pipeline, Operations, monitoring, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)

Thu, Nov 23

Dzahn closed T181202: Enable http/2 for planet apache as Declined.

008b62f4048 should probably be reverted, its effects manually reverted (a2dismod http2) and this task be declined

Thu, Nov 23, 5:53 PM · Patch-For-Review, Wikimedia-Planet, Operations
Dzahn closed T181202: Enable http/2 for planet apache, a subtask of T168490: upgrade planet instances to stretch, as Declined.
Thu, Nov 23, 5:53 PM · Wikimedia-Planet, Patch-For-Review, Operations
Dzahn added a comment to T181204: delete project "gerrit-staging".

Sorry, my bad again. That was supposed to be in the other project too. Instance deleted now and recreated where it should have been. You can go ahead.

Thu, Nov 23, 7:46 AM · Cloud-VPS
Dzahn added a comment to T177196: Port non-deprecated Diamond collectors to Prometheus.

Is there an issue with the memcached exporter since this looks empty:

Thu, Nov 23, 12:21 AM · cloud-services-team (Kanban), Patch-For-Review, User-fgiunchedi, Goal, Operations
Dzahn added a comment to T162070: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases.

Created subtask to make quarry use the mariadb module since that is one of the few things still using it.

Thu, Nov 23, 12:14 AM · Patch-For-Review, Operations, DBA
Dzahn added a subtask for T162070: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases: T181205: let quarry use the mariadb module.
Thu, Nov 23, 12:13 AM · Patch-For-Review, Operations, DBA
Dzahn added a parent task for T181205: let quarry use the mariadb module: T162070: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases.
Thu, Nov 23, 12:13 AM · cloud-services-team (Kanban), Operations, Quarry
Dzahn created T181205: let quarry use the mariadb module.
Thu, Nov 23, 12:13 AM · cloud-services-team (Kanban), Operations, Quarry
Dzahn assigned T181204: delete project "gerrit-staging" to Andrew.
Thu, Nov 23, 12:08 AM · Cloud-VPS
Dzahn created T181204: delete project "gerrit-staging".
Thu, Nov 23, 12:07 AM · Cloud-VPS

Wed, Nov 22

Dzahn added a comment to T181202: Enable http/2 for planet apache.

Yeah! planet-hotdog.wmflabs.org supports HTTP/2.0.

Wed, Nov 22, 11:26 PM · Patch-For-Review, Wikimedia-Planet, Operations
Dzahn updated the task description for T181202: Enable http/2 for planet apache.
Wed, Nov 22, 11:18 PM · Patch-For-Review, Wikimedia-Planet, Operations
Dzahn updated the task description for T181202: Enable http/2 for planet apache.
Wed, Nov 22, 11:17 PM · Patch-For-Review, Wikimedia-Planet, Operations
Dzahn added a comment to T177225: Uninstall ganglia from the fleet.

Since https://gerrit.wikimedia.org/r/#/c/391241/ was merged (thanks Filippo!) i was able to re-revert the "remove HHVM ganglia from appservers" change, so files (https://gerrit.wikimedia.org/r/#/c/392764/) have been removed from all appservers

Wed, Nov 22, 11:16 PM · Patch-For-Review, Operations, monitoring
Dzahn added a comment to P6370 Draft email to legal.

Hi WMF Legal Team,

Wed, Nov 22, 10:31 PM
Dzahn awarded T141324: Look into shoving gerrit logs into logstash a Y So Serious token.
Wed, Nov 22, 10:23 PM · Release-Engineering-Team (Someday), Patch-For-Review, Technical-Debt, Wikimedia-Logstash, Gerrit
Dzahn reassigned T141324: Look into shoving gerrit logs into logstash from Dzahn to Paladox.
Wed, Nov 22, 10:23 PM · Release-Engineering-Team (Someday), Patch-For-Review, Technical-Debt, Wikimedia-Logstash, Gerrit
Dzahn closed T141324: Look into shoving gerrit logs into logstash as Resolved.
Wed, Nov 22, 10:23 PM · Release-Engineering-Team (Someday), Patch-For-Review, Technical-Debt, Wikimedia-Logstash, Gerrit
Dzahn closed T141324: Look into shoving gerrit logs into logstash, a subtask of T63779: Add system logs to logstash (tracking), as Resolved.
Wed, Nov 22, 10:22 PM · Tracking, Wikimedia-Logstash
Dzahn added a comment to T141324: Look into shoving gerrit logs into logstash.

https://logstash.wikimedia.org/ now shows the first log lines from cobalt :))

Wed, Nov 22, 10:22 PM · Release-Engineering-Team (Someday), Patch-For-Review, Technical-Debt, Wikimedia-Logstash, Gerrit
Dzahn added a comment to T180978: Switch on http/2 in apache for gerrit.

I'm proposing we lower the priority on this and let another service (preferably one with less depending on it) try http2 first :)

Wed, Nov 22, 8:21 PM · Traffic, Patch-For-Review, Operations, Gerrit
Dzahn added a comment to T158196: Reimage labstore1001 and labstore1002 for DRBD storage setup.

@chasemp Should we reinstall these with stretch? I noticed them in site.pp with a comment leading to this ticket.

Wed, Nov 22, 3:50 AM · Data-Services, cloud-services-team (Kanban), Patch-For-Review, Operations
Dzahn added a comment to T181114: Acquire wikidirectory.org.

I suggest contacting the Legal team directly about this.

Wed, Nov 22, 3:15 AM · Operations, Traffic, Domains