Dzahn (Daniel Zahn)
Operations Engineer

Projects (16)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (185 w, 2 d)
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Unknown

Recent Activity

Today

Dzahn added a comment to T150672: Provide an archive endpoint for older Parsoid debs (on releases.wikimedia.org or elsewhere).

@ssastry I think this should have resolved the ticket. See my 2 comments above. Wanna try to upload a test file to https://releases.wikimedia.org/parsoid/ ?

Thu, Apr 19, 5:07 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Parsoid, Operations
Dzahn added a comment to T150672: Provide an archive endpoint for older Parsoid debs (on releases.wikimedia.org or elsewhere).

In Hiera it is defined which is the currently "active" releases server, like so (hieradata/common.yaml):

Thu, Apr 19, 5:05 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Parsoid, Operations
Dzahn added a comment to T150672: Provide an archive endpoint for older Parsoid debs (on releases.wikimedia.org or elsewhere).
  • https://releases.wikimedia.org/parsoid/ has been created
  • it's hosted on releases1001/releases2001 and active/active
  • the local path is /srv/org/wikimedia/releases/parsoid/
  • a new admin group "releasers-parsoid" has been created and added on the hosts above
  • @ssastry has been added to that group
  • permissions were fixed on the new directory to allow the group writing to it
Thu, Apr 19, 2:26 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Parsoid, Operations

Yesterday

Dzahn triaged T192472: Requesting access to analytics servers for mepps as Normal priority.
Wed, Apr 18, 8:37 PM · Ops-Access-Requests, Operations
Dzahn updated subscribers of T192472: Requesting access to analytics servers for mepps.

Hi Margaret, could you explain a little bit why you need that specific group and what you are planning to do. cc: @Nuria

Wed, Apr 18, 8:37 PM · Ops-Access-Requests, Operations
Dzahn added a comment to T189524: Add Reading Infrastructure engineers to contacts for mobileapps.

Tried to add missing contactgroup to mobileapps services with the change above (copying wdqs setup) but it seems that wasn't enough to actually make it happen yet. (ran puppet on einsteinium and scb1001)

Wed, Apr 18, 8:34 PM · Patch-For-Review, Services (watching), monitoring, Operations
Dzahn added a comment to P7010 Duplicate Icinga IRC logs .

https://icinga.wikimedia.org/cgi-bin/icinga/history.cgi?host=scb1003&service=mobileapps+endpoints+health

Wed, Apr 18, 7:47 PM
Dzahn added a parent task for T190415: Requesting access to stats machines for Lucas Werkmeister: T192452: Find out how many external uses the wbcheckconstraints API action has.
Wed, Apr 18, 6:57 PM · Operations, Ops-Access-Requests
Dzahn added a subtask for T192452: Find out how many external uses the wbcheckconstraints API action has: T190415: Requesting access to stats machines for Lucas Werkmeister.
Wed, Apr 18, 6:57 PM · Wikibase-Quality-Constraints, Wikibase-Quality, Wikidata
Dzahn added a comment to T190415: Requesting access to stats machines for Lucas Werkmeister.

@Lucas_Werkmeister_WMDE Thank you for the update! I will say the status "stalled" is still correct in that case until we now. But that was very helpful to know the background. I'll just keep this open and make it a subtask.

Wed, Apr 18, 6:56 PM · Operations, Ops-Access-Requests
Dzahn closed T191478: Requesting access to shell (snapshot, dumpsdata) for springle as Resolved.
Wed, Apr 18, 6:54 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn added a comment to T191478: Requesting access to shell (snapshot, dumpsdata) for springle.

For SSH config also see https://wikitech.wikimedia.org/wiki/Production_shell_access#Standard_config

Wed, Apr 18, 6:54 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn added a comment to T191478: Requesting access to shell (snapshot, dumpsdata) for springle.

on bast1002: Notice: /Stage[main]/Admin/Admin::Hashuser[springle]/Admin::User[springle]/User[springle]/ensure: created

Wed, Apr 18, 6:53 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn closed T192115: sudoer access for pnorman on maps servers as Resolved.

The request has been approved in ops meeting and the change above has been merged.

Wed, Apr 18, 5:07 PM · Patch-For-Review, Operations, Ops-Access-Requests, Maps-Sprint
Dzahn triaged T190717: Update wikidiff2 library on the WMF production cluster as Normal priority.
Wed, Apr 18, 12:44 AM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
Dzahn closed T191029: Scap stalled at sync-masters, ok: 1, left: 1 as Resolved.
Wed, Apr 18, 12:42 AM · Patch-For-Review, Release-Engineering-Team, Operations, Scap
Dzahn triaged T191625: Create RC feed for login.wikimedia as Low priority.
Wed, Apr 18, 12:41 AM · Wikimedia-IRC-RC-Server, Stewards-and-global-tools, Operations
Dzahn added a comment to T191625: Create RC feed for login.wikimedia.

Afaict the way it works is that the bot has the right to create channels and it joins a channel when the first message comes in.
(We know because after restarts channels didn't get created until the first actual message from the bot).

Wed, Apr 18, 12:40 AM · Wikimedia-IRC-RC-Server, Stewards-and-global-tools, Operations
Dzahn placed T192185: request to assign WMF3565 as terbium equivalent up for grabs.

I think it kind of blocks the "never use PHP5" / "switch to PHP7" thing which also affects appservers and deployment servers. In the last Service Operations meeting it was brought up that we need to replace terbium as part of that. But let me confirm the urgency with @Joe

Wed, Apr 18, 12:38 AM · hardware-requests, Operations
Dzahn added a comment to T191625: Create RC feed for login.wikimedia.
Wed, Apr 18, 12:26 AM · Wikimedia-IRC-RC-Server, Stewards-and-global-tools, Operations
Dzahn added a comment to T191842: Deployment git server can't supply ORES hosts in parallel.

Which deployment server is this about? Production or deployment-prep or another?

Wed, Apr 18, 12:17 AM · Scap, Operations, Scoring-platform-team, Release-Engineering-Team

Tue, Apr 17

Dzahn triaged T192185: request to assign WMF3565 as terbium equivalent as High priority.
Tue, Apr 17, 11:54 PM · hardware-requests, Operations
Dzahn changed the status of T192092: setup replacement for terbium (maintenance_server) on stretch from Open to Stalled.
Tue, Apr 17, 11:54 PM · Patch-For-Review, Operations
Dzahn triaged T191767: Important critical Etherpad release – 1.6.4 as High priority.
Tue, Apr 17, 11:22 PM · Wikimedia-Etherpad, Operations, Security
Dzahn triaged T190085: Reclaim/Decommission Silver.wikimedia.org as Normal priority.
Tue, Apr 17, 10:34 PM · hardware-requests, Operations
Dzahn triaged T192115: sudoer access for pnorman on maps servers as Normal priority.
Tue, Apr 17, 10:34 PM · Patch-For-Review, Operations, Ops-Access-Requests, Maps-Sprint
Dzahn updated subscribers of T192115: sudoer access for pnorman on maps servers.
Tue, Apr 17, 10:33 PM · Patch-For-Review, Operations, Ops-Access-Requests, Maps-Sprint
Dzahn added a comment to T192115: sudoer access for pnorman on maps servers.

@Gehel Ok, so after looking at it again, i have a new suggestion. Adding permissions to the existing maps-admin group. There are just 5 people in that and at least 2 would be the same people. Adding yet another level of admin group between "admin" and "root" seems going to far.

Tue, Apr 17, 10:31 PM · Patch-For-Review, Operations, Ops-Access-Requests, Maps-Sprint
Dzahn updated subscribers of T192256: Add Tim_WMDE to the ldap/wmde group.

@RStallman-legalteam Hi, we got another NDA request for WMDE here. Thanks!

Tue, Apr 17, 8:03 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn triaged T192256: Add Tim_WMDE to the ldap/wmde group as Normal priority.
Tue, Apr 17, 8:02 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn closed T192159: Requesting access to deployment for pmiazga as Resolved.

@pmiazga Now you should have the +2 and things should just work. I'm calling this ticket resolved. Let us know if any unexpected issues.

Tue, Apr 17, 8:02 PM · Release-Engineering-Team, Gerrit, Operations, Ops-Access-Requests
Dzahn changed the status of T188561: SSL cert for links.email.wikimedia.org from Open to Stalled.
Tue, Apr 17, 5:49 PM · Operations, Traffic, fundraising-tech-ops, Fundraising-Backlog
Dzahn changed the status of T188453: Google Search Console access for Search Platform team from Open to Stalled.
Tue, Apr 17, 5:40 PM · Discovery-Search (Current work), Ops-Access-Requests, Operations
Dzahn changed the status of T191478: Requesting access to shell (snapshot, dumpsdata) for springle from Open to Stalled.
Tue, Apr 17, 5:33 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn added a comment to T190415: Requesting access to stats machines for Lucas Werkmeister.

Setting to Stalled, unless that meeting has already happened. In that case, please let us know the status here on ticket.

Tue, Apr 17, 5:28 PM · Operations, Ops-Access-Requests
Dzahn changed the status of T190415: Requesting access to stats machines for Lucas Werkmeister from Open to Stalled.
Tue, Apr 17, 5:28 PM · Operations, Ops-Access-Requests
Dzahn closed T192060: Request to add Tarrow to the ldap/wmde group as Resolved.

@Tarrow All done. Let us know if any unexpected issues.

Tue, Apr 17, 5:27 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn updated the task description for T192060: Request to add Tarrow to the ldap/wmde group .
Tue, Apr 17, 5:27 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn added a comment to T192060: Request to add Tarrow to the ldap/wmde group .

"tarrow" has been added to the "wmde" LDAP group.

Tue, Apr 17, 5:27 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn triaged T192060: Request to add Tarrow to the ldap/wmde group as Normal priority.
Tue, Apr 17, 5:25 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn claimed T192060: Request to add Tarrow to the ldap/wmde group .
Tue, Apr 17, 5:25 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn changed the status of T191523: Request to add Matthias Geisler to the ldap/wmde group from Open to Stalled.
Tue, Apr 17, 5:24 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn added a comment to T192317: wikistats.wmflabs.org: 502 Bad Gateway.

It was broken because - the instance was shut down. I don't know what or who shut it down, but it was simply not running.

Tue, Apr 17, 3:03 PM · VPS-project-Wikistats

Sat, Apr 14

Dzahn added a comment to T185504: Netbox: add Icinga check for PostgreSQL.

regarding a health check on the master itself: Which type of check exactly would that be on the master? So far we just have the replication check on slaves and there are all these aliases for check_postgres that we could use:

Sat, Apr 14, 1:20 AM · Patch-For-Review, monitoring, Operations
Dzahn added a comment to T185504: Netbox: add Icinga check for PostgreSQL.

`ERROR: FATAL: no pg_hba.conf entry for host "2620:0:860:4:208:80:153:110", user "replication", database "template1", SSL on
FATAL: no pg_hba.conf entry for host "2620:0:860:4:208:80:153:110", user "replication", database "template1", SSL off`

Sat, Apr 14, 12:53 AM · Patch-For-Review, monitoring, Operations
Dzahn added a comment to T185504: Netbox: add Icinga check for PostgreSQL.
Sat, Apr 14, 12:52 AM · Patch-For-Review, monitoring, Operations
Dzahn added a comment to T185504: Netbox: add Icinga check for PostgreSQL.

The package providing /usr/bin/check_postgres_hot_standby_delay is check-postgres

Sat, Apr 14, 12:19 AM · Patch-For-Review, monitoring, Operations
Dzahn renamed T185504: Netbox: add Icinga check for PostgreSQL from Netbox: add Icinga check for PosgreSQL to Netbox: add Icinga check for PostgreSQL.
Sat, Apr 14, 12:13 AM · Patch-For-Review, monitoring, Operations

Fri, Apr 13

Dzahn updated subscribers of T191478: Requesting access to shell (snapshot, dumpsdata) for springle.

@faidon @mark Since Monday is a US holiday so i guess no ops meeting but i will be on duty next week: If you want to approve this for Sean, i can go ahead and resolve it.

Fri, Apr 13, 11:59 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn awarded T191478: Requesting access to shell (snapshot, dumpsdata) for springle a Like token.
Fri, Apr 13, 11:57 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn added a comment to T192115: sudoer access for pnorman on maps servers.

How about a new group that lets you run any command as the users postgres and osmupdater (as requested) but not any command as any user. Would that sound ok?

Fri, Apr 13, 11:56 PM · Patch-For-Review, Operations, Ops-Access-Requests, Maps-Sprint
Dzahn added a comment to T192092: setup replacement for terbium (maintenance_server) on stretch.

The name will be *nihonium* (element 113), not nunki as in the first version of the change above. This is still eqiad, not codfw.

Fri, Apr 13, 11:53 PM · Patch-For-Review, Operations
Dzahn added a comment to T192185: request to assign WMF3565 as terbium equivalent.

The name would be nihonium, element 113.

Fri, Apr 13, 11:52 PM · hardware-requests, Operations
Dzahn updated subscribers of T192185: request to assign WMF3565 as terbium equivalent.
Fri, Apr 13, 11:42 PM · hardware-requests, Operations
Dzahn updated the task description for T192185: request to assign WMF3565 as terbium equivalent.
Fri, Apr 13, 11:41 PM · hardware-requests, Operations
Dzahn created T192185: request to assign WMF3565 as terbium equivalent.
Fri, Apr 13, 11:40 PM · hardware-requests, Operations
Dzahn updated subscribers of T192159: Requesting access to deployment for pmiazga.

I _think_ it's that you are missing here:

Fri, Apr 13, 5:29 PM · Release-Engineering-Team, Gerrit, Operations, Ops-Access-Requests
Dzahn added projects to T192159: Requesting access to deployment for pmiazga: Gerrit, Release-Engineering-Team.
Fri, Apr 13, 5:23 PM · Release-Engineering-Team, Gerrit, Operations, Ops-Access-Requests
Dzahn added a comment to T192159: Requesting access to deployment for pmiazga.

Deployment access for Piotr has already been granted in T148477 back in October 2016 (including approval in Ops meeting).

Fri, Apr 13, 5:22 PM · Release-Engineering-Team, Gerrit, Operations, Ops-Access-Requests
Dzahn updated subscribers of T189524: Add Reading Infrastructure engineers to contacts for mobileapps.

Adding a custom contact group to the "LVS HTTP IPv4" service doesn't look trivial to me.

Fri, Apr 13, 2:09 AM · Patch-For-Review, Services (watching), monitoring, Operations

Thu, Apr 12

Dzahn added a comment to T189524: Add Reading Infrastructure engineers to contacts for mobileapps.

I debugged this by looking at the generated Icinga config directly on the server.

Thu, Apr 12, 11:54 PM · Patch-For-Review, Services (watching), monitoring, Operations
Dzahn added a comment to T191727: Add mr_IN locale to toolsforge.

18:24 < mutante> Notice: /Stage[main]/Toollabs::Exec_environ/Package[language-pack-mr]/ensure: created
18:25 < mutante> --- /var/lib/locales/supported.d/localI2015-04-22 18:32:38.847090958 +0000
18:25 < mutante> Scheduling refresh of Exec[locale-gen]

Thu, Apr 12, 10:31 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge
Dzahn added a comment to T192060: Request to add Tarrow to the ldap/wmde group .

@Tarrow yea, the first line can be removed. The second line can be amended to say that RStallman-legalteam should be added to do that. The third line can be amended to say that "Operations" should be added to actually do the technical change after the NDA has been signed. It could also say that the person who is currently listed as "on duty" on the #wikimedia-operations IRC channel can be contacted in case it's blocked by ops. Thanks!

Thu, Apr 12, 8:01 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn added a comment to T192060: Request to add Tarrow to the ldap/wmde group .

Afaict here also T191523#4125233 applies so L2 is not actually correct and whereever that template is that these tickets are created from, it should probably be updated.

Thu, Apr 12, 7:35 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn added a comment to T192060: Request to add Tarrow to the ldap/wmde group .

@RStallman-legalteam And.. here's another request for NDA for a WMDE developer, renamed ticket to keep them apart.

Thu, Apr 12, 7:34 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn renamed T191523: Request to add Matthias Geisler to the ldap/wmde group from Request to be added to the ldap/wmde group to Request to add Matthias Geisler to the ldap/wmde group.
Thu, Apr 12, 7:33 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn renamed T192060: Request to add Tarrow to the ldap/wmde group from Request to be added to the ldap/wmde group to Request to add Tarrow to the ldap/wmde group .
Thu, Apr 12, 7:33 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn added a comment to T191821: Host packaged helm charts at https://releases.wikimedia.org/charts.

https://releases.wikimedia.org/charts/

Thu, Apr 12, 6:03 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline
Dzahn claimed T192092: setup replacement for terbium (maintenance_server) on stretch.
Thu, Apr 12, 4:32 PM · Patch-For-Review, Operations
Dzahn created T192092: setup replacement for terbium (maintenance_server) on stretch.
Thu, Apr 12, 4:32 PM · Patch-For-Review, Operations
Dzahn added a comment to T189524: Add Reading Infrastructure engineers to contacts for mobileapps.

Hi, can you both login and then look at the "Logged in as " line showing up in the Icinga web ui and copy/paste that over here? Unfortunately there is a caveat where even capitalization matters, so let's compare the exact user name there.

Thu, Apr 12, 4:12 PM · Patch-For-Review, Services (watching), monitoring, Operations
Dzahn added a comment to T189524: Add Reading Infrastructure engineers to contacts for mobileapps.

@Mholloway Try this please:

Thu, Apr 12, 12:57 AM · Patch-For-Review, Services (watching), monitoring, Operations

Wed, Apr 11

Dzahn added a comment to T191478: Requesting access to shell (snapshot, dumpsdata) for springle.

Not sure if the patchset gives access to the bastions, otherwise I think we're ok.

Wed, Apr 11, 10:10 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn added a comment to T191523: Request to add Matthias Geisler to the ldap/wmde group.

adding @RStallman-legalteam for the NDA step

Wed, Apr 11, 10:03 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn updated subscribers of T191523: Request to add Matthias Geisler to the ldap/wmde group.
Wed, Apr 11, 10:02 PM · LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn added a comment to T186751: Restablish RESTBase dev environment with Cassandra 3.11.2.

fyi: T189050 / T189050#4124163 means you should not have worry anymore about scheduling downtimes for these services when on the dev environment.

Wed, Apr 11, 5:33 PM · Services (doing), User-Eevans
Dzahn renamed T189050: restbase/cassandra: skip (some) icinga monitoring if on "dev" machines from restbase: skip (some) icinga monitoring if on "dev" machines to restbase/cassandra: skip (some) icinga monitoring if on "dev" machines.
Wed, Apr 11, 5:32 PM · Patch-For-Review, monitoring, Operations
Dzahn updated subscribers of T189050: restbase/cassandra: skip (some) icinga monitoring if on "dev" machines.
Wed, Apr 11, 5:32 PM · Patch-For-Review, monitoring, Operations
Dzahn added a comment to T189050: restbase/cassandra: skip (some) icinga monitoring if on "dev" machines.

The change above will now ensure that cassandra Icinga checks are not added if on the dev cluster. We don't see the results yet because puppet is currently disabled on restbase-dev due to ongoing upgrades.

Wed, Apr 11, 5:32 PM · Patch-For-Review, monitoring, Operations
Dzahn closed T191972: Scap sync-file failing for deploy1001.eqiad.wmnet as Resolved.
Wed, Apr 11, 5:21 PM · Patch-For-Review, Deployments, Release-Engineering-Team, Operations
Dzahn added a comment to T191972: Scap sync-file failing for deploy1001.eqiad.wmnet.

deploy1001 has been removed from scap hosts and puppet ran on tin. This should have fixed the immediate scap issue.

Wed, Apr 11, 5:14 PM · Patch-For-Review, Deployments, Release-Engineering-Team, Operations
Dzahn claimed T191972: Scap sync-file failing for deploy1001.eqiad.wmnet.
Wed, Apr 11, 4:20 PM · Patch-For-Review, Deployments, Release-Engineering-Team, Operations
Dzahn added a comment to T191673: Update SSH key in production hosts for @Sharvaniharan.

@Sharvaniharan @MoritzMuehlenhoff The user name is now correct. The remaining issue (that looks like the same from outside but is a different problem now) is:

Wed, Apr 11, 6:02 AM · Patch-For-Review, Operations

Tue, Apr 10

Dzahn added a comment to T191673: Update SSH key in production hosts for @Sharvaniharan.

I see in your config you have

Tue, Apr 10, 11:33 PM · Patch-For-Review, Operations
Dzahn added a comment to T125976: Run mediawiki::maintenance scripts in Beta Cluster.

I suggest to create a fresh instance (that is not named after a hostname in prod but has a generic name) and apply role(mediawiki_maintenance) to it. Then you will see which errors you actually get (or not). The ones that just work you can keep and the ones that are breaking you disable in Hiera (the puppet class makes this easy, already disables all the crons on the inactive maintenance server (currently codfw). I don't think that "keep a list of what should be run manually" is going to work that well.

Tue, Apr 10, 8:59 PM · User-Addshore, Beta-Cluster-Infrastructure
Dzahn reopened T191673: Update SSH key in production hosts for @Sharvaniharan as "Open".
Tue, Apr 10, 7:03 PM · Patch-For-Review, Operations
Dzahn added a comment to T191673: Update SSH key in production hosts for @Sharvaniharan.

@Sharvaniharan I checked on releases1001 but i don't see any failed attempts to login with your user. It looks like it's already failing before that at the connection to the bastion host. Could you please paste your ssh config? Which bastion server are you connecting to to jump to releases1001? What errror do you see when you try it?

Tue, Apr 10, 7:02 PM · Patch-For-Review, Operations
Dzahn added a comment to T185504: Netbox: add Icinga check for PostgreSQL.

Thanks Gehel for pointing that out. Using the existing package is the better option. Abandoned.

Tue, Apr 10, 6:47 PM · Patch-For-Review, monitoring, Operations
Dzahn added a comment to T175288: setup/install/deploy deploy1001 as deployment server.

per the last ops meeting and joe's comments:

Tue, Apr 10, 6:15 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Operations
Dzahn added a comment to T190111: VirtualHost for mod_status breaks debugging Apache/MediaWiki from localhost.

The version of apache2.conf that canaries and mwdebug has matches the puppet repo template:

Tue, Apr 10, 6:02 PM · Performance-Team (Radar), Wikimedia-Apache-configuration, Operations
Dzahn added a comment to T190111: VirtualHost for mod_status breaks debugging Apache/MediaWiki from localhost.

I investigated a bit on the part ".. on mwdebug1001 and mwdebug1002, .. behaves differently on a pooled app server (e.g. mw1299)".

Tue, Apr 10, 5:55 PM · Performance-Team (Radar), Wikimedia-Apache-configuration, Operations
Dzahn added a comment to T185504: Netbox: add Icinga check for PostgreSQL.

seems like the best one. hast the most votes:

Tue, Apr 10, 2:01 AM · Patch-For-Review, monitoring, Operations
Dzahn added a comment to T185504: Netbox: add Icinga check for PostgreSQL.

https://exchange.nagios.org/directory/Plugins/Databases/PostgresQL

Tue, Apr 10, 1:58 AM · Patch-For-Review, monitoring, Operations
Dzahn closed T189910: upgrade bromine to stretch / reinstall as Resolved.

this is done in parent task.

Tue, Apr 10, 1:55 AM · Operations
Dzahn closed T189910: upgrade bromine to stretch / reinstall, a subtask of T188163: create codfw-equivalent of bromine, make webserver_misc_static active/active in misc varnish, as Resolved.
Tue, Apr 10, 1:55 AM · Availability, Patch-For-Review, Operations

Mon, Apr 9

Dzahn closed T190439: add tftpd monitoring as Resolved.
Mon, Apr 9, 9:51 PM · Patch-For-Review, Operations, monitoring
Dzahn added a comment to T191613: planet: Image in blog post appear missing.

I have the same suspicion. There are a lot of characters before that 'src=' starts after the image tag. That seems the uncommon part about it. While the software does log which feeds it is parsing and whether there is a general error, it does not log details about the parsing/image stripping part .

Mon, Apr 9, 2:23 PM · Wikimedia-Planet

Sat, Apr 7

Dzahn created T191683: missing html links for Phabricator Blog feeds.
Sat, Apr 7, 1:11 AM · Phabricator (Upstream), Upstream, Wikimedia-Planet
Dzahn closed T188163: create codfw-equivalent of bromine, make webserver_misc_static active/active in misc varnish as Resolved.

All done!

Sat, Apr 7, 12:48 AM · Availability, Patch-For-Review, Operations
Dzahn updated the task description for T188163: create codfw-equivalent of bromine, make webserver_misc_static active/active in misc varnish.
Sat, Apr 7, 12:47 AM · Availability, Patch-For-Review, Operations