Page MenuHomePhabricator

Dzahn (Daniel Zahn)
Operations EngineerAdministrator

Projects (21)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (295 w, 11 h)
Roles
Administrator
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Unknown

Recent Activity

Yesterday

Dzahn reassigned T211692: Duplicate definitions found in Icinga configuration from Dzahn to herron.
Tue, May 26, 4:20 PM · Operations, observability
Dzahn closed T211692: Duplicate definitions found in Icinga configuration as Resolved.

Thank you @herron ! Yes, confirmed. No more duplicates right now.

Tue, May 26, 4:19 PM · Operations, observability
Dzahn added a comment to T251726: Certificate *.wikipedia.org valid until 2020-06-20.

IMHO 7 / 3 is not enough for the unified cert even when LE is the issuer considering our anti clock skew measures and that acme-chief should issue the new cert 30 days before the valid one expires

Tue, May 26, 3:25 PM · Patch-For-Review, Traffic, serviceops, Operations
Dzahn added a comment to T253640: Requesting access to deployment rights for BPirkle .

To SRE doing clinic duty: This is an easy one. Really just need to move the existing user from group "restricted" to group "deployment" and the other steps are all done from the previous request.

Tue, May 26, 3:09 PM · Operations, SRE-Access-Requests
Dzahn added a comment to T224549: Track remaining jessie systems in production.

percentage of jessie systems left as of today (because we were asked): 4.2%

Tue, May 26, 1:55 PM · Operations
Dzahn closed T242606: No mw canary servers in codfw as Resolved.
Tue, May 26, 12:57 PM · Operations, serviceops
Dzahn added a comment to T242606: No mw canary servers in codfw.

mw2187, mw2188 are new canary appservers, replacing mw2271, mw2272

Tue, May 26, 12:57 PM · Operations, serviceops
Dzahn added a comment to T168459: Cleanup planet.wikimedia.org feeds database.

I'm afraid this is one of those tickets that you can never really close because at any given time there will be 1 or more feeds with errors. Some will be fixed later, some will go away, some will change URLs due to new software... It is just a constant maintenance and one has to check the logs every once in a while. I am not sure I want to keep a ticket open forever though.. .so at some point it will have to be "good enough for now".

Tue, May 26, 10:10 AM · Wikimedia-Planet
Dzahn added a comment to T168459: Cleanup planet.wikimedia.org feeds database.

The 2 boxes not checked on this ticket are meanwhile working again. A good example why deleting all feeds with an occasional 404 too quickly is not the best idea.

Tue, May 26, 9:42 AM · Wikimedia-Planet
Dzahn closed T253296: decom people1001, a subtask of T247649: upgrade people.wikimedia.org backend to buster, as Resolved.
Tue, May 26, 9:31 AM · serviceops, Operations
Dzahn closed T253296: decom people1001 as Resolved.
Tue, May 26, 9:30 AM · Patch-For-Review, serviceops, Operations

Fri, May 22

Dzahn added a comment to T247018: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers.

Technically resolved because we made more than enough room for the 5 (not 15 anymore, 10 were used for T252185) servers.

Fri, May 22, 2:40 PM · decommission, Patch-For-Review, Operations, ops-codfw, serviceops
Dzahn changed the status of T241852: (Need by: TBD) rack/setup/install 86 new codfw mw systems from Stalled to Open.

23 servers from rack C3 have been decom'ed. mw2150 through mw2172. (lower part of the rack)

Fri, May 22, 2:34 PM · ops-codfw, serviceops, Operations
Dzahn created T253384: add monitoring of sustained memcached TKO rates.
Fri, May 22, 1:44 PM · Sustainability (Incident Followup), observability, Operations
Dzahn changed the status of T247021: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet), a subtask of T241852: (Need by: TBD) rack/setup/install 86 new codfw mw systems, from Stalled to Open.
Fri, May 22, 12:19 PM · ops-codfw, serviceops, Operations
Dzahn changed the status of T247021: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) from Stalled to Open.
Fri, May 22, 12:19 PM · serviceops, Operations
Dzahn added a comment to T247018: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers.

@Papaul 20 servers from rack C3 have been decom'ed. mw2150 through mw2169. (lower part of the rack)

Fri, May 22, 12:18 PM · decommission, Patch-For-Review, Operations, ops-codfw, serviceops
Dzahn reopened T242606: No mw canary servers in codfw as "Open".

reopening because i am decom'ing servers in T247018 and that included some canaries.

Fri, May 22, 11:35 AM · Operations, serviceops
Dzahn renamed T247018: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers from codfw: decom at least 15 appservers(mw2158 through mw2172) in codfw rack C3 to make room for new servers to codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers.
Fri, May 22, 11:14 AM · decommission, Patch-For-Review, Operations, ops-codfw, serviceops
Dzahn added a comment to T247649: upgrade people.wikimedia.org backend to buster.

These were changed in https://gerrit.wikimedia.org/r/c/operations/dns/+/595959/2/templates/wmnet

Fri, May 22, 9:56 AM · serviceops, Operations
Dzahn raised the priority of T149924: Clear /srv/.git on contint1001; move integration.wikimedia.org docroot to new location from Low to Medium.
Fri, May 22, 8:32 AM · Release-Engineering-Team-TODO, Release-Engineering-Team (CI & Testing services), Patch-For-Review, Technical-Debt, Continuous-Integration-Infrastructure
Dzahn added a comment to T149924: Clear /srv/.git on contint1001; move integration.wikimedia.org docroot to new location.

As disussed on IRC, i'd rather not use /srv/docroot and would prefer instead if we can clean out the repo called "docroot" to actually just contain files for the docroot.

Fri, May 22, 8:31 AM · Release-Engineering-Team-TODO, Release-Engineering-Team (CI & Testing services), Patch-For-Review, Technical-Debt, Continuous-Integration-Infrastructure
Dzahn added a comment to T251732: wikiworkshop.org has Facebook button, external statcounter, https to http redirect.

Is there a specific thing we are waiting for?

Fri, May 22, 8:17 AM · Privacy, Research, Privacy Engineering, Traffic, Operations
Dzahn renamed T252815: iegreview: missing grants@ sender address (was: login failing with csrf token missing warning) from Iegreview login failing with csrf token missing warning to iegreview: missing grants@ sender address (was: login failing with csrf token missing warning).
Fri, May 22, 8:15 AM · Wikimedia-IEG-grant-review
Dzahn added a comment to T252815: iegreview: missing grants@ sender address (was: login failing with csrf token missing warning).

@Mjohnson_WMF I temporarily added an alias for grants@ to myself on the mail servers and then hit the password recovery form again. I see it has now delivered an email to you. Please check if you received it and you are now unblocked.

Fri, May 22, 8:08 AM · Wikimedia-IEG-grant-review
Dzahn added a comment to T252815: iegreview: missing grants@ sender address (was: login failing with csrf token missing warning).

It seems like the previously existing grants@wikimedia.org Google group has been deleted as part of T191881 or otherwise.

Fri, May 22, 7:55 AM · Wikimedia-IEG-grant-review
Dzahn added a comment to T252815: iegreview: missing grants@ sender address (was: login failing with csrf token missing warning).

@Mjohnson_WMF @bd808 I just tested sending the password recovery mail to you while watching log files on the server and I found:

Fri, May 22, 7:46 AM · Wikimedia-IEG-grant-review
Dzahn reopened T252815: iegreview: missing grants@ sender address (was: login failing with csrf token missing warning) as "Open".
Fri, May 22, 6:29 AM · Wikimedia-IEG-grant-review

Thu, May 21

Dzahn added a comment to T252526: serve tftpboot environment from the install servers and create one in each edge POP.

wmcs team reported not being able to do installs from the cloudvirt VLAN (cloudnet1004)

Thu, May 21, 2:53 PM · Operations
Dzahn added a comment to T67074: Set up dev.wikimedia.org portal.

please also see T246945 which might conflict with this effort and introduce yet another place for documentation rather than having a single place for developers to go to

Thu, May 21, 12:00 PM · WorkType-NewFunctionality, Developer-Advocacy, Wikimedia-General-or-Unknown
Dzahn added a comment to T246945: New Public Wiki for the API Portal.
Thu, May 21, 11:57 AM · Release-Engineering-Team-TODO, CPT Initiatives (API Gateway), User-brennen, User-Ladsgroup, User-Urbanecm, Wiki-Setup (Create), Core Platform Team
Dzahn changed the status of T247018: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers from Stalled to Open.
Thu, May 21, 11:42 AM · decommission, Patch-For-Review, Operations, ops-codfw, serviceops
Dzahn created T253296: decom people1001.
Thu, May 21, 10:43 AM · Patch-For-Review, serviceops, Operations
Dzahn closed T247649: upgrade people.wikimedia.org backend to buster, a subtask of T247045: Migrate all of production metal and VMs to Buster or later, as Resolved.
Thu, May 21, 10:43 AM · Operations, Epic
Dzahn closed T247649: upgrade people.wikimedia.org backend to buster as Resolved.

This has happened and an announcement has been sent to ops and wikitech-l lists.

Thu, May 21, 10:43 AM · serviceops, Operations
Dzahn added a comment to T253263: Add a second Gerrit connection in Zuul config.

@QChris @hashar Do you see a new for a DNS name like "gerrit-new.wikimedia.org" or someting similar which then can be used for the baseurl in this example and other things?

Thu, May 21, 9:39 AM · Patch-For-Review, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Zuul, Continuous-Integration-Infrastructure
Dzahn closed T253277: Add LMata to wmf ldap group as Resolved.

@lmata Welcome! This is done. You should now be able to log into Icinga, Grafana etc.

Thu, May 21, 9:02 AM · Operations, SRE-Access-Requests, LDAP-Access-Requests
Dzahn placed T167066: New wikistats interface takes minutes to load the mediawikis list up for grabs.

unassigning from me because i got the "cookie-licking" warning from T228575

Thu, May 21, 8:53 AM · VPS-project-Wikistats
Dzahn added a comment to T253292: check_http and SNI support.

I think option 3) is the easiest of all, has no risk to break existing checks and we already have many different check_commands using check_http in different ways so another one should not hurt us.

Thu, May 21, 8:50 AM · Operations, Traffic
Dzahn moved T253086: Add Daniel Cipoletti to analytics-privatedata-users from Backlog to Awaiting User Input on the LDAP-Access-Requests board.
Thu, May 21, 8:14 AM · Operations, LDAP-Access-Requests
Dzahn changed the status of T252210: Eqiad: 1VM request for Peek (PM service in use by Security Team), a subtask of T242285: Create status mechanism(s) for security-team@ combining Asana and Phab, from Open to Stalled.
Thu, May 21, 8:04 AM · PM, Security-Team
Dzahn changed the status of T252210: Eqiad: 1VM request for Peek (PM service in use by Security Team) from Open to Stalled.

Giving it back to the pool and setting to stalled because of ongoing discussion whether this should be on a dedicated VM or on mwmaint.

Thu, May 21, 8:04 AM · serviceops, PM, Security-Team, vm-requests, Operations
Dzahn reassigned T251349: Request for srv/phab/phabricator/bin/bulk make-silent --id * command via SSH for moving tasks quarterly from Dzahn to MBinder_WMF.

Thanks for taking care of this @RLazarus I can confirm Max's user exists on the Phabricator prod server, is in the new group and that group has the sudo privileges to run the requested comment.

Thu, May 21, 8:01 AM · SRE-Access-Requests, Operations
Dzahn added a comment to T249916: access request on cumin[1-2]001 for John Clark.

from the log files on bast1002 and cumin1001 I can see there are 2 different keys involved.

Thu, May 21, 7:06 AM · SRE-Access-Requests, Operations, DC-Ops
Dzahn added a comment to T249506: Create Wiktionary Konkani.

I already created an item on it a couple of days ago so I've just merged it https://www.wikidata.org/wiki/Q94700087

Thu, May 21, 6:38 AM · User-Ladsgroup, MW-1.35-notes (1.35.0-wmf.30; 2020-04-28), Wiki-Setup (Create), User-Urbanecm
Dzahn added a comment to T191182: Stop using Differential for code review.

@Dzahn's explanation ignores all of T191182#4935787 and the literal fact that there are nearly 350 Diffusion repos for Toolforge tools which have been created through Striker. My declining moving 1 of these 350 repos to gerrit was not the root cause of the blockage here.

Thu, May 21, 6:37 AM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Phabricator, Gerrit

Wed, May 20

Dzahn added a comment to T249506: Create Wiktionary Konkani.

added to Wikidata: https://www.wikidata.org/wiki/Q94952969

Wed, May 20, 2:01 PM · User-Ladsgroup, MW-1.35-notes (1.35.0-wmf.30; 2020-04-28), Wiki-Setup (Create), User-Urbanecm
Dzahn added a comment to T249506: Create Wiktionary Konkani.

Added to wikistats.wmflabs.org

Wed, May 20, 1:55 PM · User-Ladsgroup, MW-1.35-notes (1.35.0-wmf.30; 2020-04-28), Wiki-Setup (Create), User-Urbanecm
Dzahn added a comment to T252870: Add Wikidata support for awawiki.

https://www.wikidata.org/wiki/Q94694371

Wed, May 20, 1:45 PM · Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, Wikidata
Dzahn added a comment to T252870: Add Wikidata support for awawiki.

https://www.wikidata.org/wiki/Q94952446

Wed, May 20, 1:37 PM · Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, Wikidata
Dzahn closed T252869: Add awawiki to wikistats as Resolved.
Wed, May 20, 1:35 PM · VPS-project-Wikistats
Dzahn added a comment to T252869: Add awawiki to wikistats.

re-added now that the wiki has been created.

Wed, May 20, 1:35 PM · VPS-project-Wikistats
Dzahn added a comment to T253229: Wikistats is out of date for miraheze.

Thanks for reporting. Likely it will be similar to:

Wed, May 20, 1:22 PM · User-RhinosF1, VPS-project-Wikistats
Dzahn added a comment to T253024: Create a Ganeti VM for Wikidough.

+---[RSA 2048]----+ +---[ECDSA 256]---+ +--[ED25519 256]--+

=+....
. o.o.. + . ..
.. E + o.X o. ..
+. +.* . .o o *+.. ..
+ = S o+S+.=.o .. B = oS.. .
E o & = .+ *o+.=.o .= X .+E. .. .
o* .. + o X. = Xo=o. o= =.. o ..oo.
= =o+*o+ = B. o +.B .oo +.+ .. .oo.
. .=OB+o=.o... .o=o....*.... ..

+----[SHA256]-----+ +----[SHA256]-----+ +----[SHA256]-----+

Wed, May 20, 1:16 PM · Patch-For-Review, Traffic, Operations, vm-requests
Dzahn added a comment to T252132: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) public resolver.

A VM called malmok.wikimedia.org has been created and can be used now. Currently it has the "insetup" role in site.pp.

Wed, May 20, 1:16 PM · Patch-For-Review, Operations, Traffic
Dzahn closed T253024: Create a Ganeti VM for Wikidough as Resolved.
Wed, May 20, 1:15 PM · Patch-For-Review, Traffic, Operations, vm-requests
Dzahn added a comment to T253024: Create a Ganeti VM for Wikidough.

@ssingh The VM has been created (now with public IP).

Wed, May 20, 1:15 PM · Patch-For-Review, Traffic, Operations, vm-requests
Dzahn added a comment to T252703: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue).

SOworu worked. Now in Turnilo. Thanks.

Wed, May 20, 12:49 PM · Analytics-Kanban, Analytics, LDAP-Access-Requests, Operations
Dzahn added a comment to T252703: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue).

@soworu I found the following comment in the Apache config of superset:

Wed, May 20, 12:45 PM · Analytics-Kanban, Analytics, LDAP-Access-Requests, Operations
Dzahn added a comment to T252703: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue).

@soworu Looks like it works now, i think i just saw you login in the log files, am i right?

Wed, May 20, 12:34 PM · Analytics-Kanban, Analytics, LDAP-Access-Requests, Operations
Dzahn added a comment to T252703: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue).

@soworu Try again now, please. Looks like i forgot to add the -01 part when adding you the right group. Sorry about that.

Wed, May 20, 12:28 PM · Analytics-Kanban, Analytics, LDAP-Access-Requests, Operations
Dzahn added a comment to T252703: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue).

Please try the following variations of the username including the capitalization:

Wed, May 20, 12:08 PM · Analytics-Kanban, Analytics, LDAP-Access-Requests, Operations
Dzahn added a comment to T252703: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue).

@soworu Use the same user/password you used on https://wikitech.wikimedia.org when you created your account there.

I still can't login into Superset and Turnilo using my Wikitech login details. Can you help?

Wed, May 20, 12:06 PM · Analytics-Kanban, Analytics, LDAP-Access-Requests, Operations
Dzahn added a comment to T165348: Check long-running screen/tmux sessions.

I would like to separate better "things that can wait" from "outages or potential outages" on different dashboards.

Wed, May 20, 11:10 AM · Patch-For-Review, observability, Operations
Dzahn changed the status of T191182: Stop using Differential for code review from Open to Stalled.

This is apparently stalled because T252910 has been declined with a reason that there were only few commits in a specific repo. That seems to cause a dependency cycle though which i find pretty unfortunate because what EddieGP describes here is totally legit and really the worst possible outcome.

Wed, May 20, 10:48 AM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Phabricator, Gerrit
Dzahn added a comment to T228926: rack/setup/instal (4) CI ganeti nodes.

These 4 hosts have been reimaged and now have RAID5 instead of RAID1 after gerrit:597261

Wed, May 20, 9:49 AM · Operations
Dzahn reassigned T228924: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet from Dzahn to akosiaris.

Handing back over for the next "init" command steps you have mentioned are needed next.

Wed, May 20, 9:33 AM · Patch-For-Review, serviceops, Operations
Dzahn added a comment to T228924: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet.

@akosiaris All of these hosts have RAID5 now:

Wed, May 20, 9:32 AM · Patch-For-Review, serviceops, Operations
Dzahn added a comment to T228926: rack/setup/instal (4) CI ganeti nodes.

Enabled remote IPMI on these machines which was disabled but is needed. (wikitech how to

Wed, May 20, 9:08 AM · Operations
Dzahn added a comment to T228924: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet.

@RobH @Cmjohnson I noticed by chance there are more ganeti machines beyond ganeti1018. ganeti1019-ganeti1022 are in netbox but i don't see a racking ticket for them. Should there be one?

Wed, May 20, 9:04 AM · Patch-For-Review, serviceops, Operations
Dzahn added a comment to T253024: Create a Ganeti VM for Wikidough.
Ready to create Ganeti VM malmok.codfw.wmnet in the ganeti01.svc.codfw.wmnet cluster on row A with 2 vCPUs, 8GB of RAM, 30GB of disk in the private network.
Wed, May 20, 8:53 AM · Patch-For-Review, Traffic, Operations, vm-requests
Dzahn updated subscribers of T228924: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet.

@RobH Remote IPMI was disabled on these hosts which popped up when i tried to run the reimage cookbook (to change software RAID level from 1 to 5) and it failed.

Wed, May 20, 8:28 AM · Patch-For-Review, serviceops, Operations
Dzahn added a comment to T224591: Migrate contint* hosts to Buster.

@Dzahn would you be available at the beginning of next week (Monday/Tuesday)? (we can sync up over irc to find a good time)

Wed, May 20, 7:46 AM · Patch-For-Review, Release-Engineering-Team-TODO, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure (phase-out-jessie), Operations
Dzahn added a comment to T252932: Forwarding or alias for fundraising@.
Wed, May 20, 7:31 AM · Operations, Mail
Dzahn added a comment to T205361: Make an HTML dump of the output of the CodeReview extension on MediaWiki.org.

Also, is it MediaWiki/1.html or MediaWiki/rev/1.html. I've seen both versions. It seems we're back to the former?

Wed, May 20, 6:36 AM · Core Platform Team Workboards (Clinic Duty Team), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), MediaWiki-extensions-CodeReview
Dzahn closed T252875: LDAP access request - add Christian Aistleitner to "nda" (or "wmf"), a subtask of T200739: Upgrade to Gerrit 2.16.13, as Resolved.
Wed, May 20, 6:24 AM · Patch-For-Review, Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), Epic, Developer Productivity, Release-Engineering-Team (Development services), Gerrit
Dzahn closed T252875: LDAP access request - add Christian Aistleitner to "nda" (or "wmf") as Resolved.

@QChris I added you to the nda group. You should now be able to login to Icinga.

Wed, May 20, 6:24 AM · Operations, LDAP-Access-Requests
Dzahn claimed T252875: LDAP access request - add Christian Aistleitner to "nda" (or "wmf").
Wed, May 20, 6:20 AM · Operations, LDAP-Access-Requests

Tue, May 19

Dzahn claimed T253024: Create a Ganeti VM for Wikidough.
Tue, May 19, 12:22 PM · Patch-For-Review, Traffic, Operations, vm-requests
Dzahn changed the status of T252869: Add awawiki to wikistats from Open to Stalled.
Tue, May 19, 11:13 AM · VPS-project-Wikistats
Dzahn triaged T252869: Add awawiki to wikistats as Medium priority.
Tue, May 19, 11:13 AM · VPS-project-Wikistats
Dzahn reopened T252869: Add awawiki to wikistats as "Open".

Deleted it again for now because adding it before the wiki has been created cause more issues like the one linked above and then T253115.

Tue, May 19, 11:13 AM · VPS-project-Wikistats
Dzahn added a comment to T253115: wikimedia_sites.py wrongly gives "awa" as a new site.

@Xqt I have deleted "awa" from the wikistats db again. I will reopen the task to re-add it once it has actually been created.

Tue, May 19, 11:11 AM · Upstream, VPS-project-Wikistats, Pywikibot-Scripts, Pywikibot
Dzahn added a comment to T253115: wikimedia_sites.py wrongly gives "awa" as a new site.

I think the best solution is probably that i don't add sites before they are created and simply delete it for now.

Tue, May 19, 11:09 AM · Upstream, VPS-project-Wikistats, Pywikibot-Scripts, Pywikibot
Dzahn closed T162070: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases as Resolved.

done! the module has been removed

Tue, May 19, 11:03 AM · Patch-For-Review, Operations, DBA
Dzahn added a comment to T253097: wikistats_tests fails.

I did the latter and set total= and good= to 0 in the db. Did that fix it, @Xqt?

Tue, May 19, 9:36 AM · Upstream, VPS-project-Wikistats, Pywikibot, Pywikibot-tests
Dzahn added a comment to T252869: Add awawiki to wikistats.

(If this requires creating the wiki then wiki creation is a subtask not a parent task, as the subtask first needs to be solved)

Tue, May 19, 9:32 AM · VPS-project-Wikistats
Dzahn added a comment to T253097: wikistats_tests fails.

I could also just set "total" to 0 manually for now so that it has a number.

Tue, May 19, 9:31 AM · Upstream, VPS-project-Wikistats, Pywikibot, Pywikibot-tests
Dzahn added a comment to T253097: wikistats_tests fails.

I did not even know "wikistats_tests" is a thing.

Tue, May 19, 9:31 AM · Upstream, VPS-project-Wikistats, Pywikibot, Pywikibot-tests
Dzahn added a comment to T253024: Create a Ganeti VM for Wikidough.

Is the requested hostname "homer" a copy/paste error?

Tue, May 19, 8:27 AM · Patch-For-Review, Traffic, vm-requests, Operations
Dzahn closed T252190: delete the puppet module "apache" as Resolved.

This has happened. The module is gone now.

Tue, May 19, 8:11 AM · Puppet, serviceops, Operations
Dzahn added a comment to T252956: Add Git LFS support for research/wikiworkshop.

I think both Commons and Youtube, depending on licenses, would be good places to store the actual video. The wikiworkshop site could then link to them or embed them directly.

Tue, May 19, 6:54 AM · Operations, Research

Mon, May 18

Dzahn closed T208878: Puppet errors on glampipe.glampipe.eqiad.wmflabs, a subtask of T215662: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it, as Resolved.
Mon, May 18, 1:00 PM · cloud-services-team (Kanban), Technical-Debt, Patch-For-Review, serviceops, Puppet, Cloud-VPS
Dzahn closed T208878: Puppet errors on glampipe.glampipe.eqiad.wmflabs, a subtask of T208856: Instances with Puppet failures, as Resolved.
Mon, May 18, 1:00 PM · Tracking-Neverending, Cloud-VPS, cloud-services-team (Kanban)
Dzahn closed T208878: Puppet errors on glampipe.glampipe.eqiad.wmflabs as Resolved.

I fixed this by creating "role::simplelamp2" which uses mariadb instead of mysql. This has been applied on glampipe instead of the old simplelamp role and puppet runs fine again.

Mon, May 18, 12:59 PM · Cloud-VPS
Dzahn closed T202574: convert cloud VPS projects from apache to httpd module as Resolved.

This is done.

Mon, May 18, 12:55 PM · cloud-services-team (Kanban), Cloud-VPS, Operations
Dzahn closed T165348: Check long-running screen/tmux sessions as Resolved.

Ok, resolving. Note: The thresholds are currently set to 240 hours (10 days) for WARN and 480 hours (20 days) for CRIT.

Mon, May 18, 9:04 AM · Patch-For-Review, observability, Operations
Dzahn added a comment to T165348: Check long-running screen/tmux sessions.

I think "how to handle Icinga warnings" is not something specific to this task about monitoring screens.

Mon, May 18, 8:50 AM · Patch-For-Review, observability, Operations
Dzahn closed T215662: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it, a subtask of T162070: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases, as Resolved.
Mon, May 18, 8:44 AM · Patch-For-Review, Operations, DBA
Dzahn closed T215662: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it, a subtask of T128642: role::simplelamp fails to start mysql due to apparmor, as Resolved.
Mon, May 18, 8:44 AM · Operations, Puppet, Cloud-VPS