Page MenuHomePhabricator

Dzahn (Daniel Zahn)
SREAdministrator

Projects (26)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (443 w, 3 d)
Roles
Administrator
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Mutante [ Global Accounts ]

Recent Activity

Today

Dzahn updated subscribers of T333212: replace dzahn as signer in pwstore.
Sat, Apr 1, 1:36 AM · Infrastructure Security, serviceops-collab
Dzahn updated subscribers of T333212: replace dzahn as signer in pwstore.
Sat, Apr 1, 1:36 AM · Infrastructure Security, serviceops-collab
Dzahn added a project to T333212: replace dzahn as signer in pwstore: Infrastructure Security.
Sat, Apr 1, 1:35 AM · Infrastructure Security, serviceops-collab
Dzahn renamed T333212: replace dzahn as signer in pwstore from replace myself as signer in pwstore to replace dzahn as signer in pwstore.
Sat, Apr 1, 1:34 AM · Infrastructure Security, serviceops-collab

Yesterday

Dzahn closed T327976: create blackbox::http monitoring for miscweb, a subtask of T329587: http::blackbox monitoring for all https services (let all serviceops-collab alertmanager alerts create tickets), as Resolved.
Fri, Mar 31, 10:55 PM · Patch-For-Review, serviceops-collab
Dzahn closed T327976: create blackbox::http monitoring for miscweb as Resolved.
Fri, Mar 31, 10:55 PM · serviceops-collab
Dzahn updated the task description for T327976: create blackbox::http monitoring for miscweb.
Fri, Mar 31, 10:54 PM · serviceops-collab
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.

I wouldn't mind merging a patch or uploading one that goes back to "every 8 hours".

Fri, Mar 31, 6:17 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn awarded T333663: mwbot-rs large cache usage on Gitlab CI a Like token.
Fri, Mar 31, 6:15 PM · GitLab (CI & Job Runners), mwbot-rs
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.

Now that I think about it more, perhaps running clear-docker-cache.timer hourly is too aggressive, as in, folks would not benefit from the cache?

Fri, Mar 31, 5:34 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn updated subscribers of T333212: replace dzahn as signer in pwstore.

also: maybe it should be more than 2 people for all of SRE nowadays? not sure. cc: @Muehlenhoff

Fri, Mar 31, 5:05 PM · Infrastructure Security, serviceops-collab
Aklapper awarded T333212: replace dzahn as signer in pwstore a Like token.
Fri, Mar 31, 4:34 PM · Infrastructure Security, serviceops-collab
Dzahn added a comment to T333212: replace dzahn as signer in pwstore.

Since I am planning to go on a sabbatical I should find someone to replace me as one of the only 2 users who can add/remove users in pwstore.

Fri, Mar 31, 4:20 PM · Infrastructure Security, serviceops-collab
Dzahn closed T333683: ProbeDown - gitlab2002 as Resolved.

works again per logstash (no errors shown)

Fri, Mar 31, 4:19 PM · serviceops-collab
Dzahn added a project to T333656: Decommission query-preview.wikdiata.org: serviceops-collab.
Fri, Mar 31, 3:57 PM · serviceops-collab, Wikidata, Wikidata-Query-Service
Dzahn renamed T333683: ProbeDown - gitlab2002 from ProbeDown to ProbeDown - gitlab2002.
Fri, Mar 31, 1:50 PM · serviceops-collab

Thu, Mar 30

Dzahn updated the task description for T331901: check icinga for any alerts custom to sre-collab related services.
Thu, Mar 30, 10:33 PM · Patch-For-Review, Observability-Alerting, serviceops-collab
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.

well there is still https://gerrit.wikimedia.org/r/c/operations/puppet/+/904616 as a follow-up action.. so maybe it should still be open.

Thu, Mar 30, 10:05 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn closed T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space as Resolved.

available disk space in /var/lib/docker is back to: used: 21G available: 17G usage: 56%

Thu, Mar 30, 10:04 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.

For the immediate action one can delete the volumes on the runner and that solves this task.

Thu, Mar 30, 10:02 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.
root@runner-1030:/var/lib/docker/volumes# for volume in $(du -hs * | grep G | cut -d "G" -f2 | xargs); do ls -1 ${volume}/_data/*; du -hs ${volume}; done
mwbot
mwbot.tmp
1.1G	runner-m4mqfjvt-project-1177-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
mwbot
mwbot.tmp
1.7G	runner-m4mqfjvt-project-1177-concurrent-1-cache-c33bcaa1fd2c77edfc3893b41966cea8
mwbot
mwbot.tmp
4.5G	runner-m4mqfjvt-project-1177-concurrent-2-cache-c33bcaa1fd2c77edfc3893b41966cea8
mwbot
mwbot.tmp
1.1G	runner-m4mqfjvt-project-1187-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
poty-stuff
poty-stuff.tmp
2.0G	runner-m4mqfjvt-project-1215-concurrent-2-cache-c33bcaa1fd2c77edfc3893b41966cea8
apt-browser
apt-browser.tmp
1.7G	runner-m4mqfjvt-project-828-concurrent-2-cache-c33bcaa1fd2c77edfc3893b41966cea8
upcoming-mainpage
upcoming-mainpage.tmp
1.8G	runner-m4mqfjvt-project-837-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
mwbot-rs
3.2G	runner-m4mqfjvt-project-860-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
mwbot-rs
1.7G	runner-m4mqfjvt-project-860-concurrent-2-cache-c33bcaa1fd2c77edfc3893b41966cea8
mwbot-rs
1.6G	runner-m4mqfjvt-project-860-concurrent-3-cache-c33bcaa1fd2c77edfc3893b41966cea8
data-engineering
1.4G	runner-m4mqfjvt-project-93-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
data-engineering
2.1G	runner-m4mqfjvt-project-93-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
data-engineering
1.1G	runner-m4mqfjvt-project-93-concurrent-1-cache-c33bcaa1fd2c77edfc3893b41966cea8
Thu, Mar 30, 9:52 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn updated the task description for T333605: for all our TCP monitoring checks, add an actual send/expect pattern.
Thu, Mar 30, 9:30 PM · serviceops-collab
Dzahn created T333605: for all our TCP monitoring checks, add an actual send/expect pattern.
Thu, Mar 30, 9:29 PM · serviceops-collab
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.

Does it make sense to consider running the cron jobs more frequently?

Thu, Mar 30, 7:08 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.

Alright, thanks dancy!

Thu, Mar 30, 5:43 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.

I wonder if this happens to be the runner from:

Thu, Mar 30, 5:04 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn added projects to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space: serviceops-collab, Release-Engineering-Team.
Thu, Mar 30, 5:01 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn added a comment to T333586: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space.

usage right now in / is 40% and in /var/lib/docker it's 85%

Thu, Mar 30, 5:00 PM · Release-Engineering-Team, serviceops-collab, GitLab
Dzahn updated subscribers of T329974: Show "other assignee" avatar on tasks in workboard.
Thu, Mar 30, 3:43 PM · Phabricator, Patch-For-Review, Release-Engineering-Team (Radar), User-brennen
Dzahn updated the task description for T327976: create blackbox::http monitoring for miscweb.
Thu, Mar 30, 1:38 AM · serviceops-collab
Dzahn moved T294276: contint hardware refresh from Backlog to Work in Progress on the serviceops-collab board.
Thu, Mar 30, 1:37 AM · Continuous-Integration-Infrastructure, serviceops-collab, Release-Engineering-Team (Seen)
Dzahn moved T319477: Migrate doc hosts to Bullseye from Backlog to Work in Progress on the serviceops-collab board.
Thu, Mar 30, 1:37 AM · Patch-For-Review, Continuous-Integration-Infrastructure, serviceops-collab
Dzahn moved T324659: contint2002 service implementation tracking from Backlog to Work in Progress on the serviceops-collab board.
Thu, Mar 30, 1:37 AM · Patch-For-Review, Continuous-Integration-Infrastructure, serviceops-collab, SRE
Dzahn changed the status of T324659: contint2002 service implementation tracking from Open to In Progress.
Thu, Mar 30, 1:37 AM · Patch-For-Review, Continuous-Integration-Infrastructure, serviceops-collab, SRE
Dzahn changed the status of T324659: contint2002 service implementation tracking, a subtask of T294276: contint hardware refresh, from Open to In Progress.
Thu, Mar 30, 1:37 AM · Continuous-Integration-Infrastructure, serviceops-collab, Release-Engineering-Team (Seen)
Dzahn added a comment to T324659: contint2002 service implementation tracking.

The production role for ci::master is now applied on contint2002.

Thu, Mar 30, 1:36 AM · Patch-For-Review, Continuous-Integration-Infrastructure, serviceops-collab, SRE
Dzahn added a project to T333516: Phabricator deployment 2023-03-30: serviceops-collab.
Thu, Mar 30, 1:31 AM · serviceops-collab, User-brennen, Release-Engineering-Team, Phabricator
Dzahn closed T333517: ProbeDown - commons_query_wikimedia_org, a subtask of T333510: clean up commons-query on miscweb setup, as Resolved.
Thu, Mar 30, 1:30 AM · serviceops-collab
Dzahn closed T333517: ProbeDown - commons_query_wikimedia_org as Resolved.

01:20 <+jinxer-wm> (ProbeDown) resolved: (7) Service miscweb1002:443 has failed probes (http_commons_query_wikimedia_org_ip4) -

Thu, Mar 30, 1:30 AM · serviceops-collab
Dzahn added a parent task for T333517: ProbeDown - commons_query_wikimedia_org: T333510: clean up commons-query on miscweb setup.
Thu, Mar 30, 1:29 AM · serviceops-collab
Dzahn added a subtask for T333510: clean up commons-query on miscweb setup: T333517: ProbeDown - commons_query_wikimedia_org.
Thu, Mar 30, 1:29 AM · serviceops-collab
Dzahn renamed T333517: ProbeDown - commons_query_wikimedia_org from ProbeDown to ProbeDown - commons_query_wikimedia_org.
Thu, Mar 30, 1:28 AM · serviceops-collab
Dzahn added a parent task for T333515: ProbeDown - commons_query_wikimedia_org: T333510: clean up commons-query on miscweb setup.
Thu, Mar 30, 1:24 AM · Patch-For-Review, serviceops-collab
Dzahn added a subtask for T333510: clean up commons-query on miscweb setup: T333515: ProbeDown - commons_query_wikimedia_org.
Thu, Mar 30, 1:24 AM · serviceops-collab
Dzahn closed T333515: ProbeDown - commons_query_wikimedia_org as Resolved.

01:20 <+jinxer-wm> (ProbeDown) resolved: (7) Service miscweb1002:443 has failed probes (http_commons_query_wikimedia_org_ip4) -

Thu, Mar 30, 1:23 AM · Patch-For-Review, serviceops-collab
Dzahn closed T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb) as Resolved.
Thu, Mar 30, 1:23 AM · Patch-For-Review, serviceops-collab
Dzahn closed T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb), a subtask of T333510: clean up commons-query on miscweb setup, as Resolved.
Thu, Mar 30, 1:23 AM · serviceops-collab
Dzahn added a subtask for T333510: clean up commons-query on miscweb setup: T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).
Thu, Mar 30, 1:23 AM · serviceops-collab
Dzahn added a parent task for T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb): T333510: clean up commons-query on miscweb setup.
Thu, Mar 30, 1:23 AM · Patch-For-Review, serviceops-collab
Dzahn added a comment to T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).

finally everthing ok in https://thanos.wikimedia.org/graph?g0.deduplicate=1&g0.expr=probe_success%7Binstance%3D~%22.*miscweb.*%22%7D&g0.max_source_resolution=0s&g0.partial_response=0&g0.range_input=1h&g0.stacked=0&g0.store_matches=%5B%5D&g0.tab=1

Thu, Mar 30, 1:22 AM · Patch-For-Review, serviceops-collab
Dzahn added a comment to T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).

also see T333510

Thu, Mar 30, 1:21 AM · Patch-For-Review, serviceops-collab
Dzahn added a comment to T333515: ProbeDown - commons_query_wikimedia_org.

finally everything working per https://thanos.wikimedia.org/graph?g0.deduplicate=1&g0.expr=probe_success%7Binstance%3D~%22.*miscweb.*%22%7D&g0.max_source_resolution=0s&g0.partial_response=0&g0.range_input=1h&g0.stacked=0&g0.store_matches=%5B%5D&g0.tab=1

Thu, Mar 30, 1:20 AM · Patch-For-Review, serviceops-collab
Dzahn added a comment to T333515: ProbeDown - commons_query_wikimedia_org.

fixing with https://gerrit.wikimedia.org/r/c/operations/puppet/+/904384

Thu, Mar 30, 1:12 AM · Patch-For-Review, serviceops-collab
Dzahn renamed T333515: ProbeDown - commons_query_wikimedia_org from ProbeDown to ProbeDown - commons_query_wikimedia_org.
Thu, Mar 30, 1:12 AM · Patch-For-Review, serviceops-collab
Dzahn renamed T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb) from ProbeDown to ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).
Thu, Mar 30, 12:56 AM · Patch-For-Review, serviceops-collab
Dzahn added a comment to T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).

https://gerrit.wikimedia.org/r/c/operations/puppet/+/904374

Thu, Mar 30, 12:41 AM · Patch-For-Review, serviceops-collab
Dzahn added a comment to T333507: ProbeDown - integration.wikimedia.org (contint2002) - commons-query.wikimedia.org - transparency.wikimedia.org (miscweb).

filed T333510 for the root cause of this

Thu, Mar 30, 12:24 AM · Patch-For-Review, serviceops-collab
Dzahn updated the task description for T333510: clean up commons-query on miscweb setup.
Thu, Mar 30, 12:21 AM · serviceops-collab
Dzahn added a comment to T333510: clean up commons-query on miscweb setup.
host commons-query.wikimedia.org
commons-query.wikimedia.org is an alias for dyna.wikimedia.org.
dyna.wikimedia.org has address 208.80.154.224
dyna.wikimedia.org has IPv6 address 2620:0:861:ed1a::1
Thu, Mar 30, 12:20 AM · serviceops-collab
Dzahn created T333510: clean up commons-query on miscweb setup.
Thu, Mar 30, 12:18 AM · serviceops-collab

Wed, Mar 29

Dzahn added a comment to T333451: Phabricator tasks are not reliably indexed by search engines.

Overly strict robots.txt?

Wed, Mar 29, 11:18 PM · Security, Phabricator
Dzahn updated the task description for T327976: create blackbox::http monitoring for miscweb.
Wed, Mar 29, 7:08 PM · serviceops-collab
Dzahn updated the task description for T327976: create blackbox::http monitoring for miscweb.
Wed, Mar 29, 7:04 PM · serviceops-collab
Dzahn updated the task description for T327976: create blackbox::http monitoring for miscweb.
Wed, Mar 29, 6:34 PM · serviceops-collab
Dzahn updated the task description for T327976: create blackbox::http monitoring for miscweb.
Wed, Mar 29, 5:23 PM · serviceops-collab
Dzahn added a comment to T327976: create blackbox::http monitoring for miscweb.

these can all be seen together at:

Wed, Mar 29, 5:19 PM · serviceops-collab
Dzahn updated the task description for T327976: create blackbox::http monitoring for miscweb.
Wed, Mar 29, 5:19 PM · serviceops-collab
Dzahn changed the status of T327976: create blackbox::http monitoring for miscweb, a subtask of T329587: http::blackbox monitoring for all https services (let all serviceops-collab alertmanager alerts create tickets), from Open to In Progress.
Wed, Mar 29, 5:13 PM · Patch-For-Review, serviceops-collab
Dzahn changed the status of T327976: create blackbox::http monitoring for miscweb from Open to In Progress.
Wed, Mar 29, 5:13 PM · serviceops-collab
Dzahn updated the task description for T327976: create blackbox::http monitoring for miscweb.
Wed, Mar 29, 5:13 PM · serviceops-collab
Dzahn updated the task description for T329587: http::blackbox monitoring for all https services (let all serviceops-collab alertmanager alerts create tickets).
Wed, Mar 29, 5:11 PM · Patch-For-Review, serviceops-collab
Dzahn added a comment to T327976: create blackbox::http monitoring for miscweb.

multiple of these have been done in T329587 meanwhile

Wed, Mar 29, 5:04 PM · serviceops-collab
Dzahn added a comment to T319477: Migrate doc hosts to Bullseye.

Yes, the issue with mismatched UIDs on hosts that rsync has come up multiple times before and the preferred fix in SRE is definitely to use reserved UIDs. We have applied this to other hosts and it ends the problems once and for the future.

Wed, Mar 29, 4:22 PM · Patch-For-Review, Continuous-Integration-Infrastructure, serviceops-collab
Dzahn added a comment to T331896: upgrade miscweb VMs to bullseye.

Backups are now flowing ok, reverting ignore list.

Wed, Mar 29, 4:14 PM · Patch-For-Review, serviceops-collab
Dzahn updated the task description for T332379: Post-creation work for anpwiki.
Wed, Mar 29, 4:12 PM · Countervandalism-Network, Platform Engineering, Wiki-Setup
Dzahn updated subscribers of T332379: Post-creation work for anpwiki.

I did not get my own subtask to add new wikis to wikistats as usual in the past.

Wed, Mar 29, 4:12 PM · Countervandalism-Network, Platform Engineering, Wiki-Setup
Dzahn awarded T332869: Phabricator's access log may have some problems in log rotation a Mountain of Wealth token.
Wed, Mar 29, 4:05 PM · Patch-For-Review, Phabricator, serviceops-collab
Dzahn added a comment to T332918: determine whether https://iegreview.wikimedia.org can be sunset.

Given that this blocks getting off of "miscweb on buster" and "switch miscweb to codfw" I would prioritize it a bit higher.

Wed, Mar 29, 4:04 PM · Wikimedia-IEG-grant-review, serviceops-collab
Dzahn added a comment to T331831: Create Central Kurdish Wiktionary.

The wiki is live, any extra work should have its own ticket.

Wed, Mar 29, 4:01 PM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), MW-1.41-notes (1.41.0-wmf.1; 2023-03-20), Wiki-Setup (Create), User-Urbanecm
Dzahn added a comment to T333447: Repository by URL import from Gerrit to GitLab fails.

I have definitely imported a repo from Gerrit before, seems like this must have been introduced in a gitlab version upgrade.

Wed, Mar 29, 3:41 PM · serviceops-collab, GitLab
Dzahn added a comment to T332093: Post-creation work for ckbwiktionary.

The usual ticket to add a new wiki to wikistats seems to be notably missing but used to be automatic.

Wed, Mar 29, 12:26 AM · Wiki-Setup, User-Urbanecm, Countervandalism-Network, Platform Engineering

Tue, Mar 28

Dzahn updated the task description for T331901: check icinga for any alerts custom to sre-collab related services.
Tue, Mar 28, 11:58 PM · Patch-For-Review, Observability-Alerting, serviceops-collab
Dzahn updated the task description for T331901: check icinga for any alerts custom to sre-collab related services.
Tue, Mar 28, 11:37 PM · Patch-For-Review, Observability-Alerting, serviceops-collab
Dzahn changed the status of T333264: Add analytics-platform-eng-admins on stat* hosts from Resolved to Declined.
Tue, Mar 28, 5:37 PM · Data-Engineering
Dzahn added a comment to T305082: Request for Private repos to be enabled.

my best path right now is to download that notebook from the stat box I was working on and then either email it or DM it to them via Slack, which they will need to download to view.

Tue, Mar 28, 5:35 PM · Privacy Engineering, Release-Engineering-Team (Priority Backlog 📥), Privacy, User-brennen, GitLab (Administration, Settings & Policy), Product-Analytics
Dzahn added a comment to T331896: upgrade miscweb VMs to bullseye.

@jcrespo Oh, I noticed now before it was listening only on 127.0.0.1 but after it is listening on 0.0.0.0. Must be a race condition. Seems rare though. Thanks for confirming :)

Tue, Mar 28, 3:42 PM · Patch-For-Review, serviceops-collab
Dzahn added a comment to T331896: upgrade miscweb VMs to bullseye.

@jcrespo The bacula-fd service was running and listening on port 9102 but still refusing connections. Restarting the service fixed it though and now a connection from backup1001 to port 9102 on miscweb2003 works.

Tue, Mar 28, 3:16 PM · Patch-For-Review, serviceops-collab
valerio.bozzolan awarded T302528: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration a Love token.
Tue, Mar 28, 4:28 AM · serviceops-collab, Release-Engineering-Team (Seen), Upstream, Phabricator (Upstream)

Mon, Mar 27

Dzahn closed T333251: ProbeDown as Resolved.

23:47 < mutante> !log people1003 - taking down apache to provoke monitoring alert (inactive instances) and confirm IRC alerting change works

Mon, Mar 27, 11:58 PM · serviceops-collab
Dzahn added a comment to T331647: Grant Hal deployment rights.

Hi @Htriedman and @MoritzMuehlenhoff,

Mon, Mar 27, 11:42 PM · SRE, SRE-Access-Requests
Dzahn added a comment to T333135: Offboard nfraison.

We got the "widespread puppet failures" alert which made me look at some random failed hosts in the list. I found the reason was this offboarding, because:

Mon, Mar 27, 11:02 PM · SRE, Infrastructure-Foundations, Infrastructure Security
Dzahn assigned T332868: Grant Grafana access to babiola to Ladsgroup.
Mon, Mar 27, 7:46 PM · SRE, LDAP-Access-Requests
Dzahn changed the status of T332868: Grant Grafana access to babiola from Open to In Progress.
Mon, Mar 27, 7:46 PM · SRE, LDAP-Access-Requests
Dzahn added a comment to T332868: Grant Grafana access to babiola.

@larissagaulia Thank you for adding the information. Does "until July" mean "until last day of June"? I uploaded a code change above that is now in review.

Mon, Mar 27, 7:44 PM · SRE, LDAP-Access-Requests
Dzahn added a comment to T331680: Update footer links to direct to proper locations on Foundation Governance Wiki.

Thanks @Varnent no problem:)

Mon, Mar 27, 6:24 PM · MediaWiki-extensions-General, WMF-General-or-Unknown, Wikimedia Foundation Governance Wiki (foundation.wikimedia.org), MediaWiki-Internationalization
Dzahn added a project to T333212: replace dzahn as signer in pwstore: serviceops-collab.
Mon, Mar 27, 6:23 PM · Infrastructure Security, serviceops-collab
Dzahn created T333212: replace dzahn as signer in pwstore.
Mon, Mar 27, 6:23 PM · Infrastructure Security, serviceops-collab
Dzahn added a comment to T330797: Uninstall Differential.

fyi, regardless of how one feels about differential, there are already no more active repositories on Phabricator.

Mon, Mar 27, 5:51 PM · Phabricator
Dzahn added a comment to P43914 phorge install example.

The code that installs it is at modules/profile/manifests/phorge.pp in operations/puppet git repo

Mon, Mar 27, 5:00 PM
Dzahn added a comment to P43914 phorge install example.

https://phabricator.wikimedia.org/T328595

Mon, Mar 27, 4:59 PM