Page MenuHomePhabricator

jijiki (effie mouzeli)
is an animal

Projects (12)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Aug 14 2018, 10:50 AM (291 w, 6 d)
Availability
Available
IRC Nick
effie
LDAP User
Effie Mouzeli
MediaWiki User
EMouzeli (WMF) [ Global Accounts ]

Recent Activity

Wed, Mar 6

jijiki created T359375: make better use of spicerack's service_catalog().
Wed, Mar 6, 2:20 PM · Datacenter-Switchover, serviceops

Tue, Mar 5

jijiki created T359177: Migrate switchdc cookbooks to class-based API..
Tue, Mar 5, 2:13 PM · serviceops
jijiki updated the task description for T359154: Update DC switchover cookbooks to handle mw-jobrunners.
Tue, Mar 5, 11:33 AM · Datacenter-Switchover, serviceops
jijiki created T359154: Update DC switchover cookbooks to handle mw-jobrunners.
Tue, Mar 5, 11:32 AM · Datacenter-Switchover, serviceops
jijiki added a comment to T346690: mcrouter daemonset on mw-on-k8s.

mw-mcrouter ds has been deployed on staging mw-mcrouter staging

Tue, Mar 5, 9:17 AM · MediaWiki-Platform-Team (Radar), Patch-For-Review, serviceops, MW-on-K8s

Mon, Mar 4

jijiki added a comment to T358233: MoveComms support for Northward Datacentre Switchover (March 2024).

Looks alright!

Mon, Mar 4, 3:23 PM · User-notice, serviceops, MoveComms-Support
jijiki added a comment to T358233: MoveComms support for Northward Datacentre Switchover (March 2024).

@Trizek-WMF as per our off-phabricator discussion, the major change is that this is not a procedure we test anymore, but it has become standard practice. Please edit the message as you see fit to reflect that.

Mon, Mar 4, 3:08 PM · User-notice, serviceops, MoveComms-Support

Thu, Feb 29

jijiki changed the status of T346690: mcrouter daemonset on mw-on-k8s from Stalled to In Progress.
Thu, Feb 29, 4:26 PM · MediaWiki-Platform-Team (Radar), Patch-For-Review, serviceops, MW-on-K8s
jijiki changed the status of T346690: mcrouter daemonset on mw-on-k8s, a subtask of T277711: Memcached, mcrouter in MediaWiki on Kubernetes, from Stalled to In Progress.
Thu, Feb 29, 4:24 PM · serviceops, SRE

Tue, Feb 27

jijiki claimed T358597: urldownloader1003's network is unresponsive.
Tue, Feb 27, 4:51 PM · serviceops
jijiki updated subscribers of T358597: urldownloader1003's network is unresponsive.

Looking into the issue, we found that around 26th Feb @ ~21:45 UTC, the urldownloader1003 (ganeti VM running on ganeti1027 ie cluster master) lost network connectivity

Tue, Feb 27, 4:45 PM · serviceops
jijiki added a comment to T358595: Google and Yandex Translate are not available from cxserver.

This was most likely related to T358597

Tue, Feb 27, 4:33 PM · Unplanned-Sprint-Work, Language-Team (Language-2024-January-March), SectionTranslation, CX-cxserver
jijiki added a comment to T358597: urldownloader1003's network is unresponsive.

After issuing a restart, the VM came back to life normally.

Tue, Feb 27, 3:57 PM · serviceops
jijiki closed T358597: urldownloader1003's network is unresponsive as Resolved.
Tue, Feb 27, 3:56 PM · serviceops
jijiki triaged T358597: urldownloader1003's network is unresponsive as High priority.
Tue, Feb 27, 3:43 PM · serviceops
jijiki created T358597: urldownloader1003's network is unresponsive.
Tue, Feb 27, 3:43 PM · serviceops

Fri, Feb 23

jijiki updated the task description for T358286: SRE comms for Northward Datacentre Switchover (March 2024).
Fri, Feb 23, 11:55 AM · Datacenter-Switchover, serviceops

Thu, Feb 22

jijiki updated the task description for T358286: SRE comms for Northward Datacentre Switchover (March 2024).
Thu, Feb 22, 9:16 PM · Datacenter-Switchover, serviceops
jijiki created T358286: SRE comms for Northward Datacentre Switchover (March 2024).
Thu, Feb 22, 8:34 PM · Datacenter-Switchover, serviceops
jijiki updated the task description for T357547: ☂️ Northward Datacentre Switchover (March 2024) .
Thu, Feb 22, 3:43 PM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic
jijiki added a subtask for T357547: ☂️ Northward Datacentre Switchover (March 2024) : T358233: MoveComms support for Northward Datacentre Switchover (March 2024).
Thu, Feb 22, 3:42 PM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic
jijiki added a parent task for T358233: MoveComms support for Northward Datacentre Switchover (March 2024): T357547: ☂️ Northward Datacentre Switchover (March 2024) .
Thu, Feb 22, 3:42 PM · User-notice, serviceops, MoveComms-Support
jijiki created T358233: MoveComms support for Northward Datacentre Switchover (March 2024).
Thu, Feb 22, 3:34 PM · User-notice, serviceops, MoveComms-Support
jijiki renamed T357547: ☂️ Northward Datacentre Switchover (March 2024) from Northward Datacentre Switchover (March 2024) to ☂️ Northward Datacentre Switchover (March 2024) .
Thu, Feb 22, 3:05 PM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic

Wed, Feb 21

bd808 awarded T345868: Rename the shellbox service to shellbox-score a Like token.
Wed, Feb 21, 9:57 PM · Shellbox, serviceops

Feb 14 2024

jijiki updated the task description for T357547: ☂️ Northward Datacentre Switchover (March 2024) .
Feb 14 2024, 4:48 PM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic
jijiki created T357547: ☂️ Northward Datacentre Switchover (March 2024) .
Feb 14 2024, 4:36 PM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic
jijiki closed T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service as Resolved.

Things are progressing well after the last change, please reopen if this resurfaces. Shoutout to @akosiaris for lending a hand

Feb 14 2024, 3:52 PM · serviceops, iPoid-Service (iPoid 1.0)
jijiki updated the task description for T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.
Feb 14 2024, 1:42 PM · serviceops, iPoid-Service (iPoid 1.0)
jijiki added a comment to T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.
  • meanwhile, we restarted the envoyproxies, which seems to have significantly improved the issue; 503s for now are around 2 per hour

Hmm, I am not seeing that reflected here https://logstash.wikimedia.org/goto/d99b2beddeb28d71ddc74e7298c69cc8

image.png (596×890 px, 41 KB)

Feb 14 2024, 12:42 PM · serviceops, iPoid-Service (iPoid 1.0)

Feb 13 2024

jijiki updated the task description for T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.
Feb 13 2024, 3:10 PM · serviceops, iPoid-Service (iPoid 1.0)
jijiki added a comment to T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.
  • tcpdump shows that upstream sends an RST, but nothing else useful
Feb 13 2024, 2:19 PM · serviceops, iPoid-Service (iPoid 1.0)
jijiki added a comment to T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.

so far:

  • dumped traffic at the pod level (ipoid has only one pod, service is low traffic), and I never saw a packet from an appserver whic
  • from the pod's perspective, there are no 500x errors grafana envoy-telemetry-k8s
  • nothing standing out on lvs2013
  • nothing odd on the k8s node itself

Is there something more we could log from the MW side that would help debug this? Is it possible there is some special routing happening because the originating request to ipoid happens in a POST request context (so it always originates from the primary DC)?

Feb 13 2024, 11:10 AM · serviceops, iPoid-Service (iPoid 1.0)

Feb 12 2024

jijiki updated the task description for T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.
Feb 12 2024, 4:49 PM · serviceops, iPoid-Service (iPoid 1.0)
jijiki updated the task description for T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.
Feb 12 2024, 4:48 PM · serviceops, iPoid-Service (iPoid 1.0)
jijiki added a comment to T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.

so far:

Feb 12 2024, 4:34 PM · serviceops, iPoid-Service (iPoid 1.0)

Feb 8 2024

jijiki added a project to T355333: Possible firmware issues reimaging mw2282: ops-codfw.
Feb 8 2024, 10:07 AM · SRE, ops-codfw, DC-Ops, serviceops
jijiki added a comment to T355333: Possible firmware issues reimaging mw2282.

I set the host as inactive since I noticed a bit of log spam on lvs2013

Feb 8 2024, 10:04 AM · SRE, ops-codfw, DC-Ops, serviceops

Feb 7 2024

jijiki created T356885: Update app.job module in deployment-charts.
Feb 7 2024, 4:01 PM · serviceops
jijiki added a comment to T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.

I can see in the appserver logs

Feb 7 2024, 3:23 PM · serviceops, iPoid-Service (iPoid 1.0)

Feb 6 2024

jijiki added a comment to T356766: Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service.

We'll take a look and get back to you

Feb 6 2024, 4:55 PM · serviceops, iPoid-Service (iPoid 1.0)

Feb 1 2024

jijiki changed the status of T346690: mcrouter daemonset on mw-on-k8s, a subtask of T277711: Memcached, mcrouter in MediaWiki on Kubernetes, from Open to Stalled.
Feb 1 2024, 6:40 PM · serviceops, SRE
jijiki changed the status of T346690: mcrouter daemonset on mw-on-k8s from Open to Stalled.
Feb 1 2024, 6:40 PM · MediaWiki-Platform-Team (Radar), Patch-For-Review, serviceops, MW-on-K8s
jijiki moved T350807: Package latest version of prometheus-memcached-exporter (v0.14.1) from Incoming 🐫 to Doing 😎 on the serviceops board.
Feb 1 2024, 6:16 PM · serviceops
jijiki moved T352880: Memcache improvements and essential work (FY 23-24) from Backlog FY23-24 🚜 to Doing 😎 on the serviceops board.
Feb 1 2024, 6:15 PM · serviceops
jijiki moved T344324: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) from this.quarter 🍕 to Stalled 🐌 on the serviceops board.
Feb 1 2024, 6:11 PM · Content-Transform-Team, serviceops, Essential-Work, Wikimedia-Incident
jijiki changed the status of T344324: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) from In Progress to Stalled.

Stalled until T356412 is picked up by Data-Persistence

Feb 1 2024, 6:09 PM · Content-Transform-Team, serviceops, Essential-Work, Wikimedia-Incident
jijiki changed the status of T344324: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023), a subtask of T343987: Switch thanos-fe to cfssl, from In Progress to Stalled.
Feb 1 2024, 6:09 PM · Observability-Metrics
jijiki added a subtask for T344324: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023): T356412: Consolidate TLS cert puppetry for ms and thanos swift frontends.
Feb 1 2024, 6:08 PM · Content-Transform-Team, serviceops, Essential-Work, Wikimedia-Incident
jijiki added a parent task for T356412: Consolidate TLS cert puppetry for ms and thanos swift frontends: T344324: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023).
Feb 1 2024, 6:08 PM · SRE, SRE-swift-storage

Jan 31 2024

jijiki added a comment to T346690: mcrouter daemonset on mw-on-k8s.

I think we can also set it in the php-fpm pool conf like

env[MCROUTER_SERVER] = $MCROUTER_SERVER

and set $MCROUTER_SERVER through kubernetes?

Jan 31 2024, 11:44 AM · MediaWiki-Platform-Team (Radar), Patch-For-Review, serviceops, MW-on-K8s

Jan 29 2024

jijiki added a project to T355237: Update cache.mrouter modules in deployment-charts: serviceops.
Jan 29 2024, 11:16 AM · serviceops
jijiki closed T355237: Update cache.mrouter modules in deployment-charts as Resolved.
Jan 29 2024, 11:15 AM · serviceops

Jan 17 2024

jijiki created T355237: Update cache.mrouter modules in deployment-charts.
Jan 17 2024, 3:34 PM · serviceops

Jan 11 2024

jijiki closed T354768: Unable to deploy ipoid to staging or eqiad as Resolved.

We merged a fix to unblock this, your are good to go.

Jan 11 2024, 9:47 AM · serviceops, iPoid-Service

Jan 10 2024

jijiki added a comment to T354768: Unable to deploy ipoid to staging or eqiad.

We are working on updating the underline template which is causing the errors you are seeing, I will let you know when it is properly fixed. Sorry for the inconvenience!

Jan 10 2024, 4:32 PM · serviceops, iPoid-Service

Jan 9 2024

jijiki moved T352885: Enable extstore to a subset of memcached servers (experiment) from Incoming 🐫 to this.quarter 🍕 on the serviceops board.
Jan 9 2024, 9:52 AM · serviceops
jijiki moved T348284: Handle sidecar containers in one-off Kubernetes jobs from Incoming 🐫 to Doing 😎 on the serviceops board.
Jan 9 2024, 9:51 AM · MW-on-K8s, serviceops
jijiki moved T350767: Rebuild PHP 7.4 packages for Bullseye from Incoming 🐫 to this.quarter 🍕 on the serviceops board.
Jan 9 2024, 9:51 AM · SRE, serviceops
jijiki moved T352880: Memcache improvements and essential work (FY 23-24) from Incoming 🐫 to Backlog FY23-24 🚜 on the serviceops board.
Jan 9 2024, 9:51 AM · serviceops
jijiki moved T349619: Migrate roles to puppet7 from Incoming 🐫 to Doing 😎 on the serviceops board.
Jan 9 2024, 9:50 AM · Patch-For-Review, Data-Platform-SRE (2024.03.04 - 2024.03.24), serviceops, collaboration-services, SRE-tools, Puppet-Core, Puppet (Puppet 7.0), Infrastructure-Foundations, SRE
jijiki moved T352891: Upgrade memcache and memcached gutter pools to Bookworm from Incoming 🐫 to this.quarter 🍕 on the serviceops board.
Jan 9 2024, 9:50 AM · serviceops

Dec 15 2023

jijiki updated the task description for T349619: Migrate roles to puppet7.
Dec 15 2023, 3:34 PM · Patch-For-Review, Data-Platform-SRE (2024.03.04 - 2024.03.24), serviceops, collaboration-services, SRE-tools, Puppet-Core, Puppet (Puppet 7.0), Infrastructure-Foundations, SRE
jijiki added a project to T349619: Migrate roles to puppet7: serviceops.
Dec 15 2023, 3:32 PM · Patch-For-Review, Data-Platform-SRE (2024.03.04 - 2024.03.24), serviceops, collaboration-services, SRE-tools, Puppet-Core, Puppet (Puppet 7.0), Infrastructure-Foundations, SRE
jijiki added a comment to T349619: Migrate roles to puppet7.

Prep work for memcached hosts is in place; those hosts are using each host's puppet certs for TLS, and migrating to puppet7 needs a minor tweak due to the puppet 7 ca cert not being present on puppet 5 hosts. Hosts will be migrated to puppet 7 in January.

Dec 15 2023, 3:23 PM · Patch-For-Review, Data-Platform-SRE (2024.03.04 - 2024.03.24), serviceops, collaboration-services, SRE-tools, Puppet-Core, Puppet (Puppet 7.0), Infrastructure-Foundations, SRE
jijiki added a comment to T350807: Package latest version of prometheus-memcached-exporter (v0.14.1).

Started putting together a debian package on https://gitlab.wikimedia.org/repos/sre/prometheus-memcached-exporter

Dec 15 2023, 3:18 PM · serviceops
jijiki added a subtask for T352880: Memcache improvements and essential work (FY 23-24): T353511: Migrate memcached servers to PKI.
Dec 15 2023, 10:24 AM · serviceops
jijiki added a parent task for T353511: Migrate memcached servers to PKI: T352880: Memcache improvements and essential work (FY 23-24).
Dec 15 2023, 10:24 AM · serviceops
jijiki created T353511: Migrate memcached servers to PKI.
Dec 15 2023, 10:23 AM · serviceops

Dec 11 2023

jijiki closed T351387: Requesting access to wmf and analytics-privatedata-users for EHughes (superset access with no server access) as Resolved.

done:)

Dec 11 2023, 8:55 AM · SRE, SRE-Access-Requests
jijiki closed T352684: Grant Access to ldap/wmf for mcastro-wmf as Resolved.
Dec 11 2023, 8:55 AM · SRE, LDAP-Access-Requests
jijiki added a comment to T352684: Grant Access to ldap/wmf for mcastro-wmf.

@Mcastro done, please reopen if something is not right

Dec 11 2023, 8:54 AM · SRE, LDAP-Access-Requests

Dec 8 2023

jijiki closed T352475: Grant Access to archiva-deployers for pfischer as Resolved.

User is present in archiva-deployers LDAP group, please reopen if there is anything else

Dec 8 2023, 4:14 PM · SRE, LDAP-Access-Requests

Dec 7 2023

jijiki added a member for WMF-NDA: Mcastro.
Dec 7 2023, 5:03 PM
jijiki reassigned T351387: Requesting access to wmf and analytics-privatedata-users for EHughes (superset access with no server access) from eoghan to ehughes.
Dec 7 2023, 4:54 PM · SRE, SRE-Access-Requests
jijiki edited projects for T246062: Install private instance of gnomon for greater SRE team, added: Infrastructure-Foundations; removed SRE.
Dec 7 2023, 4:53 PM · Infrastructure-Foundations, Patch-Needs-Improvement
jijiki added a comment to T351387: Requesting access to wmf and analytics-privatedata-users for EHughes (superset access with no server access).

@ehughes please sign the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document so we can proceed

Dec 7 2023, 4:17 PM · SRE, SRE-Access-Requests
jijiki triaged T352902: enable lists.wikimedia.org or wikimedia.org email addresses to receive dmarc reports for *.wmflabs.org as Medium priority.
Dec 7 2023, 4:11 PM · Moderator-Tools-Team (Kanban), Cloud-VPS
jijiki edited projects for T351570: molly-guard does not apply to `systemctl reboot`, added: Infrastructure-Foundations; removed SRE.
Dec 7 2023, 4:09 PM · Infrastructure-Foundations, Upstream
jijiki removed a project from T340068: Regular expression "х[ÿý]и" matches "х и" in Abusefilter: SRE.

Are there any actionables here for serviceops ?

Dec 7 2023, 4:08 PM · PHP 7.4 support, serviceops, AbuseFilter

Dec 6 2023

jijiki renamed T350807: Package latest version of prometheus-memcached-exporter (v0.14.1) from Package latest version of prometheus-memcached-exporter to Package latest version of prometheus-memcached-exporter (v0.14.1).
Dec 6 2023, 4:57 PM · serviceops
jijiki added a subtask for T352885: Enable extstore to a subset of memcached servers (experiment): T273950: Modernise memcached systemd unit / sync to current buster setup.
Dec 6 2023, 4:57 PM · serviceops
jijiki added a parent task for T273950: Modernise memcached systemd unit / sync to current buster setup: T352885: Enable extstore to a subset of memcached servers (experiment).
Dec 6 2023, 4:57 PM · serviceops, User-jijiki, SRE
jijiki added a parent task for T352891: Upgrade memcache and memcached gutter pools to Bookworm: T352880: Memcache improvements and essential work (FY 23-24).
Dec 6 2023, 4:56 PM · serviceops
jijiki added a subtask for T352880: Memcache improvements and essential work (FY 23-24): T352891: Upgrade memcache and memcached gutter pools to Bookworm.
Dec 6 2023, 4:56 PM · serviceops
jijiki created T352891: Upgrade memcache and memcached gutter pools to Bookworm.
Dec 6 2023, 4:55 PM · serviceops
jijiki added a subtask for T352880: Memcache improvements and essential work (FY 23-24): T352885: Enable extstore to a subset of memcached servers (experiment).
Dec 6 2023, 4:44 PM · serviceops
jijiki added a parent task for T352885: Enable extstore to a subset of memcached servers (experiment): T352880: Memcache improvements and essential work (FY 23-24).
Dec 6 2023, 4:44 PM · serviceops
jijiki updated the task description for T350807: Package latest version of prometheus-memcached-exporter (v0.14.1).
Dec 6 2023, 4:44 PM · serviceops
jijiki added a parent task for T350807: Package latest version of prometheus-memcached-exporter (v0.14.1): T352885: Enable extstore to a subset of memcached servers (experiment).
Dec 6 2023, 4:43 PM · serviceops
jijiki added a subtask for T352885: Enable extstore to a subset of memcached servers (experiment): T350807: Package latest version of prometheus-memcached-exporter (v0.14.1).
Dec 6 2023, 4:43 PM · serviceops
jijiki updated the task description for T352885: Enable extstore to a subset of memcached servers (experiment).
Dec 6 2023, 4:43 PM · serviceops
jijiki triaged T352885: Enable extstore to a subset of memcached servers (experiment) as Medium priority.
Dec 6 2023, 4:40 PM · serviceops
jijiki triaged T352880: Memcache improvements and essential work (FY 23-24) as Medium priority.
Dec 6 2023, 4:14 PM · serviceops
jijiki closed T73466: Use memcached by default as Resolved.

boldly closing this

Dec 6 2023, 4:11 PM · MediaWiki-Vagrant

Dec 1 2023

jijiki added a comment to T346690: mcrouter daemonset on mw-on-k8s.

Most of the patches are ready to go, the only thing missing is passing the variable to php-fpm. Our options here are either

Dec 1 2023, 3:26 PM · MediaWiki-Platform-Team (Radar), Patch-For-Review, serviceops, MW-on-K8s
jijiki added a comment to T346690: mcrouter daemonset on mw-on-k8s.

(update): The config change got merged and deployed yesterday. So it's live now. Is there anything else needed from MediaWiki Platform Team side on this ticket?

If nothing, then we can re-assign as needed.

Dec 1 2023, 3:23 PM · MediaWiki-Platform-Team (Radar), Patch-For-Review, serviceops, MW-on-K8s

Nov 28 2023

jijiki closed T352156: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. as Resolved.

Files copied over :)

Nov 28 2023, 3:30 PM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), LandingCheck, Patch-For-Review, MediaWiki-extensions-WikimediaEvents, Release-Engineering-Team, MW-on-K8s, serviceops, Wikimedia-production-error
jijiki updated subscribers of T352156: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat..

@jbond @Muehlenhoff It appears that we are missing GeoIP.dat, GeoIPRegion.dat, and GeoIPCity.dat from the new puppetmasters. Would it be alright if we'd just copied them over from the old puppetmasters for now?

Nov 28 2023, 12:14 PM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), LandingCheck, Patch-For-Review, MediaWiki-extensions-WikimediaEvents, Release-Engineering-Team, MW-on-K8s, serviceops, Wikimedia-production-error

Nov 13 2023

jijiki added a project to T350903: Rename FEED_API_KEY: serviceops.
Nov 13 2023, 11:07 AM · serviceops, Puppet, iPoid-Service