Page MenuHomePhabricator

jijiki (effie mouzeli)
is an animal

Projects (11)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Aug 14 2018, 10:50 AM (240 w, 3 d)
Availability
Available
IRC Nick
effie
LDAP User
Effie Mouzeli
MediaWiki User
EMouzeli (WMF) [ Global Accounts ]

Recent Activity

Today

jijiki added a comment to T333019: Replace Nutcracker.

Thumbor is using nutcracker for memcached sharding, thus we can happily use mrouter there :)

Fri, Mar 24, 7:11 PM · Platform Team Workboards (Platform Engineering Reliability), serviceops

Wed, Mar 22

jijiki closed T285328: Migrate OSM sync alerts from icinga to AlertManager as Resolved.
Wed, Mar 22, 3:21 PM · observability, Maps
jijiki closed T285328: Migrate OSM sync alerts from icinga to AlertManager, a subtask of T288622: All Prometheus based alerts move from Icinga to alert manager exclusively, as Resolved.
Wed, Mar 22, 3:21 PM · SRE Observability (FY2022/2023-Q3)

Tue, Mar 21

jijiki claimed T324959: Scrape controller-manager and scheduler metrics.
Tue, Mar 21, 9:26 AM · Kubernetes, Prod-Kubernetes, serviceops

Mon, Mar 20

jijiki closed T314472: Re-import full planet data into eqiad and codfw as Resolved.

Closing, I will add the URL of the relevant documentation when I finish writing it

Mon, Mar 20, 11:32 AM · serviceops, Maps
jijiki closed T314472: Re-import full planet data into eqiad and codfw, a subtask of T316365: Restore the map data health and parity between clusters, as Resolved.
Mon, Mar 20, 11:32 AM · WMDE-TechWish-Sprint-2023-01-18, WMDE-TechWish-Sprint-2023-01-04, WMDE-TechWish-Sprint-2022-11-09, WMDE-TechWish-Sprint-2022-10-26, WMDE-TechWish-Sprint-2022-09-14, WMDE-GeoInfo-FocusArea, WMDE-TechWish-Sprint-2022-08-31, Maps (Maps-data)
jijiki closed T314472: Re-import full planet data into eqiad and codfw, a subtask of T323920: Deploy recent map server work to production ( Jan 2023 ), as Resolved.
Mon, Mar 20, 11:32 AM · WMDE-TechWish-Sprint-2023-01-18, Maps (Kartotherian), WMDE-GeoInfo-FocusArea

Mon, Mar 13

jijiki moved T329827: Add a second control-plane to wikikube staging clusters from Doing 😎 to Backlog FY22-23 🚜 on the serviceops board.
Mon, Mar 13, 8:19 AM · Patch-For-Review, Kubernetes, serviceops

Wed, Mar 8

jijiki closed T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye, a subtask of T291916: Tracking task for Bullseye migrations in production, as Resolved.
Wed, Mar 8, 12:40 PM · Epic, Infrastructure-Foundations, SRE
jijiki closed T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye as Resolved.
Wed, Mar 8, 12:40 PM · serviceops
jijiki claimed T329940: eqiad and codfw: 1 VM each requested for wikikube-staging.
Wed, Mar 8, 9:59 AM · SRE, vm-requests
jijiki claimed T329827: Add a second control-plane to wikikube staging clusters.
Wed, Mar 8, 9:59 AM · Patch-For-Review, Kubernetes, serviceops

Mon, Mar 6

jijiki claimed T313874: kubernetes102[34] implemetation tracking.
Mon, Mar 6, 11:17 AM · SRE, serviceops

Feb 20 2023

jijiki moved T330072: Reset management module of mc1039 from Incoming 🐫 to Doing 😎 on the serviceops board.
Feb 20 2023, 12:24 PM · SRE, serviceops, DC-Ops, ops-eqiad
jijiki added a subtask for T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye: T330072: Reset management module of mc1039.
Feb 20 2023, 12:23 PM · serviceops
jijiki added a parent task for T330072: Reset management module of mc1039: T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye.
Feb 20 2023, 12:23 PM · SRE, serviceops, DC-Ops, ops-eqiad
jijiki created T330072: Reset management module of mc1039.
Feb 20 2023, 12:19 PM · SRE, serviceops, DC-Ops, ops-eqiad

Feb 14 2023

jijiki awarded T329323: Update iDRAC and NIC firmware on mc-gp* hosts a Love token.
Feb 14 2023, 10:47 PM · SRE, serviceops, ops-codfw, ops-eqiad, DC-Ops

Feb 9 2023

jijiki added a subtask for T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye: T329323: Update iDRAC and NIC firmware on mc-gp* hosts .
Feb 9 2023, 9:17 PM · serviceops
jijiki added a parent task for T329323: Update iDRAC and NIC firmware on mc-gp* hosts : T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye.
Feb 9 2023, 9:17 PM · SRE, serviceops, ops-codfw, ops-eqiad, DC-Ops
jijiki created T329323: Update iDRAC and NIC firmware on mc-gp* hosts .
Feb 9 2023, 6:32 PM · SRE, serviceops, ops-codfw, ops-eqiad, DC-Ops
jijiki updated the task description for T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye.
Feb 9 2023, 5:36 PM · serviceops

Feb 7 2023

jijiki renamed T328768: Wikitech issues for datacentre switchover (March 2023) from Check wikitech switchover from labweb eqiad to Wikitech issues for datacentre switchover (March 2023).
Feb 7 2023, 10:25 AM · wikitech.wikimedia.org, cloud-services-team, Data-Persistence, serviceops, Datacenter-Switchover, SRE

Jan 30 2023

jijiki closed T277183: Phase out nutcracker from mediawiki servers as Resolved.

This work is done

Jan 30 2023, 10:29 AM · Patch-For-Review, Performance-Team (Radar), SRE, serviceops
jijiki closed T277183: Phase out nutcracker from mediawiki servers, a subtask of T267581: Phase out "redis_sessions" cluster and away from memcached cluster, as Resolved.
Jan 30 2023, 10:28 AM · Patch-For-Review, Performance-Team (Radar), Platform Engineering, serviceops, SRE

Jan 26 2023

jijiki awarded T306865: Missing lakes in maps (again) a Baby Tequila token.
Jan 26 2023, 1:31 PM · Product-Infrastructure-Team-Backlog-Deprecated, Maps

Jan 25 2023

WMDE-Fisch awarded T314472: Re-import full planet data into eqiad and codfw a Love token.
Jan 25 2023, 8:37 AM · serviceops, Maps

Jan 24 2023

jijiki added a comment to T314472: Re-import full planet data into eqiad and codfw.

We are happily serving maps from codfw, and both datacentres are up to date 🎉

Jan 24 2023, 4:19 PM · serviceops, Maps
jijiki updated the task description for T314472: Re-import full planet data into eqiad and codfw.
Jan 24 2023, 3:26 PM · serviceops, Maps

Jan 23 2023

jijiki moved T326544: High average POST latency for mw requests on api_appserver in codfw on alert1001 from Incoming 🐫 to this.quarter 🍕 on the serviceops board.
Jan 23 2023, 4:39 PM · SRE, SRE Observability, Icinga, serviceops
jijiki triaged T327513: Upgrade maps servers to bullseye as Medium priority.
Jan 23 2023, 4:38 PM · serviceops, Maps
jijiki moved T327513: Upgrade maps servers to bullseye from Doing 😎 to Backlog FY22-23 🚜 on the serviceops board.
Jan 23 2023, 4:38 PM · serviceops, Maps
jijiki moved T327513: Upgrade maps servers to bullseye from Incoming 🐫 to Doing 😎 on the serviceops board.
Jan 23 2023, 4:38 PM · serviceops, Maps
jijiki moved T327663: Create a visual representation of where each service is active from, any given time from Incoming 🐫 to Backlog FY22-23 🚜 on the serviceops board.
Jan 23 2023, 4:34 PM · serviceops, observability
jijiki moved T327665: Create a cookbook to help us depool *all* services from a datacentre from Incoming 🐫 to Backlog FY22-23 🚜 on the serviceops board.
Jan 23 2023, 4:33 PM · serviceops, Infrastructure-Foundations
jijiki created T327665: Create a cookbook to help us depool *all* services from a datacentre.
Jan 23 2023, 3:58 PM · serviceops, Infrastructure-Foundations
jijiki created T327663: Create a visual representation of where each service is active from, any given time.
Jan 23 2023, 3:42 PM · serviceops, observability
jijiki updated the task description for T314472: Re-import full planet data into eqiad and codfw.
Jan 23 2023, 3:32 PM · serviceops, Maps
jijiki added a comment to T314472: Re-import full planet data into eqiad and codfw.

Import to codfw has been completed, and we have bootstrapped its tile storage using https://gerrit.wikimedia.org/r/c/operations/puppet/+/875973

Jan 23 2023, 1:07 PM · serviceops, Maps

Jan 17 2023

jijiki updated the task description for T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye.
Jan 17 2023, 7:04 PM · serviceops
jijiki updated the task description for T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye.
Jan 17 2023, 7:04 PM · serviceops

Jan 16 2023

jijiki moved T326252: k8s deployment-charts mesh module should allow use of mesh without public_port Service from Incoming 🐫 to this.quarter 🍕 on the serviceops board.
Jan 16 2023, 6:09 PM · Event-Platform Value Stream, Data-Engineering-Planning, serviceops
jijiki moved T159687: etcd switchover/enhancements from Incoming 🐫 to 🛎 Services & Oids on the serviceops board.
Jan 16 2023, 6:07 PM · serviceops, User-Joe, SRE
jijiki moved T326617: Decide on new Pod and Sevice IPv4 ranges for wikikube clusters from Incoming 🐫 to Backlog FY22-23 🚜 on the serviceops board.
Jan 16 2023, 6:06 PM · Kubernetes, Prod-Kubernetes, serviceops
jijiki moved T326729: Remove the .Values.kubernetesApi hack from Incoming 🐫 to Doing 😎 on the serviceops board.
Jan 16 2023, 6:05 PM · Kubernetes, Prod-Kubernetes, serviceops
jijiki moved T325890: Sudden increase in shellbox-syntaxhighlighing requests lead to api_appservers running out of idle workers from Incoming 🐫 to Backlog FY22-23 🚜 on the serviceops board.
Jan 16 2023, 5:56 PM · serviceops, Shellbox
jijiki moved T312722: Thumbor units failing / service general slowness from Incoming 🐫 to 🛎 Services & Oids on the serviceops board.
Jan 16 2023, 5:55 PM · serviceops, SRE, Thumbor
jijiki moved T275319: Raise limit of $wgMaxArticleSize for Hebrew Wikisource from Incoming 🐫 to 🌻Mediawiki on the serviceops board.
Jan 16 2023, 5:53 PM · serviceops, Performance-Team (Radar), SRE, Wikimedia-Site-requests
jijiki moved T242500: www.wikipedia.org/robots.txt should not be a redirect from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.
Jan 16 2023, 5:53 PM · serviceops, Regression, SRE, Wikimedia-Portals
jijiki moved T288629: Automated validation of mediawiki-multiversion images from Incoming 🐫 to 🌻Mediawiki on the serviceops board.
Jan 16 2023, 5:39 PM · serviceops, Release-Engineering-Team (Priority Backlog 📥), Patch-For-Review, SRE, MW-on-K8s
jijiki moved T308893: Increase $wgMaxArticleSize to 4MB for ruwikisource from Incoming 🐫 to 🌻Mediawiki on the serviceops board.
Jan 16 2023, 5:38 PM · serviceops, Performance-Team (Radar), SRE, Wikimedia-Site-requests, Russian-Sites
jijiki added a comment to T313733: Decommission mc20[19-27] and mc20[29-37].

@Papaul, this is my bad, thank you for taking care of Netbox (or the gentle soul that did so).

Jan 16 2023, 4:18 PM · ops-codfw, SRE, DC-Ops

Jan 12 2023

jijiki placed T313733: Decommission mc20[19-27] and mc20[29-37] up for grabs.
Jan 12 2023, 6:38 PM · ops-codfw, SRE, DC-Ops
jijiki added a project to T326834: hw troubleshooting: DIMM_B2 for mc2040.codfw.wmnet: ops-eqiad.
Jan 12 2023, 6:03 PM · SRE, ops-codfw, DC-Ops
jijiki renamed T277711: Memcached, mcrouter in MediaWiki on Kubernetes from Memcached, mcrouter, nutcracker's future in MediaWiki on Kubernetes to Memcached, mcrouter in MediaWiki on Kubernetes.
Jan 12 2023, 6:03 PM · serviceops, SRE
jijiki updated the task description for T314472: Re-import full planet data into eqiad and codfw.
Jan 12 2023, 2:52 PM · serviceops, Maps
jijiki added a comment to T314472: Re-import full planet data into eqiad and codfw.

Import to eqiad has been completed and traffic is being served via eqiad.

Jan 12 2023, 2:52 PM · serviceops, Maps
jijiki added a comment to T228970: Test memsniff as possible replacement of memkeys.

For the time being, we have packaged memkeys for bullseye so not to block T293216

Jan 12 2023, 9:13 AM · User-Elukey, SRE, serviceops
jijiki updated the task description for T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye.
Jan 12 2023, 9:12 AM · serviceops

Jan 10 2023

jijiki triaged T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye as Medium priority.
Jan 10 2023, 9:21 PM · serviceops
jijiki updated the task description for T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye.
Jan 10 2023, 9:19 PM · serviceops
jijiki moved T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye from Backlog FY22-23 🚜 to Doing 😎 on the serviceops board.
Jan 10 2023, 8:55 PM · serviceops
jijiki closed T244852: Upgrade and improve our application object caching service (memcached) as Resolved.
Jan 10 2023, 8:53 PM · Sustainability (Incident Followup), Performance-Team (Radar), SRE, serviceops
jijiki closed T270315: Upgrade memcached to version 1.6.x as Resolved.

Bluntly closing this as we are moving to mediawiki to kubernetes

Jan 10 2023, 8:52 PM · Patch-For-Review, User-jijiki, SRE, serviceops
jijiki closed T270315: Upgrade memcached to version 1.6.x, a subtask of T244852: Upgrade and improve our application object caching service (memcached), as Resolved.
Jan 10 2023, 8:51 PM · Sustainability (Incident Followup), Performance-Team (Radar), SRE, serviceops
jijiki closed T270315: Upgrade memcached to version 1.6.x, a subtask of T264604: Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe, as Resolved.
Jan 10 2023, 8:51 PM · MW-1.36-notes, MW-1.37-notes (1.37.0-wmf.1; 2021-04-13), Patch-For-Review, User-jijiki, SRE, serviceops, Performance-Team
jijiki closed T270315: Upgrade memcached to version 1.6.x, a subtask of T271967: Enable TLS on memcached for cross-dc replication, as Resolved.
Jan 10 2023, 8:51 PM · Patch-For-Review, Performance-Team (Radar), User-jijiki, SRE, serviceops
jijiki reassigned T313733: Decommission mc20[19-27] and mc20[29-37] from jijiki to Jclark-ctr.
Jan 10 2023, 7:58 PM · ops-codfw, SRE, DC-Ops
jijiki updated subscribers of T313733: Decommission mc20[19-27] and mc20[29-37].

@Jclark-ctr please note that mc2020 and mc2021 are probably still bootable due to a failure during running the decomm script

Jan 10 2023, 7:57 PM · ops-codfw, SRE, DC-Ops
jijiki renamed T313733: Decommission mc20[19-27] and mc20[29-37] from Decommission mc2019-mc2037 to Decommission mc20[19-27] and mc20[29-37].
Jan 10 2023, 7:52 PM · ops-codfw, SRE, DC-Ops
jijiki moved T198901: Migrate production services to kubernetes using the pipeline from Incoming 🐫 to 🗄 Projects on the serviceops board.
Jan 10 2023, 1:36 PM · serviceops, Release-Engineering-Team (Seen), Platform Team Legacy (Watching / External), Epic, Services (watching), SRE, Release Pipeline

Jan 9 2023

jijiki closed T258779: Roll out remote-DC gutter pool for /*/mw-wan/, a subtask of T244852: Upgrade and improve our application object caching service (memcached), as Resolved.
Jan 9 2023, 6:02 PM · Sustainability (Incident Followup), Performance-Team (Radar), SRE, serviceops
jijiki closed T258779: Roll out remote-DC gutter pool for /*/mw-wan/ as Resolved.
Jan 9 2023, 6:02 PM · Performance-Team (Radar), User-jijiki, serviceops

Jan 3 2023

jijiki added a comment to T325293: OSM import fails on both eqiad/codfw because of wrong data input.

I will explore our option to upgrade to 0.11.1 and get back to you

Jan 3 2023, 4:18 PM · Content-Transform-Team-WIP, serviceops, Maps

Dec 19 2022

jijiki awarded T325244: cloudweb hosts are using the profile::mediawiki::nutcracker profile to configure nutcracker a Love token.
Dec 19 2022, 10:19 AM · cloud-services-team (Kanban), wikitech.wikimedia.org, serviceops
jijiki closed T267581: Phase out "redis_sessions" cluster and away from memcached cluster as Resolved.
Dec 19 2022, 10:16 AM · Patch-For-Review, Performance-Team (Radar), Platform Engineering, serviceops, SRE

Dec 16 2022

jijiki moved T313733: Decommission mc20[19-27] and mc20[29-37] from 🛠 Upgrades and Hardware to Doing 😎 on the serviceops board.
Dec 16 2022, 11:30 AM · ops-codfw, SRE, DC-Ops
jijiki renamed T313733: Decommission mc20[19-27] and mc20[29-37] from Decommission mc2019-mc2036 to Decommission mc2019-mc2037.
Dec 16 2022, 11:29 AM · ops-codfw, SRE, DC-Ops
jijiki moved T325293: OSM import fails on both eqiad/codfw because of wrong data input from Incoming 🐫 to Doing 😎 on the serviceops board.
Dec 16 2022, 9:52 AM · Content-Transform-Team-WIP, serviceops, Maps
jijiki moved T325243: Evaluate out redis_misc cluster from Incoming 🐫 to Backlog FY22-23 🚜 on the serviceops board.
Dec 16 2022, 9:52 AM · serviceops

Dec 14 2022

jijiki added a comment to T325244: cloudweb hosts are using the profile::mediawiki::nutcracker profile to configure nutcracker .

That sounds alright, but if wikitech is still using redis for sessions (via nutcracker), then the problem remains. I propose the two profiles (profile::mediawiki::nutcracker and profile::openstack::base::nutcracker) to be merged to one under profile::openstack::base::nutcracker, which will unblock T277183

Dec 14 2022, 9:50 PM · cloud-services-team (Kanban), wikitech.wikimedia.org, serviceops
jijiki created T325244: cloudweb hosts are using the profile::mediawiki::nutcracker profile to configure nutcracker .
Dec 14 2022, 9:00 PM · cloud-services-team (Kanban), wikitech.wikimedia.org, serviceops
jijiki triaged T325243: Evaluate out redis_misc cluster as Low priority.
Dec 14 2022, 8:50 PM · serviceops
jijiki created T325243: Evaluate out redis_misc cluster.
Dec 14 2022, 8:50 PM · serviceops
jijiki updated the task description for T293216: Upgrade mc* and mc-gp* hosts to Debian Bullseye.
Dec 14 2022, 5:38 PM · serviceops
jijiki closed T293012: Productionise mc20[38-55] as Resolved.

All hosts are in production.

Dec 14 2022, 5:31 PM · Patch-For-Review, serviceops
jijiki closed T194997: Track more detailed disk usage on maps servers as Resolved.

Given that this task was opened when the infra was completely different, I am bluntly closing this task. I am happy to re-open if/when we believe there are reasons to do anything more than rely in the default disk usage monitors we already have for all hosts

Dec 14 2022, 3:21 PM · serviceops, Maps (Maps-data), SRE
jijiki closed T194997: Track more detailed disk usage on maps servers, a subtask of T268741: [Maps] Provide efficient monitoring capabilities to support maps, as Resolved.
Dec 14 2022, 3:21 PM · Maps, Product Infrastructure Roadmap, Epic, Product-Infrastructure-Team-Backlog-Deprecated

Dec 13 2022

jijiki edited projects for T279013: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution", added: serviceops-collab; removed serviceops.
Dec 13 2022, 4:08 PM · serviceops-collab, User-brennen, Phabricator
jijiki moved T274034: nodejs can't connect to mysqld via tcp/localhost any longer (was: mariadb failing on testreduce1001) from 🙈🙉🙊Backlog to 🍦IceBox on the serviceops board.
Dec 13 2022, 4:07 PM · Data-Persistence (work done), Parsoid (Tracking), serviceops
jijiki added a comment to T274034: nodejs can't connect to mysqld via tcp/localhost any longer (was: mariadb failing on testreduce1001).

@ssastry do you think we could close this task?

Dec 13 2022, 4:07 PM · Data-Persistence (work done), Parsoid (Tracking), serviceops
jijiki moved T268427: Make Shellbox actually do streaming from 🙈🙉🙊Backlog to 🍦IceBox on the serviceops board.
Dec 13 2022, 4:03 PM · Shellbox, Platform Team Workboards (Purple), MW-on-K8s, serviceops, SRE
jijiki moved T291620: Better observability/visualization for MediaWiki jobs from 🍦IceBox to serviceops-radar on the serviceops board.
Dec 13 2022, 4:02 PM · serviceops-radar, Platform Team Workboards (Platform Engineering Reliability), Data-Engineering, Wikibase change dispatching scripts to jobs
jijiki moved T314240: Enable rolling restart for all MW servers (tracking) from 🙈🙉🙊Backlog to serviceops-radar on the serviceops board.
Dec 13 2022, 4:02 PM · serviceops-radar, Performance-Team (Radar), Developer Productivity
jijiki moved T311385: Netbox and Redis from 🙈🙉🙊Backlog to serviceops-radar on the serviceops board.
Dec 13 2022, 3:53 PM · Patch-For-Review, serviceops-radar, Infrastructure-Foundations, netbox
jijiki added a comment to T311385: Netbox and Redis.

@ayounsi If you wish to use our redis_misc cluster, you can assign a pair/port/db combination here: https://wikitech.wikimedia.org/wiki/Redis. We have 2 pairs (primary-secondary) in each DC. Please keep in mind that:

Dec 13 2022, 3:52 PM · Patch-For-Review, serviceops-radar, Infrastructure-Foundations, netbox
jijiki moved T291620: Better observability/visualization for MediaWiki jobs from 🙈🙉🙊Backlog to 🍦IceBox on the serviceops board.
Dec 13 2022, 3:14 PM · serviceops-radar, Platform Team Workboards (Platform Engineering Reliability), Data-Engineering, Wikibase change dispatching scripts to jobs
jijiki moved T290020: Evaluate and enable audit logging for kubeapi-server from 🙈🙉🙊Backlog to ⎈Kubernetes on the serviceops board.
Dec 13 2022, 3:13 PM · Prod-Kubernetes, serviceops, Kubernetes
jijiki moved T275026: Use a separate key for service account token issuer from 🙈🙉🙊Backlog to ⎈Kubernetes on the serviceops board.
Dec 13 2022, 3:07 PM · serviceops, Prod-Kubernetes, Kubernetes
jijiki edited projects for T296944: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints, added: serviceops-collab; removed serviceops.
Dec 13 2022, 3:05 PM · serviceops-collab, Release-Engineering-Team (Yak Shaving 🐃🪒), Upstream, GitLab (Infrastructure)