Page MenuHomePhabricator

jijiki (effie mouzeli)
is an animal

Projects (8)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Aug 14 2018, 10:50 AM (85 w, 6 d)
Availability
Available
IRC Nick
effie
LDAP User
Effie Mouzeli
MediaWiki User
EMouzeli (WMF) [ Global Accounts ]

Recent Activity

Feb 29 2020

jijiki updated the task description for T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 29 2020, 9:18 AM · Performance-Team, Operations, serviceops

Feb 28 2020

jijiki added a comment to T240684: Test gutter pool failover in production and memcached 1.5.x.

@aaron Please have a look at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569541/ and let us know if it reflects what we are going for, or if there is some nitpicking to be done. We can afterwards merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/574200/ to test routes for the /*/wm-wan/ keys.

Feb 28 2020, 9:39 PM · Performance-Team, Operations, serviceops
jijiki added a comment to T240684: Test gutter pool failover in production and memcached 1.5.x.

Yeah, the eviction ("tko") delay should be low to avoid prolonging DB traffic spikes. Gutter tombstoning and FailoverWithExptimeRoute helps mitigate flapping from low TKO trigger thresholds.

Feb 28 2020, 8:54 PM · Performance-Team, Operations, serviceops
jijiki triaged T246470: Measure segfaults in mediawiki and parsoid servers as Medium priority.
Feb 28 2020, 6:13 PM · serviceops
jijiki created T246470: Measure segfaults in mediawiki and parsoid servers .
Feb 28 2020, 6:13 PM · serviceops
jijiki added a comment to T245888: Rename wtp* servers to parse* (Parsoid PHP servers).

done, sigh

Feb 28 2020, 6:03 PM · Core Platform Team, serviceops, DC-Ops, Operations
jijiki renamed T245888: Rename wtp* servers to parse* (Parsoid PHP servers) from Rename wtp* servers to parsoid* (Parsoid PHP servers) to Rename wtp* servers to parse* (Parsoid PHP servers).
Feb 28 2020, 6:03 PM · Core Platform Team, serviceops, DC-Ops, Operations
jijiki updated the task description for T245888: Rename wtp* servers to parse* (Parsoid PHP servers).
Feb 28 2020, 5:40 PM · Core Platform Team, serviceops, DC-Ops, Operations
jijiki changed the status of T213089: Upgrade memcached for Debian Stretch/Buster, a subtask of T244852: Upgrade and improve our application object caching service (memcached), from Open to Stalled.
Feb 28 2020, 4:39 PM · Operations, serviceops
jijiki changed the status of T213089: Upgrade memcached for Debian Stretch/Buster, a subtask of T224549: Track remaining jessie systems in production, from Open to Stalled.
Feb 28 2020, 4:39 PM · Operations
jijiki changed the status of T213089: Upgrade memcached for Debian Stretch/Buster from Open to Stalled.

This is stalled until we completely stop using redis and we have put the gutter pool in production

Feb 28 2020, 4:39 PM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
jijiki added a comment to T217924: Make canary wait time configurable .

I had a discussion with Tyler just now. We plan the following:

  • add to docs/ in the source tree; this ends up in doc.wikimedia.org
  • add to debian/changelog in the source tree
  • update the wikitech Scap page
  • notify ops list
  • notify wikitech-l list
  • have it mentioned in Scrum of Scrums
Feb 28 2020, 4:09 PM · Operations, Scap, serviceops
jijiki added a comment to T217924: Make canary wait time configurable .

Great we merged this patch! Do we have a plan of how we will communicate this to the deployers when we release scap as well as how to test that it is all good in production? Thank you!

Feb 28 2020, 2:41 PM · Operations, Scap, serviceops
jijiki closed T244472: Stream a subset of mediawiki apache logs to logstash as Resolved.

Thank you @herron and @fgiunchedi for your help!

Feb 28 2020, 12:49 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki updated the task description for T244472: Stream a subset of mediawiki apache logs to logstash .
Feb 28 2020, 12:48 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki created T246431: Minor fixes in pybal checks.
Feb 28 2020, 10:20 AM · Traffic, Operations, Pybal

Feb 27 2020

jijiki created T246348: Log the real X-Client-IP in apache mediawiki logs.
Feb 27 2020, 3:05 PM · Operations
jijiki added a comment to T245841: mcrouter proxies and scap proxies.

What would having all scap proxies also be mcrouter proxies change in terms of the scenario you described above?

Feb 27 2020, 7:39 AM · Release-Engineering-Team, Operations, serviceops
jijiki added a comment to T246002: Move CORE REST API to be served from the MW API Cluster.

Hey Will,

Feb 27 2020, 7:31 AM · serviceops, Core Platform Team Workboards (Green), CPT Initiatives (Core REST API in PHP)

Feb 26 2020

jijiki reopened T245841: mcrouter proxies and scap proxies as "Open".

(Reopening to discuss this a bit more)

Feb 26 2020, 1:33 PM · Release-Engineering-Team, Operations, serviceops

Feb 25 2020

jijiki added a comment to T240684: Test gutter pool failover in production and memcached 1.5.x.

@aaron @Krinkle Please let us know if the configuration and the information on this task are enough to proceed with testing the gutter proxies for /*/wm-wan/ or if there is anything that needs further investigation/explanation.

Feb 25 2020, 8:18 PM · Performance-Team, Operations, serviceops
jijiki added a comment to T244472: Stream a subset of mediawiki apache logs to logstash .

After a lot of fiddling with @herron, we are finally at this https://phabricator.wikimedia.org/P10513 !

Feb 25 2020, 2:58 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki created P10513 parsed log.
Feb 25 2020, 2:56 PM
jijiki added a comment to T245841: mcrouter proxies and scap proxies.

If there are no objections, I would like to proceed with this

Feb 25 2020, 11:05 AM · Release-Engineering-Team, Operations, serviceops

Feb 24 2020

ori awarded T245530: Deploy scap 3.13.0-1 a Party Time token.
Feb 24 2020, 6:33 PM · Release-Engineering-Team-TODO, serviceops, Scap
LarsWirzenius awarded T245530: Deploy scap 3.13.0-1 a Like token.
Feb 24 2020, 11:50 AM · Release-Engineering-Team-TODO, serviceops, Scap
jijiki closed T245530: Deploy scap 3.13.0-1, a subtask of T99740: Use static php array files for l10n cache at WMF (instead of CDB), as Resolved.
Feb 24 2020, 11:42 AM · Performance-Team, Release-Engineering-Team-TODO, Scap, MediaWiki-Internationalization
jijiki closed T245530: Deploy scap 3.13.0-1 as Resolved.

Let me know if we come across any issues:)

Feb 24 2020, 11:42 AM · Release-Engineering-Team-TODO, serviceops, Scap
jijiki added a project to T240187: mw1280 crashed logging correctable memory errors: serviceops.
Feb 24 2020, 11:07 AM · serviceops, Operations, ops-eqiad

Feb 21 2020

jijiki created T245888: Rename wtp* servers to parse* (Parsoid PHP servers).
Feb 21 2020, 11:15 PM · Core Platform Team, serviceops, DC-Ops, Operations
jijiki added a comment to T244472: Stream a subset of mediawiki apache logs to logstash .

With a little bit more fiddling, I managed to run puppet on ssh deployment-mediawiki-09.deployment-prep.eqiad.wmflabs! @herron does that unblock you?

Feb 21 2020, 9:49 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki added a comment to T244472: Stream a subset of mediawiki apache logs to logstash .

I have uploaded a patch which I manually tried on beta, this seems to work, but sadly, puppet breaks a bit further down the road

Feb 21 2020, 9:28 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki updated subscribers of T236048: Move debugging symbols and tools to a new class.
Feb 21 2020, 7:57 PM · Operations, serviceops
jijiki added a comment to T245530: Deploy scap 3.13.0-1.

Package has been build and pushed to wikimedia repo, I'll roll it to production on Monday

Feb 21 2020, 7:55 PM · Release-Engineering-Team-TODO, serviceops, Scap
jijiki edited projects for T244472: Stream a subset of mediawiki apache logs to logstash , added: Beta-Cluster-Infrastructure; removed Patch-For-Review.
Feb 21 2020, 7:39 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki added a comment to T244472: Stream a subset of mediawiki apache logs to logstash .

@herron any ideas how to proceed here? Is there someone who can help? Apparently this patch could potentially break beta.

Feb 21 2020, 7:39 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki updated the task description for T245841: mcrouter proxies and scap proxies.
Feb 21 2020, 4:21 PM · Release-Engineering-Team, Operations, serviceops
jijiki created T245841: mcrouter proxies and scap proxies.
Feb 21 2020, 4:21 PM · Release-Engineering-Team, Operations, serviceops

Feb 19 2020

jijiki added a comment to T244472: Stream a subset of mediawiki apache logs to logstash .

It appears that on beta the variable $server_role = $::_role.split('/')[-1 is not evaluated properly, while in production, it looks just fine https://puppet-compiler.wmflabs.org/compiler1001/20912/mwdebug1001.eqiad.wmnet/, I believe this has something to do with this server's role in beta?

Feb 19 2020, 5:51 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki claimed T245530: Deploy scap 3.13.0-1.
Feb 19 2020, 12:03 PM · Release-Engineering-Team-TODO, serviceops, Scap
jijiki added a comment to T240684: Test gutter pool failover in production and memcached 1.5.x.

To proceed with testing, we will puppetise the following configuration, and roll it to a couple of canary servers, and block traffic towards mc* like we did earlier. Our goal is to test memcached 1.5.x on debian Buster, since that is what the gutter pool is running

Feb 19 2020, 12:00 PM · Performance-Team, Operations, serviceops
jijiki updated the task description for T244852: Upgrade and improve our application object caching service (memcached).
Feb 19 2020, 11:56 AM · Operations, serviceops
jijiki updated subscribers of T240684: Test gutter pool failover in production and memcached 1.5.x.

Very nice summary, thanks!
A couple of questions:

FailoverWithExptimeRoute, where we define how long the values keys updated (eg set/add) during the failover will live on the gutter. All keys set during a failover with the following config, had a TTL of 10s.

What happens if we try to set a key with TTL of say 5s? Will it get TTL 10s or 5s? I would hope for 5s but I suspect it might not be the case. I am asking since setting a high enough TTL like 10/20/30 mins could be good if keys set with lower TTLs are not affected.

Feb 19 2020, 11:36 AM · Performance-Team, Operations, serviceops
jijiki updated the task description for T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 19 2020, 11:10 AM · Performance-Team, Operations, serviceops
jijiki added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

Weights of mw1349-mw1355 were switched to 30

Feb 19 2020, 10:17 AM · serviceops, Operations

Feb 17 2020

jijiki added a comment to T240684: Test gutter pool failover in production and memcached 1.5.x.

Test if failover works and when to failover

Feb 17 2020, 8:29 PM · Performance-Team, Operations, serviceops
jijiki updated the title for P10436 eqiad-FailoverRoute-mcrouter-config from eqiad-failover-mcrouter-config to eqiad-FailoverRoute-mcrouter-config.
Feb 17 2020, 8:13 PM
jijiki created P10437 codfw-FailoverWithExptimeRoute-mcrouter-config.
Feb 17 2020, 8:13 PM
jijiki created P10436 eqiad-FailoverRoute-mcrouter-config.
Feb 17 2020, 8:12 PM
jijiki updated the task description for T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 17 2020, 8:05 PM · Performance-Team, Operations, serviceops
jijiki updated the title for P10429 mediawiki-mcrouter-TKO-failover from mediawiki-mcrouter-error to mediawiki-mcrouter-TKO-failover.
Feb 17 2020, 7:33 PM
jijiki created P10433 mcrouter-mw-more-errors.
Feb 17 2020, 7:32 PM
jijiki created P10429 mediawiki-mcrouter-TKO-failover.
Feb 17 2020, 7:10 PM
jijiki updated the language for P10425 mcrouter from autodetect to json.
Feb 17 2020, 3:57 PM
jijiki created P10425 mcrouter.
Feb 17 2020, 3:56 PM
jijiki added a comment to T241849: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet.

Thank you all!

Feb 17 2020, 9:37 AM · Performance-Team, Performance-Team-publish, serviceops, Operations

Feb 14 2020

jijiki added a comment to T244472: Stream a subset of mediawiki apache logs to logstash .

Oh great, thanks! From https://logstash-beta.wmflabs.org/, it looks that logs are flowing, so maybe we can try the change there after all

Feb 14 2020, 5:51 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability

Feb 13 2020

jijiki closed T242606: No mw canary servers in codfw as Resolved.

thank you daniel!

Feb 13 2020, 7:16 PM · Operations, serviceops
jijiki added a comment to T244472: Stream a subset of mediawiki apache logs to logstash .

@herron I fiddled a bit on beta, it appears that for some reason, nothing is being streamed there since today, I am not sure if I broke it myself while I was trying to make it work :/ FWIW, I have restored the original config on deployment-logstash03

Feb 13 2020, 7:13 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki triaged T245183: PHP7 corruption: Method call executed on unrelated object (also: Call to undefined method) as Medium priority.
Feb 13 2020, 6:30 PM · Wikimedia-production-error, serviceops, Operations
jijiki updated subscribers of T244472: Stream a subset of mediawiki apache logs to logstash .

I have uploaded a patch that could possibly work, my issue generally is that I can't find a sane and safe way to test if those logstash filters will do what we need. @herron any ideas are welcome

Feb 13 2020, 5:52 PM · Patch-For-Review, Beta-Cluster-Infrastructure, Operations, serviceops, observability
jijiki updated the task description for T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 13 2020, 5:25 PM · Performance-Team, Operations, serviceops
jijiki added a comment to T227734: Investigate whether GD is still needed on appservers.
  • getid3 (a TimedMediaHandler dependency I think?) uses it
  • BitmapHandler in core uses it, but probably only as a fallback
  • the IPTC parser in core uses it, which AIUI might be triggered for certain JPEG files
Feb 13 2020, 10:58 AM · Patch-For-Review, Release-Engineering-Team-TODO, Release-Engineering-Team (Deployment services), Technical-Debt, Operations
jijiki created T245121: RRDP status alert .
Feb 13 2020, 9:26 AM · Operations, netops

Feb 12 2020

jijiki added a comment to T244886: (2020-01-15) rack/setup/install mc-gp100[123].eqiad.wmnet.

@RobH, those hosts are delivered and in production per T241795

Feb 12 2020, 4:15 PM · Operations, ops-eqiad, DC-Ops
jijiki added a comment to T240684: Test gutter pool failover in production and memcached 1.5.x.

@elukey thank you for unblocking this !!!

Feb 12 2020, 9:47 AM · Performance-Team, Operations, serviceops
jijiki updated the task description for T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 12 2020, 9:46 AM · Performance-Team, Operations, serviceops
jijiki renamed T240684: Test gutter pool failover in production and memcached 1.5.x from Test gutter pool failover in production to Test gutter pool failover in production and memcached 1.5.x.
Feb 12 2020, 9:44 AM · Performance-Team, Operations, serviceops

Feb 11 2020

jijiki renamed T244852: Upgrade and improve our application object caching service (memcached) from Upgrade and improve our application object caching service (memcachedd) to Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 1:51 PM · Operations, serviceops
jijiki added a subtask for T244852: Upgrade and improve our application object caching service (memcached): T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 11 2020, 1:51 PM · Operations, serviceops
jijiki added a parent task for T240684: Test gutter pool failover in production and memcached 1.5.x: T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 1:51 PM · Performance-Team, Operations, serviceops
jijiki renamed T240684: Test gutter pool failover in production and memcached 1.5.x from Upgrade and improve our application object caching service (memcached) to Test gutter pool failover in production .
Feb 11 2020, 1:50 PM · Performance-Team, Operations, serviceops
jijiki added a comment to T244410: LDAP access to the wmf group for CherRaye Glenn (superset, turnilo, hue).

@CGlenn it appears you are already in the ldap wmf group as 'keepit-ssh', which is the username (shell name) in wikitech as well. Can you check please if your access is in order?

Feb 11 2020, 1:49 PM · Analytics-Kanban, Analytics, Operations, LDAP-Access-Requests
jijiki created P10383 mcrouter gutter test.
Feb 11 2020, 1:10 PM · serviceops
jijiki triaged T244852: Upgrade and improve our application object caching service (memcached) as Medium priority.
Feb 11 2020, 12:57 PM · Operations, serviceops
jijiki removed a parent task for T213089: Upgrade memcached for Debian Stretch/Buster: T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 11 2020, 12:55 PM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
jijiki removed a parent task for T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen: T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 11 2020, 12:55 PM · Performance-Team (Radar), User-Elukey, MediaWiki-Cache, Operations
jijiki removed subtasks for T240684: Test gutter pool failover in production and memcached 1.5.x: T244476: Test and deploy mcrouter 0.41 , T244340: Reduce read pressure on memcached servers by adding a machine-local Memcache instance, T239249: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet, T213089: Upgrade memcached for Debian Stretch/Buster, Unknown Object (Task), T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet, T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen.
Feb 11 2020, 12:55 PM · Performance-Team, Operations, serviceops
jijiki removed a parent task for T239249: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet: T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 11 2020, 12:55 PM · Operations, ops-codfw
jijiki removed a parent task for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet: T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 11 2020, 12:55 PM · serviceops, Operations
jijiki removed a parent task for T244340: Reduce read pressure on memcached servers by adding a machine-local Memcache instance: T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 11 2020, 12:55 PM · Operations, serviceops
jijiki removed a parent task for T244476: Test and deploy mcrouter 0.41 : T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 11 2020, 12:55 PM · serviceops, Operations
jijiki added a parent task for T213089: Upgrade memcached for Debian Stretch/Buster: T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:55 PM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
jijiki added a parent task for T223863: memkeys segfaults on Debian Stretch: T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:54 PM · Patch-For-Review, User-Elukey, Operations
jijiki added a parent task for T217020: Test different growth factors for memcached (prep step for upgrade to newer versions): T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:54 PM · User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
jijiki added a parent task for T239249: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet: T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:54 PM · Operations, ops-codfw
jijiki added a parent task for T244340: Reduce read pressure on memcached servers by adding a machine-local Memcache instance: T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:54 PM · Operations, serviceops
jijiki added a parent task for T244476: Test and deploy mcrouter 0.41 : T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:54 PM · serviceops, Operations
jijiki added subtasks for T244852: Upgrade and improve our application object caching service (memcached): Unknown Object (Task), Unknown Object (Task), T213089: Upgrade memcached for Debian Stretch/Buster, T217020: Test different growth factors for memcached (prep step for upgrade to newer versions), T223863: memkeys segfaults on Debian Stretch, T239249: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet, T244340: Reduce read pressure on memcached servers by adding a machine-local Memcache instance, T244476: Test and deploy mcrouter 0.41 .
Feb 11 2020, 12:54 PM · Operations, serviceops
jijiki added a parent task for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet: T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:52 PM · serviceops, Operations
jijiki added a subtask for T244852: Upgrade and improve our application object caching service (memcached): T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Feb 11 2020, 12:52 PM · Operations, serviceops
jijiki added a parent task for T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen: T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:52 PM · Performance-Team (Radar), User-Elukey, MediaWiki-Cache, Operations
jijiki added a subtask for T244852: Upgrade and improve our application object caching service (memcached): T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen.
Feb 11 2020, 12:52 PM · Operations, serviceops
jijiki created T244852: Upgrade and improve our application object caching service (memcached).
Feb 11 2020, 12:51 PM · Operations, serviceops
jijiki added a comment to T244410: LDAP access to the wmf group for CherRaye Glenn (superset, turnilo, hue).

will fix :)

Feb 11 2020, 9:30 AM · Analytics-Kanban, Analytics, Operations, LDAP-Access-Requests

Feb 10 2020

jijiki updated the task description for T240684: Test gutter pool failover in production and memcached 1.5.x.
Feb 10 2020, 6:57 PM · Performance-Team, Operations, serviceops
jijiki added a comment to T240684: Test gutter pool failover in production and memcached 1.5.x.

On mwdebug1001 we have deployed a config similar to T240684#5826966, where we failover in case of a TKO to the gutter pool servers (mc-gp*). We again have set --timeouts-until-tko=3.

Feb 10 2020, 6:56 PM · Performance-Team, Operations, serviceops
jijiki added a comment to T244407: Requesting access to sites from Google Search Console.

@kzimmerman oh my bad, sorry!

Feb 10 2020, 6:24 PM · Operations, SRE-Access-Requests
jijiki closed T243924: Requesting access to deployment for niedzielski as Resolved.

done

Feb 10 2020, 3:57 PM · Operations, SRE-Access-Requests
jijiki closed T244410: LDAP access to the wmf group for CherRaye Glenn (superset, turnilo, hue) as Resolved.

@CGlenn done:)

Feb 10 2020, 2:54 PM · Analytics-Kanban, Analytics, Operations, LDAP-Access-Requests