Page MenuHomePhabricator

ema (Emanuele Rocca)
Senior Site Reliability Engineer, Traffic Team

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Sep 29 2015, 8:49 PM (181 w, 1 d)
Availability
Available
IRC Nick
ema
LDAP User
Ema
MediaWiki User
Unknown

Recent Activity

Today

ema added a comment to T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish.

We're currently using swift-rw (eqiad only) as the origin server for upload cache misses. Thumb traffic can however be served active/active by swift-ro. Doing this with remap rules requires using the regex remap plugin, which isn't great. Let's do it in Lua instead.

Thu, Mar 21, 9:25 AM · Patch-For-Review, Operations, Traffic
ema created P8251 new_conn.stp.
Thu, Mar 21, 8:40 AM

Mon, Mar 18

ema added a comment to T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish.

RAM cache usage issue submitted upstream: https://github.com/apache/trafficserver/issues/5179

Mon, Mar 18, 3:58 PM · Patch-For-Review, Operations, Traffic
ema added a comment to T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish.

I have disabled puppet on cp2015 and manually set proxy.config.cache.ram_cache.size to 1G to test if, by setting an explicit limit, the value is actually enforced.

Mon, Mar 18, 1:32 PM · Patch-For-Review, Operations, Traffic

Fri, Mar 15

ema added a comment to T216006: nuke_limit often reached on esams varnish frontends.

The issue should be fixed. @VladimirAlexiev, @Addshore: can you confirm?

Fri, Mar 15, 5:44 PM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema added a comment to T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish.

RAM cache usage seems to be growing non-stop. I have left proxy.config.cache.ram_cache.size to the default value of -1, which according to the docs means that the RAM cache size is automatically determined. That seems to be true, and ATS decided that our RAM caches should be ~970M in size. However, the amount of data in RAM cache has now grown much larger than the limit:

Fri, Mar 15, 4:20 PM · Patch-For-Review, Operations, Traffic
ema edited P8205 ats_cache_scan.py.
Fri, Mar 15, 2:38 PM
ema created P8205 ats_cache_scan.py.
Fri, Mar 15, 10:33 AM

Thu, Mar 14

ema edited P8197 304-update-headers.vtc.
Thu, Mar 14, 8:10 AM
ema created P8197 304-update-headers.vtc.
Thu, Mar 14, 7:26 AM

Wed, Mar 13

ema added a comment to T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish.

Preliminary testing went well: the cp-ats cluster in codfw served all frontend misses from cp2002 for about one hour between 17:18 and 18:28 without any immediately obvious issue.

Wed, Mar 13, 6:34 PM · Patch-For-Review, Operations, Traffic
TheDJ awarded T213214: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) a Stroopwafel token.
Wed, Mar 13, 9:04 AM · User-Ryasmeen, Traffic, Wikimedia-Apache-configuration, Operations, VisualEditor
ema closed T217893: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing as Resolved.

Closing this as cp1077 is fine and back in service. For the general Varnish scalability issue, the solution we've identified is moving to ATS (ongoing, see T213263).

Wed, Mar 13, 8:48 AM · Patch-For-Review, Operations, Traffic
ema closed T213214: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) as Resolved.

I did notice that shift + refresh seemed to fix it after some time, so that makes sense.

Can you still reproduce this, or can the task be closed?

I can't, so it looks like it's fixed.

Wed, Mar 13, 7:03 AM · User-Ryasmeen, Traffic, Wikimedia-Apache-configuration, Operations, VisualEditor
ema created P8191 (An Untitled Masterwork).
Wed, Mar 13, 7:02 AM
ema added a comment to T213214: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...).

I did notice that shift + refresh seemed to fix it after some time, so that makes sense.

Wed, Mar 13, 6:59 AM · User-Ryasmeen, Traffic, Wikimedia-Apache-configuration, Operations, VisualEditor

Tue, Mar 12

ema moved T217893: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing from Triage to Caching on the Traffic board.
Tue, Mar 12, 2:07 PM · Patch-For-Review, Operations, Traffic
ema triaged T217897: Reduce / remove the aggessive cache busting behaviour of wdqs-updater as Normal priority.
Tue, Mar 12, 2:06 PM · User-Smalyshev, Wikidata, Wikidata-Query-Service, Traffic, Operations
ema moved T217897: Reduce / remove the aggessive cache busting behaviour of wdqs-updater from Triage to Caching on the Traffic board.
Tue, Mar 12, 2:06 PM · User-Smalyshev, Wikidata, Wikidata-Query-Service, Traffic, Operations
ema added a comment to T217893: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing.

Re-pooling the service caused the issue to show up again. For some reason, the cron jobs restarting varnish.service do not seem to have worked.

Tue, Mar 12, 11:33 AM · Patch-For-Review, Operations, Traffic
ema added a comment to T217893: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing.

Re-pooling the service caused the issue to show up again. For some reason, the cron jobs restarting varnish.service do not seem to have worked. Although cron logged two restarts, one on Mar 08 and one on Mar 12:

Tue, Mar 12, 9:04 AM · Patch-For-Review, Operations, Traffic
ema triaged T217893: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing as Normal priority.

At the time of the issue, cp1077 was failing to fetch objects from the origin servers and was affected heavily by mbox lag.

Tue, Mar 12, 8:41 AM · Patch-For-Review, Operations, Traffic
ema reopened T128188: Make CI run Varnish VCL tests as "Open".

I am pretty sure @ema finished up the integration of varnishtest with CI / rake test.

Tue, Mar 12, 7:59 AM · Varnish, Patch-For-Review, Operations, Continuous-Integration-Infrastructure, Traffic

Thu, Mar 7

ema added a comment to T216006: nuke_limit often reached on esams varnish frontends.

Interestingly, the problem is not reproducible with larger objects, as varnish autonomously decides they're too large and does not cache them

Thu, Mar 7, 5:05 PM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema renamed T216006: nuke_limit often reached on esams varnish frontends from Some esams<->eqiad varnish backend connections closed by peer to nuke_limit often reached on esams varnish frontends.
Thu, Mar 7, 3:04 PM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema added a comment to T216006: nuke_limit often reached on esams varnish frontends.

Those connection resets on the varnish backend layer happen when frontend caches are full and varnish cannot make space for a newly fetched object body:

Thu, Mar 7, 3:02 PM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema added a comment to T216006: nuke_limit often reached on esams varnish frontends.

Varnishlog of the varnish backend instance serving the request in esams reports the following:

Thu, Mar 7, 10:19 AM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema renamed T216006: nuke_limit often reached on esams varnish frontends from Wikidata sometimes cuts off entity RDF to Some esams<->eqiad varnish backend connections closed by peer .
Thu, Mar 7, 10:07 AM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema merged T215389: esams cache layer mangles downloads of specific url into T216006: nuke_limit often reached on esams varnish frontends.
Thu, Mar 7, 10:06 AM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema merged task T215389: esams cache layer mangles downloads of specific url into T216006: nuke_limit often reached on esams varnish frontends.
Thu, Mar 7, 10:06 AM · Operations, Traffic
ema added a comment to T202966: Make cp1099 the new pinkunicorn.

cp1099 is the last standing host between me and powering off asw-c-eqiad.

From this task and the prompt cp1099 is a Unpuppetised system for testing (test) it should be fine to move its uplink anytime.

Thu, Mar 7, 9:35 AM · Patch-For-Review, Operations, Traffic
ema moved T120085: Serve Main Page of WMF wikis from a consistent URL from Triage to General on the Traffic board.
Thu, Mar 7, 9:30 AM · Core Platform Team Backlog (Watching / External), Performance-Team, Operations, Traffic, TechCom-RFC, SEO, Wikimedia-Site-requests

Wed, Mar 6

ema created P8166 ats-be conftool-sync.
Wed, Mar 6, 2:13 PM
ema added a comment to T216006: nuke_limit often reached on esams varnish frontends.

I haven't done any investigation yet, but it sounds similar to T215389.

Wed, Mar 6, 10:20 AM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema moved T216006: nuke_limit often reached on esams varnish frontends from Triage to Caching on the Traffic board.
Wed, Mar 6, 10:19 AM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema added a project to T216006: nuke_limit often reached on esams varnish frontends: Traffic.
Wed, Mar 6, 10:19 AM · Patch-For-Review, Traffic, User-Addshore, Operations, Wikidata-Campsite, Wikidata
ema triaged T215956: Consider stashing data-parsoid for VE as Normal priority.

it seems like we will need to add a rule to Varnish to pass on these requests

Wed, Mar 6, 10:16 AM · Operations, Traffic, User-Eevans, User-mobrovac, Core Platform Team Backlog (Designing), Parsoid, VisualEditor, RESTBase, Services (designing)
ema added a comment to T216716: cp5007 correctable mem errors.

Can this be closed?

Wed, Mar 6, 10:12 AM · Operations, ops-eqsin, Traffic
ema triaged T216716: cp5007 correctable mem errors as Normal priority.
Wed, Mar 6, 10:12 AM · Operations, ops-eqsin, Traffic
ema added a comment to T216717: cp5006 correctable mem errors.

Anything else to be done here?

Wed, Mar 6, 10:11 AM · ops-eqsin, Traffic, Operations
ema triaged T216717: cp5006 correctable mem errors as Normal priority.
Wed, Mar 6, 10:11 AM · ops-eqsin, Traffic, Operations
ema moved T217407: Indexing of https://www.wikidata.org in the Yandex Search Engine from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:58 AM · Operations, Traffic
ema moved T216717: cp5006 correctable mem errors from Triage to Hardware on the Traffic board.
Wed, Mar 6, 9:56 AM · ops-eqsin, Traffic, Operations
ema moved T216716: cp5007 correctable mem errors from Triage to Hardware on the Traffic board.
Wed, Mar 6, 9:56 AM · Operations, ops-eqsin, Traffic
ema moved T216691: amber light on cp5006/5007 from Triage to Hardware on the Traffic board.
Wed, Mar 6, 9:56 AM · Traffic, ops-eqsin, Operations
ema moved T216339: Normalize thumbnail request URLs in Varnish to avoid cachebusting from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:55 AM · Patch-For-Review, Traffic, Performance-Team, Operations
ema moved T216140: Investigating using CI to automate testing VCL changes against all cluster/dc combos from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:55 AM · Operations, Traffic
ema moved T215987: Verify that hit/miss stats in WebRequest are correct from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:55 AM · Operations, Traffic, Core Platform Team Backlog (Later), Analytics, Services (blocked), RESTBase
ema added a comment to T215987: Verify that hit/miss stats in WebRequest are correct.

Is there anything to do here? :-)

Wed, Mar 6, 9:55 AM · Operations, Traffic, Core Platform Team Backlog (Later), Analytics, Services (blocked), RESTBase
ema moved T215956: Consider stashing data-parsoid for VE from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:52 AM · Operations, Traffic, User-Eevans, User-mobrovac, Core Platform Team Backlog (Designing), Parsoid, VisualEditor, RESTBase, Services (designing)
ema moved T215624: AMC – Opt-in for logged out users from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:48 AM · Operations, User-Joe, Traffic, Advanced Mobile Contributions
ema moved T215389: esams cache layer mangles downloads of specific url from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:47 AM · Operations, Traffic
ema moved T215071: Merge Wikipedia subdomains into one, to discourage censorship from Triage to General on the Traffic board.
Wed, Mar 6, 9:45 AM · Domains, DNS, Traffic, Operations, HTTPS
ema moved T214998: Remove .m. subdomain, serve mobile and desktop variants through the same URL from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:42 AM · TechCom-RFC, Readers-Web-Backlog (Tracking), Traffic, MobileFrontend, Operations
ema moved T214525: remove IBM/Silverpop 1024-bit domain key from Triage to DNS Names on the Traffic board.
Wed, Mar 6, 9:37 AM · Traffic, Operations, fundraising-tech-ops, DNS
ema moved T214253: en.wikipedia.com [sic] serves an invalid certificate from Triage to TLS on the Traffic board.
Wed, Mar 6, 9:36 AM · Operations, Traffic, HTTPS
ema moved T213769: Zero VCL removal from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:36 AM · Patch-For-Review, Zero, Operations, Traffic
ema moved T213371: Document and possibly fine-tune how Proton interacts with Varnish from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:34 AM · Readers-Web-Backlog (Tracking), Services (watching), serviceops, Traffic, Reading-Infrastructure-Team-Backlog, Proton, Operations
ema moved T213214: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) from Triage to Watching on the Traffic board.
Wed, Mar 6, 9:34 AM · User-Ryasmeen, Traffic, Wikimedia-Apache-configuration, Operations, VisualEditor
ema moved T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish from Triage to Caching on the Traffic board.
Wed, Mar 6, 9:33 AM · Patch-For-Review, Operations, Traffic

Fri, Mar 1

ema triaged T217403: Jenkins job builder ignores BUILD_TIMEOUT as Normal priority.
Fri, Mar 1, 10:43 AM · Upstream, Release-Engineering-Team (Kanban), Traffic, Operations, Continuous-Integration-Infrastructure
ema created T217403: Jenkins job builder ignores BUILD_TIMEOUT.
Fri, Mar 1, 10:43 AM · Upstream, Release-Engineering-Team (Kanban), Traffic, Operations, Continuous-Integration-Infrastructure

Jan 10 2019

ema triaged T213417: lvs2002: raid battery failure as Normal priority.
Jan 10 2019, 12:33 PM · Operations, ops-codfw, Traffic
ema created T213417: lvs2002: raid battery failure.
Jan 10 2019, 12:32 PM · Operations, ops-codfw, Traffic

Jan 9 2019

ema updated the task description for T206339: Separate Traffic layer caches for PHP7/HHVM.
Jan 9 2019, 11:30 AM · Patch-For-Review, Traffic, Operations
ema triaged T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish as Normal priority.
Jan 9 2019, 11:10 AM · Patch-For-Review, Operations, Traffic
ema created T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish.
Jan 9 2019, 11:10 AM · Patch-For-Review, Operations, Traffic
ema closed T209590: HTTP/2 requests fail with too-long URLs as Resolved.

The patch by @Vgutierrez fixed this bug. Closing.

Jan 9 2019, 11:02 AM · Patch-For-Review, Traffic, Operations
ema moved T212197: Deliver mobile-based version for automatic translations from Triage to Caching on the Traffic board.
Jan 9 2019, 10:52 AM · MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), Patch-For-Review, Operations, Traffic, ExternalGuidance
ema closed T207048: ATS production-ready as a backend cache layer as Resolved.

We've added TLS support for maps and fixed the SAN list on swift to ensure proper TLS connections with upload origin servers. This is thus done.

Jan 9 2019, 9:50 AM · Patch-For-Review, Operations, Traffic

Jan 4 2019

ema added a comment to T209590: HTTP/2 requests fail with too-long URLs.

Note it's ok if the server implements some limits. The problem here is that the client isn't getting the expected 4xx response (e.g. 414 or 431) when the limit is hit, all it sees is a dropped connection.

Jan 4 2019, 9:01 PM · Patch-For-Review, Traffic, Operations
ema moved T209590: HTTP/2 requests fail with too-long URLs from Triage to TLS on the Traffic board.
Jan 4 2019, 2:37 PM · Patch-For-Review, Traffic, Operations
ema updated subscribers of T209590: HTTP/2 requests fail with too-long URLs.

P7916, which worked for @Huji, does not work for me. The path in P7916 (/wiki/Main_Page?_=xxxx...) is 4698 characters long.

Jan 4 2019, 2:34 PM · Patch-For-Review, Traffic, Operations
ema moved T212312: prometheus-based graph significantly slower than statsd equivalent from Triage to Watching on the Traffic board.
Jan 4 2019, 1:30 PM · monitoring, Traffic, Operations
ema moved T212914: Redirecting incoming queries to non-existent subpages (due to Godaddy behavior on some external WikiJournal sites) from Triage to DNS Names on the Traffic board.
Jan 4 2019, 1:30 PM · Operations, Traffic, Domains
ema triaged T212914: Redirecting incoming queries to non-existent subpages (due to Godaddy behavior on some external WikiJournal sites) as Normal priority.
Jan 4 2019, 1:30 PM · Operations, Traffic, Domains

Jan 2 2019

ema awarded T209136: python3-etcd needs python3-dnspython a Party Time token.
Jan 2 2019, 12:03 PM · Core Platform Team Kanban (Done with CPT), Services (watching), Patch-For-Review, Operations, Operations-Software-Development
ema closed T212215: Update Subject Alternative Name field in TLS certificates for swift as Resolved.

New certificates deployed both in codfw and in eqiad.

Jan 2 2019, 12:02 PM · Patch-For-Review, media-storage, Operations, Traffic
ema closed T212215: Update Subject Alternative Name field in TLS certificates for swift, a subtask of T207048: ATS production-ready as a backend cache layer, as Resolved.
Jan 2 2019, 12:02 PM · Patch-For-Review, Operations, Traffic

Dec 21 2018

ema added a comment to T212215: Update Subject Alternative Name field in TLS certificates for swift.

Tested the new cert on ms-fe2006, looks good:

Dec 21 2018, 9:36 AM · Patch-For-Review, media-storage, Operations, Traffic

Dec 19 2018

ema triaged T212312: prometheus-based graph significantly slower than statsd equivalent as Normal priority.
Dec 19 2018, 4:18 PM · monitoring, Traffic, Operations
ema created T212312: prometheus-based graph significantly slower than statsd equivalent.
Dec 19 2018, 4:18 PM · monitoring, Traffic, Operations
ema moved T212310: varnishreqstats sends truncated statsd traffic from Triage to Caching on the Traffic board.
Dec 19 2018, 4:12 PM · Operations, Traffic
ema moved T212215: Update Subject Alternative Name field in TLS certificates for swift from Triage to TLS on the Traffic board.
Dec 19 2018, 4:12 PM · Patch-For-Review, media-storage, Operations, Traffic

Dec 18 2018

ema triaged T212219: wmf-auto-restart fails on certain legacy services as Normal priority.
Dec 18 2018, 3:06 PM · Patch-For-Review, Operations
ema created T212219: wmf-auto-restart fails on certain legacy services.
Dec 18 2018, 3:06 PM · Patch-For-Review, Operations
ema triaged T212215: Update Subject Alternative Name field in TLS certificates for swift as Normal priority.
Dec 18 2018, 2:43 PM · Patch-For-Review, media-storage, Operations, Traffic

Dec 17 2018

ema updated the task description for T210411: Applayer services without TLS.
Dec 17 2018, 1:15 PM · serviceops, Operations, Traffic
ema added a comment to T210411: Applayer services without TLS.

I do agree with @Joe, without a proper PKI this is going to be painful. For now however I've added TLS support to kartotherian (T211970) as that's part of cache_upload, which is the cluster we're gonna tackle first for the conversion to ATS.

Dec 17 2018, 1:15 PM · serviceops, Operations, Traffic
ema closed T211970: kartotherian TLS support as Resolved.
Dec 17 2018, 1:11 PM · Patch-For-Review, Maps (Kartotherian), Operations, Traffic
ema closed T211970: kartotherian TLS support, a subtask of T210411: Applayer services without TLS, as Resolved.
Dec 17 2018, 1:11 PM · serviceops, Operations, Traffic
ema moved T211970: kartotherian TLS support from Triage to TLS on the Traffic board.
Dec 17 2018, 11:58 AM · Patch-For-Review, Maps (Kartotherian), Operations, Traffic
ema closed T204355: Allow traffic team to manage the traffic blog on phame as Resolved.

Yes @mmodell, now I can. Thank you!

Dec 17 2018, 9:59 AM · Operations, Phabricator, Traffic

Dec 14 2018

ema added a project to T211970: kartotherian TLS support: Maps (Kartotherian).
Dec 14 2018, 12:18 PM · Patch-For-Review, Maps (Kartotherian), Operations, Traffic
ema triaged T211970: kartotherian TLS support as Normal priority.
Dec 14 2018, 12:17 PM · Patch-For-Review, Maps (Kartotherian), Operations, Traffic

Dec 12 2018

ema moved T211661: Automatically clean up unused thumbnails in Swift from Triage to Watching on the Traffic board.
Dec 12 2018, 2:22 PM · Patch-For-Review, Traffic, media-storage, Operations, Performance-Team
ema moved T182028: DNS repo: add CI checks for obvious configuration errors from Triage to DNS Infra on the Traffic board.
Dec 12 2018, 2:21 PM · Traffic, DNS, Patch-For-Review, Operations-Software-Development, Operations
ema moved T211697: clean up deprecated TLS certificates from the puppet repo from Triage to TLS on the Traffic board.
Dec 12 2018, 2:21 PM · Traffic, Patch-For-Review, Operations
ema triaged T211750: Introduce Python code formatters usage as Normal priority.
Dec 12 2018, 10:48 AM · Patch-For-Review, Operations, Operations-Software-Development

Dec 11 2018

ema added a comment to T211416: Put restbase201[3-8] into conftool and LVS.

And finally confctl throws an exception:

root@restbase2014:~# confctl --quiet depool --service restbase
CRITICAL:conftool:Could not load driver etcd: No module named 'dns'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/conftool/backend.py", line 15, in __init__
    exec(compile(open(driver_file).read(), driver_file, 'exec'), ctx)
  File "/usr/lib/python3/dist-packages/conftool/drivers/etcd.py", line 4, in <module>
    import etcd
  File "/usr/lib/python3/dist-packages/etcd/__init__.py", line 2, in <module>
    from .client import Client
  File "/usr/lib/python3/dist-packages/etcd/client.py", line 21, in <module>
    import dns.resolver
ImportError: No module named 'dns'
Dec 11 2018, 3:49 PM · Core Platform Team Kanban (Done with CPT), Services (done), User-fgiunchedi, Operations