ema (Emanuele Rocca)
Senior Site Reliability Engineer, Traffic Team

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Sep 29 2015, 8:49 PM (163 w, 5 d)
Availability
Available
IRC Nick
ema
LDAP User
Ema
MediaWiki User
Unknown

Recent Activity

Fri, Nov 16

ema changed the status of T204209: Define and deploy Icinga checks for ATS backends from Stalled to Open.

We fixed the verify_config issue in ATS 8.0.0-1wm2, this is not stalled anymore.

Fri, Nov 16, 4:32 PM · Patch-For-Review, Traffic, Operations
ema changed the status of T204209: Define and deploy Icinga checks for ATS backends, a subtask of T207048: ATS production-ready as a backend cache layer, from Stalled to Open.
Fri, Nov 16, 4:32 PM · Operations, Traffic
ema triaged T209703: trafficserver debian-glue builds failing on integration-slave-jessie-1001: No space left on device as Normal priority.
Fri, Nov 16, 2:21 PM · Traffic, Operations, Continuous-Integration-Infrastructure
ema created T209703: trafficserver debian-glue builds failing on integration-slave-jessie-1001: No space left on device.
Fri, Nov 16, 2:21 PM · Traffic, Operations, Continuous-Integration-Infrastructure

Thu, Nov 15

ema updated the task description for T207048: ATS production-ready as a backend cache layer.
Thu, Nov 15, 10:08 AM · Operations, Traffic
ema closed T204225: ATS: log inspection at runtime as Resolved.
Thu, Nov 15, 10:08 AM · Patch-For-Review, Operations, Traffic
ema closed T204225: ATS: log inspection at runtime, a subtask of T207048: ATS production-ready as a backend cache layer, as Resolved.
Thu, Nov 15, 10:08 AM · Operations, Traffic
ema moved T208282: Increase EventLogging limit from 2K to 5K from Triage to General on the Traffic board.
Thu, Nov 15, 10:07 AM · Performance-Team (Radar), Traffic, Analytics-EventLogging, Analytics, Operations
ema moved T209515: Renew Digicert Unified in 2019 from Triage to TLS on the Traffic board.
Thu, Nov 15, 10:06 AM · Operations, Traffic
ema moved T209337: lvs2006 crashed into (what it seems) an unrecoverable state from Triage to LoadBalancer on the Traffic board.
Thu, Nov 15, 10:06 AM · Patch-For-Review, ops-codfw, Operations, Traffic

Mon, Nov 12

ema added a comment to T209021: ATS backend-side request-mangling.

X-MediaWiki-Original is now being set by ATS with a remap rule specific to upload.wikimedia.org:

Mon, Nov 12, 1:46 PM · Patch-For-Review, Operations, Traffic

Fri, Nov 9

ema triaged T209145: Investigate network issues in codfw that caused 503 errors as Normal priority.
Fri, Nov 9, 12:35 PM · Operations, netops
ema created T209145: Investigate network issues in codfw that caused 503 errors.
Fri, Nov 9, 12:35 PM · Operations, netops
ema closed T208588: Add ex cp-misc_codfw to text and upload as Resolved.
Fri, Nov 9, 9:51 AM · Patch-For-Review, Operations, Traffic
ema created P7782 apt-cache show python3-etcd.
Fri, Nov 9, 8:21 AM
ema updated the task description for T209136: python3-conftool needs python3-dns.
Fri, Nov 9, 8:04 AM · Operations, Operations-Software-Development
ema triaged T209136: python3-conftool needs python3-dns as Normal priority.
Fri, Nov 9, 8:02 AM · Operations, Operations-Software-Development
ema created T209136: python3-conftool needs python3-dns.
Fri, Nov 9, 8:02 AM · Operations, Operations-Software-Development

Thu, Nov 8

ema added a comment to T208588: Add ex cp-misc_codfw to text and upload.

cp2006/cp2012 reimaged and added to cache_text. The nodes are currently depooled but ready to be put back into service.

Thu, Nov 8, 5:05 PM · Patch-For-Review, Operations, Traffic
ema updated the task description for T207048: ATS production-ready as a backend cache layer.
Thu, Nov 8, 8:47 AM · Operations, Traffic
ema moved T209021: ATS backend-side request-mangling from Triage to Caching on the Traffic board.
Thu, Nov 8, 8:46 AM · Patch-For-Review, Operations, Traffic
ema updated the task description for T209021: ATS backend-side request-mangling.
Thu, Nov 8, 8:16 AM · Patch-For-Review, Operations, Traffic
ema added a parent task for T209021: ATS backend-side request-mangling: T207048: ATS production-ready as a backend cache layer.
Thu, Nov 8, 8:16 AM · Patch-For-Review, Operations, Traffic
ema added a subtask for T207048: ATS production-ready as a backend cache layer: T209021: ATS backend-side request-mangling.
Thu, Nov 8, 8:16 AM · Operations, Traffic
ema updated the task description for T207048: ATS production-ready as a backend cache layer.
Thu, Nov 8, 8:16 AM · Operations, Traffic
ema triaged T209021: ATS backend-side request-mangling as Normal priority.
Thu, Nov 8, 8:16 AM · Patch-For-Review, Operations, Traffic
ema created T209021: ATS backend-side request-mangling.
Thu, Nov 8, 8:15 AM · Patch-For-Review, Operations, Traffic

Tue, Nov 6

ema changed the status of T204209: Define and deploy Icinga checks for ATS backends from Open to Stalled.

Checking whether the configuration is valid is blocked on https://github.com/apache/trafficserver/issues/4466 being fixed upstream.

Tue, Nov 6, 4:51 PM · Patch-For-Review, Traffic, Operations
ema changed the status of T204209: Define and deploy Icinga checks for ATS backends, a subtask of T207048: ATS production-ready as a backend cache layer, from Open to Stalled.
Tue, Nov 6, 4:51 PM · Operations, Traffic
ema updated the task description for T204209: Define and deploy Icinga checks for ATS backends.
Tue, Nov 6, 4:49 PM · Patch-For-Review, Traffic, Operations
ema updated subscribers of T208717: Requesting access to deployment for WMDE-leszek.

This request was approved during the SRE meeting that took place yesterday, 2018-11-05. CC: @RobH

Tue, Nov 6, 9:47 AM · Patch-For-Review, Operations, SRE-Access-Requests
ema updated subscribers of T208491: Requesting access to deployment for tarrow.

This request was approved during the SRE meeting that took place yesterday, 2018-11-05. CC: @RobH

Tue, Nov 6, 9:47 AM · Patch-For-Review, SRE-Access-Requests, Operations
ema updated subscribers of T208518: Requesting access to deployment for Lucas Werkmeister.

This request was approved during the SRE meeting that took place yesterday, 2018-11-05. CC: @RobH

Tue, Nov 6, 9:47 AM · Patch-For-Review, Operations, SRE-Access-Requests

Mon, Nov 5

ema added a comment to T208717: Requesting access to deployment for WMDE-leszek.

@WMDE-leszek: it looks like you haven't signed L3. Can you please do that? Thanks.

Mon, Nov 5, 2:10 PM · Patch-For-Review, Operations, SRE-Access-Requests
ema updated the task description for T208717: Requesting access to deployment for WMDE-leszek.
Mon, Nov 5, 2:08 PM · Patch-For-Review, Operations, SRE-Access-Requests
ema added a comment to T208491: Requesting access to deployment for tarrow.

@Tarrow: we need manager approval (WMDE), as well as a WMF sponsor. Thanks!

Mon, Nov 5, 2:04 PM · Patch-For-Review, SRE-Access-Requests, Operations
ema updated the task description for T208491: Requesting access to deployment for tarrow.
Mon, Nov 5, 1:44 PM · Patch-For-Review, SRE-Access-Requests, Operations
ema moved T208432: Requesting access to Jupyter notebook / analytics-privatedata-users for jgleeson from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Mon, Nov 5, 1:35 PM · Operations, SRE-Access-Requests
ema added a comment to T208432: Requesting access to Jupyter notebook / analytics-privatedata-users for jgleeson.

@jgleeson: we need your public SSH key (not the wikitech one) as well as manager approval.

Mon, Nov 5, 1:34 PM · Operations, SRE-Access-Requests
ema updated the task description for T208432: Requesting access to Jupyter notebook / analytics-privatedata-users for jgleeson.
Mon, Nov 5, 1:33 PM · Operations, SRE-Access-Requests
ema added a comment to T208518: Requesting access to deployment for Lucas Werkmeister.

Approval from my WMDE manager or a WMF sponsor?

In the past I believe we have kind of needed both.

Mon, Nov 5, 11:37 AM · Patch-For-Review, Operations, SRE-Access-Requests
ema updated the task description for T208491: Requesting access to deployment for tarrow.
Mon, Nov 5, 11:14 AM · Patch-For-Review, SRE-Access-Requests, Operations
ema added a comment to T208518: Requesting access to deployment for Lucas Werkmeister.

We need manager approval and SRE meeting review.

Mon, Nov 5, 11:11 AM · Patch-For-Review, Operations, SRE-Access-Requests
ema updated the task description for T208518: Requesting access to deployment for Lucas Werkmeister.
Mon, Nov 5, 11:09 AM · Patch-For-Review, Operations, SRE-Access-Requests
ema updated the task description for T208518: Requesting access to deployment for Lucas Werkmeister.
Mon, Nov 5, 10:57 AM · Patch-For-Review, Operations, SRE-Access-Requests

Fri, Nov 2

ema triaged T208432: Requesting access to Jupyter notebook / analytics-privatedata-users for jgleeson as Normal priority.
Fri, Nov 2, 2:13 PM · Operations, SRE-Access-Requests
ema moved T208267: Requesting access to netbox for bd808 from Untriaged to In Discussion on the SRE-Access-Requests board.
Fri, Nov 2, 2:11 PM · LDAP-Access-Requests, Operations, SRE-Access-Requests
ema moved T207951: Requesting access to deployment, operational logs, and analytics cluster for jlinehan from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Fri, Nov 2, 2:11 PM · Patch-For-Review, Operations, SRE-Access-Requests
ema moved T208491: Requesting access to deployment for tarrow from Untriaged to SRE Meeting Review Required on the SRE-Access-Requests board.
Fri, Nov 2, 2:11 PM · Patch-For-Review, SRE-Access-Requests, Operations
ema moved T208518: Requesting access to deployment for Lucas Werkmeister from Untriaged to SRE Meeting Review Required on the SRE-Access-Requests board.
Fri, Nov 2, 2:11 PM · Patch-For-Review, Operations, SRE-Access-Requests
ema moved T208533: Access for imarlier to wdqs servers from Untriaged to SRE Meeting Review Required on the SRE-Access-Requests board.
Fri, Nov 2, 2:11 PM · Patch-For-Review, Operations, SRE-Access-Requests
ema updated the task description for T208491: Requesting access to deployment for tarrow.
Fri, Nov 2, 2:10 PM · Patch-For-Review, SRE-Access-Requests, Operations
ema triaged T208533: Access for imarlier to wdqs servers as Normal priority.
Fri, Nov 2, 2:08 PM · Patch-For-Review, Operations, SRE-Access-Requests
ema updated the task description for T208518: Requesting access to deployment for Lucas Werkmeister.
Fri, Nov 2, 2:07 PM · Patch-For-Review, Operations, SRE-Access-Requests
ema moved T208588: Add ex cp-misc_codfw to text and upload from Triage to Caching on the Traffic board.
Fri, Nov 2, 1:50 PM · Patch-For-Review, Operations, Traffic
ema triaged T208588: Add ex cp-misc_codfw to text and upload as Normal priority.
Fri, Nov 2, 1:50 PM · Patch-For-Review, Operations, Traffic
ema created T208588: Add ex cp-misc_codfw to text and upload.
Fri, Nov 2, 1:50 PM · Patch-For-Review, Operations, Traffic
ema triaged T208586: Decommission lvs1007-1012 as Normal priority.
Fri, Nov 2, 1:44 PM · ops-eqiad, decommission, Traffic, Operations
ema moved T208586: Decommission lvs1007-1012 from Triage to Hardware on the Traffic board.
Fri, Nov 2, 1:44 PM · ops-eqiad, decommission, Traffic, Operations
ema moved T208584: Decommission old eqiad caches from Triage to Hardware on the Traffic board.
Fri, Nov 2, 1:44 PM · ops-eqiad, decommission, Operations, Traffic
ema moved T208585: Decommission esams cache_misc hosts from Triage to Hardware on the Traffic board.
Fri, Nov 2, 1:44 PM · ops-esams, decommission, Operations, Traffic
ema triaged T208585: Decommission esams cache_misc hosts as Normal priority.
Fri, Nov 2, 1:44 PM · ops-esams, decommission, Operations, Traffic
ema renamed T208585: Decommission esams cache_misc hosts from Decommission cache_misc hosts to Decommission esams cache_misc hosts.
Fri, Nov 2, 1:43 PM · ops-esams, decommission, Operations, Traffic
ema triaged T208584: Decommission old eqiad caches as Normal priority.
Fri, Nov 2, 1:42 PM · ops-eqiad, decommission, Operations, Traffic
ema renamed T208584: Decommission old eqiad caches from Decommission 18 old eqiad caches to Decommission old eqiad caches.
Fri, Nov 2, 1:42 PM · ops-eqiad, decommission, Operations, Traffic
ema added a project to T208583: Reimage eeden to test role: decommission.
Fri, Nov 2, 1:41 PM · decommission, ops-esams, Operations, Traffic
ema moved T208583: Reimage eeden to test role from Triage to Hardware on the Traffic board.
Fri, Nov 2, 1:41 PM · decommission, ops-esams, Operations, Traffic
ema triaged T208583: Reimage eeden to test role as Normal priority.
Fri, Nov 2, 1:40 PM · decommission, ops-esams, Operations, Traffic
ema created T208586: Decommission lvs1007-1012.
Fri, Nov 2, 1:40 PM · ops-eqiad, decommission, Traffic, Operations
ema created T208585: Decommission esams cache_misc hosts.
Fri, Nov 2, 1:30 PM · ops-esams, decommission, Operations, Traffic
ema created T208584: Decommission old eqiad caches.
Fri, Nov 2, 1:20 PM · ops-eqiad, decommission, Operations, Traffic
ema created T208583: Reimage eeden to test role.
Fri, Nov 2, 1:18 PM · decommission, ops-esams, Operations, Traffic
ema closed T208574: varnish systemd service reaching TaskMax as Resolved.
Fri, Nov 2, 1:13 PM · Patch-For-Review, Traffic, Operations
ema added a comment to T207816: Create URL for Mexico Awareness Campaign.

Patch merged, the site works:

Fri, Nov 2, 11:11 AM · Patch-For-Review, Operations, New-Readers
ema updated subscribers of T208574: varnish systemd service reaching TaskMax.
Fri, Nov 2, 10:56 AM · Patch-For-Review, Traffic, Operations
ema triaged T208574: varnish systemd service reaching TaskMax as High priority.
Fri, Nov 2, 10:51 AM · Patch-For-Review, Traffic, Operations
ema created T208574: varnish systemd service reaching TaskMax.
Fri, Nov 2, 10:51 AM · Patch-For-Review, Traffic, Operations

Thu, Nov 1

ema triaged T208491: Requesting access to deployment for tarrow as Normal priority.
Thu, Nov 1, 2:10 PM · Patch-For-Review, SRE-Access-Requests, Operations
ema triaged T207615: Varnish won't purge thumbnails of specific file as Normal priority.
Thu, Nov 1, 10:02 AM · Operations, Traffic

Wed, Oct 31

ema closed T207852: Requesting access to deployment and analytics-privatedata-users for sbassett as Resolved.

Very well!

Wed, Oct 31, 4:04 PM · Patch-For-Review, User-jijiki, SRE-Access-Requests, Operations
ema added a comment to T207852: Requesting access to deployment and analytics-privatedata-users for sbassett.

@sbassett please try to SSH as sbassett to one of the systems you should now have access to, for example stat1007.eqiad.wmnet.

Wed, Oct 31, 3:33 PM · Patch-For-Review, User-jijiki, SRE-Access-Requests, Operations
ema added a comment to T207951: Requesting access to deployment, operational logs, and analytics cluster for jlinehan.

@jlinehan please try to SSH as jdl to one of the systems you should now have access to, for example stat1007.eqiad.wmnet.

Wed, Oct 31, 3:31 PM · Patch-For-Review, Operations, SRE-Access-Requests
ema added a comment to T207951: Requesting access to deployment, operational logs, and analytics cluster for jlinehan.

Request approved during SRE meeting, 2018-10-29.

Wed, Oct 31, 8:42 AM · Patch-For-Review, Operations, SRE-Access-Requests
ema added a comment to T207852: Requesting access to deployment and analytics-privatedata-users for sbassett.

Request approved during SRE meeting, 2018-10-29.

Wed, Oct 31, 8:25 AM · Patch-For-Review, User-jijiki, SRE-Access-Requests, Operations
ema triaged T208191: Set hhvm.virtual_host[default][always_decode_post_data] = false as Normal priority.
Wed, Oct 31, 7:50 AM · Operations, Wikimedia-Site-requests, HHVM
ema triaged T208284: Increase "check_legal_html" coverage to group0 wikis as Normal priority.
Wed, Oct 31, 7:50 AM · Operations, goodfirstbug, Performance-Team (Radar), monitoring, Release-Engineering-Team
ema triaged T208282: Increase EventLogging limit from 2K to 5K as Normal priority.
Wed, Oct 31, 7:49 AM · Performance-Team (Radar), Traffic, Analytics-EventLogging, Analytics, Operations
ema triaged T208267: Requesting access to netbox for bd808 as Normal priority.
Wed, Oct 31, 7:48 AM · LDAP-Access-Requests, Operations, SRE-Access-Requests
ema triaged T208257: stop using mod_php anywhere as Normal priority.
Wed, Oct 31, 7:46 AM · Patch-For-Review, Operations
ema triaged T208240: Ensure jenkins on puppet.git checks for yaml syntax errors as Normal priority.
Wed, Oct 31, 7:44 AM · Patch-For-Review, Continuous-Integration-Config, Operations

Tue, Oct 30

ema moved T207615: Varnish won't purge thumbnails of specific file from Triage to Caching on the Traffic board.
Tue, Oct 30, 2:54 PM · Operations, Traffic
ema closed T206950: Icinga: check_confd_vcl_reload unknown when file is missing as Resolved.

Fixed: https://gerrit.wikimedia.org/r/470353

Tue, Oct 30, 11:35 AM · Patch-For-Review, monitoring, Traffic, Operations
ema closed T206950: Icinga: check_confd_vcl_reload unknown when file is missing, a subtask of T206861: Power incident in eqsin, as Resolved.
Tue, Oct 30, 11:35 AM · Wikimedia-Incident, Operations, Traffic

Mon, Oct 29

ema created P7730 stats_over_http-v8.0.0.json.
Mon, Oct 29, 9:11 AM

Fri, Oct 26

ema moved T207140: Add maint-announce@ to Equinix's recipient list for eqsin incidents from Triage to General on the Traffic board.
Fri, Oct 26, 8:58 AM · Wikimedia-Incident, Traffic, Operations
ema moved T207668: Increase network capacity (2018-19 Q2 Goal) from Triage to Network on the Traffic board.
Fri, Oct 26, 8:57 AM · Goal, Traffic, netops, Operations
ema added a project to T207947: Switch wdqs1003 with one of the internal wdqs cluster: Operations.
Fri, Oct 26, 7:26 AM · Operations, Patch-For-Review, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
ema added a comment to T207947: Switch wdqs1003 with one of the internal wdqs cluster.

You can see which hosts are part of which service on the PyBal Grafana dashboard. The primary LVS serving wdqs and wdqs-internal is lvs1016.eqiad.wmnet, the secondary is lvs1006.wikimedia.org.

Fri, Oct 26, 7:24 AM · Operations, Patch-For-Review, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata

Wed, Oct 24

ema triaged T207845: debdeploy: show help message if invoked with no arguments as Low priority.
Wed, Oct 24, 1:04 PM · Operations, Operations-Software-Development
ema created T207845: debdeploy: show help message if invoked with no arguments.
Wed, Oct 24, 1:03 PM · Operations, Operations-Software-Development