Page MenuHomePhabricator

Vgutierrez (Valentín Gutiérrez)
Traffic Security Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Feb 12 2018, 9:51 AM (112 w, 2 d)
Availability
Available
IRC Nick
vgutierrez
LDAP User
Vgutierrez
MediaWiki User
Unknown

Recent Activity

Yesterday

Vgutierrez awarded T249344: varnishd crashes in vbf_stp_condfetch(): cp3057 and cp3061 a Love token.
Tue, Apr 7, 6:18 AM · Patch-For-Review, Operations, Traffic

Fri, Apr 3

Vgutierrez renamed T249335: Memory leak on ats-tls 8.0.6 from cp1075 + cp1081 being Pybal-depooled/repooled frequently to Memory leak on ats-tls 8.0.6.
Fri, Apr 3, 2:45 PM · Patch-For-Review, Operations, Traffic
Vgutierrez added a comment to T249335: Memory leak on ats-tls 8.0.6.

as it can be seen in https://grafana.wikimedia.org/d/80zd3mjZk/t249335?orgId=1 it looks like there is a memory leak on ats-tls that at some point begins to hit negatively on performance

Fri, Apr 3, 2:19 PM · Patch-For-Review, Operations, Traffic
Vgutierrez triaged T249335: Memory leak on ats-tls 8.0.6 as Medium priority.
Fri, Apr 3, 1:48 PM · Patch-For-Review, Operations, Traffic
Vgutierrez closed T249280: 503 error on enwikinews as Resolved.

Thanks for you report. a 503 error usually signals a transient issue. Please reopen this task if you experience this issue frequently.

Fri, Apr 3, 6:17 AM · User-DannyS712, Traffic, Operations

Thu, Apr 2

Vgutierrez closed T245616: Provide a simple and automated SSL Ticket key generation system for ATS, a subtask of T245502: ATS TLS session cache efficiency reduced in TLSv1.3, as Resolved.
Thu, Apr 2, 7:33 AM · Traffic, Operations
Vgutierrez closed T245616: Provide a simple and automated SSL Ticket key generation system for ATS as Resolved.
Thu, Apr 2, 7:33 AM · Traffic, Operations
Vgutierrez closed T245502: ATS TLS session cache efficiency reduced in TLSv1.3, a subtask of T170567: Support TLSv1.3, as Resolved.
Thu, Apr 2, 7:33 AM · Performance-Team (Radar), Goal, Patch-For-Review, Traffic, Operations
Vgutierrez closed T245502: ATS TLS session cache efficiency reduced in TLSv1.3 as Resolved.
Thu, Apr 2, 7:33 AM · Traffic, Operations
Vgutierrez added a comment to T245502: ATS TLS session cache efficiency reduced in TLSv1.3.

This has been mitigated by providing support for TLS Session Tickets (T245616) and reducing the number of issued tickets on new connections from 2 to 1 by submitting this patch to ATS: https://github.com/apache/trafficserver/pull/6424.

Thu, Apr 2, 7:33 AM · Traffic, Operations
Vgutierrez added a comment to T238305: Servers freezing across the caching cluster (November 2019).

@faidon actually the cp hosts are running buster (T242093) since February 13th. I do believe we haven't seen more occurrences of this issue on the cache cluster since the upgrade

Thu, Apr 2, 5:37 AM · Operations, Traffic

Wed, Apr 1

Vgutierrez closed T248816: Replace cp20[01-26] with cp20[27-42] as Resolved.
Wed, Apr 1, 4:40 PM · Operations, Traffic
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Wed, Apr 1, 4:40 PM · Operations, Traffic
Vgutierrez reopened T248816: Replace cp20[01-26] with cp20[27-42] as "Open".
Wed, Apr 1, 3:52 PM · Operations, Traffic
Vgutierrez renamed T249125: decommission cp20[16,19,23].codfw.wmnet from decommission cp20[16,19,23,27].codfw.wmnet to decommission cp20[16,19,23].codfw.wmnet.
Wed, Apr 1, 3:51 PM · Operations, Traffic, ops-codfw, decommission
Vgutierrez closed T248816: Replace cp20[01-26] with cp20[27-42] as Resolved.
Wed, Apr 1, 3:47 PM · Operations, Traffic
Vgutierrez moved T249125: decommission cp20[16,19,23].codfw.wmnet from Triage to Hardware on the Traffic board.
Wed, Apr 1, 3:47 PM · Operations, Traffic, ops-codfw, decommission
Vgutierrez reassigned T249125: decommission cp20[16,19,23].codfw.wmnet from Vgutierrez to Papaul.
Wed, Apr 1, 3:46 PM · Operations, Traffic, ops-codfw, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Wed, Apr 1, 3:15 PM · Operations, Traffic
Vgutierrez created T249125: decommission cp20[16,19,23].codfw.wmnet.
Wed, Apr 1, 3:15 PM · Operations, Traffic, ops-codfw, decommission
Vgutierrez reassigned T249115: decommission cp20[18,20,22,24-26].codfw.wmnet from Vgutierrez to Papaul.
Wed, Apr 1, 3:08 PM · ops-codfw, Traffic, decommission, Operations
Vgutierrez moved T249115: decommission cp20[18,20,22,24-26].codfw.wmnet from Triage to Hardware on the Traffic board.
Wed, Apr 1, 2:53 PM · ops-codfw, Traffic, decommission, Operations
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Wed, Apr 1, 2:34 PM · Operations, Traffic
Vgutierrez created T249115: decommission cp20[18,20,22,24-26].codfw.wmnet.
Wed, Apr 1, 2:34 PM · ops-codfw, Traffic, decommission, Operations
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Wed, Apr 1, 2:31 PM · Operations, Traffic
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Wed, Apr 1, 12:23 PM · Operations, Traffic
Vgutierrez reassigned T249088: decommission cp2013.codfw.wmnet from Vgutierrez to Papaul.
Wed, Apr 1, 12:23 PM · ops-codfw, Traffic, decommission, Operations
Vgutierrez created T249088: decommission cp2013.codfw.wmnet.
Wed, Apr 1, 8:59 AM · ops-codfw, Traffic, decommission, Operations
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Wed, Apr 1, 8:31 AM · Operations, Traffic
Vgutierrez reassigned T249084: decommission cp2017.codfw.wmnet from Vgutierrez to Papaul.
Wed, Apr 1, 8:31 AM · Operations, ops-codfw, Traffic, decommission
Vgutierrez moved T249084: decommission cp2017.codfw.wmnet from Triage to Hardware on the Traffic board.
Wed, Apr 1, 8:27 AM · Operations, ops-codfw, Traffic, decommission
Vgutierrez created T249084: decommission cp2017.codfw.wmnet.
Wed, Apr 1, 7:58 AM · Operations, ops-codfw, Traffic, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Wed, Apr 1, 7:34 AM · Operations, Traffic
Vgutierrez reassigned T249080: decommission cp2012.codfw.wmnet from Vgutierrez to Papaul.
Wed, Apr 1, 7:34 AM · Traffic, ops-codfw, Operations, decommission
Vgutierrez created T249080: decommission cp2012.codfw.wmnet.
Wed, Apr 1, 6:31 AM · Traffic, ops-codfw, Operations, decommission

Tue, Mar 31

Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Tue, Mar 31, 4:15 PM · Operations, Traffic
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Tue, Mar 31, 3:50 PM · Operations, Traffic
Vgutierrez reassigned T249009: decommission cp2014.codfw.wmnet from Vgutierrez to Papaul.
Tue, Mar 31, 3:49 PM · Traffic, ops-codfw, Operations, decommission
Vgutierrez moved T249009: decommission cp2014.codfw.wmnet from Triage to Hardware on the Traffic board.
Tue, Mar 31, 3:37 PM · Traffic, ops-codfw, Operations, decommission
Vgutierrez created T249009: decommission cp2014.codfw.wmnet.
Tue, Mar 31, 3:36 PM · Traffic, ops-codfw, Operations, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Tue, Mar 31, 3:35 PM · Operations, Traffic
Vgutierrez reassigned T249002: decommission cp2010.codfw.wmnet from Vgutierrez to Papaul.
Tue, Mar 31, 3:35 PM · Traffic, ops-codfw, Operations, DC-Ops, decommission
Vgutierrez moved T249002: decommission cp2010.codfw.wmnet from Triage to Hardware on the Traffic board.
Tue, Mar 31, 3:22 PM · Traffic, ops-codfw, Operations, DC-Ops, decommission
Vgutierrez created T249002: decommission cp2010.codfw.wmnet.
Tue, Mar 31, 3:21 PM · Traffic, ops-codfw, Operations, DC-Ops, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Tue, Mar 31, 3:05 PM · Operations, Traffic
Vgutierrez added a comment to T242952: traffic_server crash upon Lua reload: attempt to concatenate a table value.

https://github.com/apache/trafficserver/pull/6571 could be handy to tune some TS lua aspects

Tue, Mar 31, 1:52 PM · Operations, Traffic
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Tue, Mar 31, 9:14 AM · Operations, Traffic
Vgutierrez reassigned T248950: decommission cp2011.codfw.wmnet from Vgutierrez to Papaul.
Tue, Mar 31, 9:14 AM · Traffic, ops-codfw, Operations, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Tue, Mar 31, 8:56 AM · Operations, Traffic
Vgutierrez created T248950: decommission cp2011.codfw.wmnet.
Tue, Mar 31, 7:48 AM · Traffic, ops-codfw, Operations, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Tue, Mar 31, 7:17 AM · Operations, Traffic
Vgutierrez reassigned T248941: decommission cp2007.codfw.wmnet from Vgutierrez to Papaul.
Tue, Mar 31, 6:59 AM · ops-codfw, Traffic, Operations, decommission
Vgutierrez moved T248941: decommission cp2007.codfw.wmnet from Triage to Hardware on the Traffic board.
Tue, Mar 31, 5:45 AM · ops-codfw, Traffic, Operations, decommission
Vgutierrez created T248941: decommission cp2007.codfw.wmnet.
Tue, Mar 31, 5:38 AM · ops-codfw, Traffic, Operations, decommission
Vgutierrez created P10823 (An Untitled Masterwork).
Tue, Mar 31, 5:28 AM
Vgutierrez added a comment to T248938: ATS ts_lua coredumps on config reload.

This issue seems to be identified by upstream at https://github.com/apache/trafficserver/pull/6403 but the fix hasn't been backported to ATS 8.x

Tue, Mar 31, 4:58 AM · Patch-For-Review, Traffic, Operations
Vgutierrez triaged T248938: ATS ts_lua coredumps on config reload as Medium priority.
TSRemapDeleteInstance stacktrace
Mar 30 12:07:56 cp2013 traffic_manager[32876]: traffic_server: received signal 11 (Segmentation fault)
Mar 30 12:07:56 cp2013 traffic_manager[32876]: traffic_server - STACK TRACE:
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x55744cb7b010]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f03732ec730]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /lib/x86_64-linux-gnu/libluajit-5.1.so.2(+0x1627d)[0x7f029100f27d]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /lib/x86_64-linux-gnu/libluajit-5.1.so.2(+0xbe36)[0x7f0291004e36]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /lib/x86_64-linux-gnu/libluajit-5.1.so.2(+0x368f0)[0x7f029102f8f0]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /lib/x86_64-linux-gnu/libluajit-5.1.so.2(lua_gc+0xd8)[0x7f0291050638]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/lib/trafficserver/modules/tslua.so(+0x1338a)[0x7f033001938a]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/lib/trafficserver/modules/tslua.so(TSRemapDeleteInstance+0x16)[0x7f033000ee56]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN11url_mappingD1Ev+0xec)[0x55744cc5de3c]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN4TrieI11url_mappingE5ClearEv+0x28)[0x55744cc65c08]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN19UrlMappingPathIndexD2Ev+0x4b)[0x55744cc64efb]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN19UrlMappingPathIndexD0Ev+0x9)[0x55744cc64f99]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN10UrlRewriteD1Ev+0x7d)[0x55744cc5e72d]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN10UrlRewriteD0Ev+0x9)[0x55744cc5ec59]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN19DeleterContinuationI10UrlRewriteE8dieEventEiPv+0x13)[0x55744cbfc6e3]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x55744ce60112]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN7EThread13process_queueEP5QueueI5EventNS1_9Link_linkEEPiS5_+0x27e)[0x55744ce60b1e]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x18f)[0x55744ce60fff]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /usr/bin/traffic_server(+0x3ad9fa)[0x55744ce5f9fa]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f03732e1fa3]
Mar 30 12:07:56 cp2013 traffic_manager[32876]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f0372eea4cf]
Tue, Mar 31, 4:53 AM · Patch-For-Review, Traffic, Operations
Vgutierrez created T248938: ATS ts_lua coredumps on config reload.
Tue, Mar 31, 4:49 AM · Patch-For-Review, Traffic, Operations

Mon, Mar 30

Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 3:34 PM · Operations, Traffic
Vgutierrez added a comment to T248864: decommission cp2008.codfw.wmnet.

@Papaul in https://netbox.wikimedia.org/dcim/devices/679/ is marked as "Decommissioning" not "active"

Mon, Mar 30, 3:30 PM · ops-codfw, Traffic, Operations, DC-Ops, decommission
Vgutierrez updated subscribers of T248864: decommission cp2008.codfw.wmnet.

hmmm @Volans apparently cp2008 is still active in netbox but the cookbook logged Set Netbox status to Decommissioning

Mon, Mar 30, 3:28 PM · ops-codfw, Traffic, Operations, DC-Ops, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 3:06 PM · Operations, Traffic
Vgutierrez reassigned T248864: decommission cp2008.codfw.wmnet from Vgutierrez to Papaul.
Mon, Mar 30, 3:05 PM · ops-codfw, Traffic, Operations, DC-Ops, decommission
Vgutierrez created T248864: decommission cp2008.codfw.wmnet.
Mon, Mar 30, 2:37 PM · ops-codfw, Traffic, Operations, DC-Ops, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 2:25 PM · Operations, Traffic
Vgutierrez reassigned T248856: decommission cp2006.codfw.wmnet from Vgutierrez to Papaul.
Mon, Mar 30, 2:25 PM · ops-codfw, Traffic, Operations, DC-Ops, decommission
Vgutierrez moved T248856: decommission cp2006.codfw.wmnet from Triage to Hardware on the Traffic board.
Mon, Mar 30, 2:25 PM · ops-codfw, Traffic, Operations, DC-Ops, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 1:47 PM · Operations, Traffic
Vgutierrez created T248856: decommission cp2006.codfw.wmnet.
Mon, Mar 30, 1:47 PM · ops-codfw, Traffic, Operations, DC-Ops, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 1:04 PM · Operations, Traffic
Vgutierrez moved T248848: decommission cp2005.codfw.wmnet from Triage to Hardware on the Traffic board.
Mon, Mar 30, 1:03 PM · ops-codfw, Traffic, Operations, decommission
Vgutierrez reassigned T248848: decommission cp2005.codfw.wmnet from Vgutierrez to Papaul.
Mon, Mar 30, 1:03 PM · ops-codfw, Traffic, Operations, decommission
Vgutierrez reassigned T248824: decommission cp2004.codfw.wmnet from Vgutierrez to Papaul.
Mon, Mar 30, 12:42 PM · ops-codfw, Traffic, Operations, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 12:35 PM · Operations, Traffic
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 12:16 PM · Operations, Traffic
Vgutierrez created T248848: decommission cp2005.codfw.wmnet.
Mon, Mar 30, 12:16 PM · ops-codfw, Traffic, Operations, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 8:59 AM · Operations, Traffic
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 8:58 AM · Operations, Traffic
Vgutierrez claimed T248824: decommission cp2004.codfw.wmnet.
Mon, Mar 30, 8:57 AM · ops-codfw, Traffic, Operations, decommission
Vgutierrez created T248824: decommission cp2004.codfw.wmnet.
Mon, Mar 30, 8:57 AM · ops-codfw, Traffic, Operations, decommission
ema awarded T248736: ats-tls ran out of FDs on cp1089 a Pterodactyl token.
Mon, Mar 30, 8:44 AM · Operations, Traffic
Vgutierrez reassigned T248818: decommission cp2002.codfw.wmnet from Vgutierrez to Papaul.
Mon, Mar 30, 8:33 AM · Traffic, ops-codfw, Operations, decommission
Vgutierrez updated the task description for T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 8:16 AM · Operations, Traffic
Vgutierrez created T248818: decommission cp2002.codfw.wmnet.
Mon, Mar 30, 7:43 AM · Traffic, ops-codfw, Operations, decommission
Vgutierrez reassigned T248815: decommission cp2001.codfw.wmnet from Vgutierrez to Papaul.
Mon, Mar 30, 7:37 AM · Traffic, ops-codfw, Operations, decommission
Vgutierrez triaged T248816: Replace cp20[01-26] with cp20[27-42] as Medium priority.
Mon, Mar 30, 7:09 AM · Operations, Traffic
Vgutierrez created T248816: Replace cp20[01-26] with cp20[27-42].
Mon, Mar 30, 7:08 AM · Operations, Traffic
Vgutierrez moved T248815: decommission cp2001.codfw.wmnet from Triage to Hardware on the Traffic board.
Mon, Mar 30, 6:16 AM · Traffic, ops-codfw, Operations, decommission
Vgutierrez created T248815: decommission cp2001.codfw.wmnet.
Mon, Mar 30, 6:15 AM · Traffic, ops-codfw, Operations, decommission

Sat, Mar 28

Vgutierrez added a comment to T248736: ats-tls ran out of FDs on cp1089.

@ema we need to check if this can be related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/583295

Sat, Mar 28, 8:58 AM · Operations, Traffic
Vgutierrez triaged T248736: ats-tls ran out of FDs on cp1089 as Medium priority.
Sat, Mar 28, 8:54 AM · Operations, Traffic
Vgutierrez added a comment to T248736: ats-tls ran out of FDs on cp1089.

even though ats-tls reports the same amount of connections being used since 12:00, from the memory graph data, it looks like ats-tls is the process leaking the sockets:


Sat, Mar 28, 8:54 AM · Operations, Traffic
Vgutierrez added a comment to T248736: ats-tls ran out of FDs on cp1089.

cp1089 shows a ramp up of inuse TCP sockets that began yesterday around 12:00:

Sat, Mar 28, 8:43 AM · Operations, Traffic
Vgutierrez created T248736: ats-tls ran out of FDs on cp1089.
Sat, Mar 28, 8:40 AM · Operations, Traffic

Thu, Mar 26

Vgutierrez created P10790 (An Untitled Masterwork).
Thu, Mar 26, 4:53 PM
Vgutierrez created P10789 (An Untitled Masterwork).
Thu, Mar 26, 3:50 PM

Wed, Mar 18

Vgutierrez created P10720 (An Untitled Masterwork).
Wed, Mar 18, 3:28 PM

Fri, Mar 13

Vgutierrez triaged T247619: Requesting new gerrit project repository "operations/software/ncmonitor" as Medium priority.
Fri, Mar 13, 4:22 PM · User-MarcoAurelio, Repository-Admins
Vgutierrez added a subtask for T247618: Track WMF owned non-canonical domains: T247619: Requesting new gerrit project repository "operations/software/ncmonitor".
Fri, Mar 13, 4:21 PM · Operations, Traffic
Vgutierrez added a parent task for T247619: Requesting new gerrit project repository "operations/software/ncmonitor": T247618: Track WMF owned non-canonical domains.
Fri, Mar 13, 4:21 PM · User-MarcoAurelio, Repository-Admins