Vgutierrez (Valentín Gutiérrez)
Traffic Security Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Feb 12 2018, 9:51 AM (26 w, 22 h)
Availability
Available
IRC Nick
vgutierrez
LDAP User
Vgutierrez
MediaWiki User
Unknown

Recent Activity

Yesterday

Vgutierrez committed rOSCCd906af82dee8: [WIP] Refactor certcentral.certificate_management() (authored by Vgutierrez).
[WIP] Refactor certcentral.certificate_management()
Mon, Aug 13, 4:35 PM
Vgutierrez committed rOSCCed8f0e8d4830: [WIP] Refactor certcentral.certificate_management() (authored by Vgutierrez).
[WIP] Refactor certcentral.certificate_management()
Mon, Aug 13, 4:35 PM

Fri, Aug 10

Vgutierrez committed rOSCC8398f45feacc: Replace acme_tiny with acme_requests (authored by Vgutierrez).
Replace acme_tiny with acme_requests
Fri, Aug 10, 4:24 PM
Vgutierrez committed rOSCC125ce67343a2: Replace acme_tiny with acme_requests (authored by Vgutierrez).
Replace acme_tiny with acme_requests
Fri, Aug 10, 3:05 PM
Vgutierrez committed rOSCC5bb58f4ad216: [WIP] CertCentral tests (authored by Vgutierrez).
[WIP] CertCentral tests
Fri, Aug 10, 3:05 PM
Vgutierrez committed rOSCC7ceee985380c: [WIP] CertCentral tests (authored by Vgutierrez).
[WIP] CertCentral tests
Fri, Aug 10, 3:05 PM
Vgutierrez committed rOSCC81459181738c: Replace acme_tiny with acme_requests (authored by Vgutierrez).
Replace acme_tiny with acme_requests
Fri, Aug 10, 3:05 PM
Vgutierrez committed rOSCC580e721aad57: [WIP] CertCentral tests (authored by Vgutierrez).
[WIP] CertCentral tests
Fri, Aug 10, 3:00 PM
Vgutierrez committed rOSCCc746a16fdb6e: Replace acme_tiny with acme_requests (authored by Vgutierrez).
Replace acme_tiny with acme_requests
Fri, Aug 10, 3:00 PM
Tgr awarded Blog Post: Wikipedia goes 100% Forward Secret a Love token.
Fri, Aug 10, 11:10 AM · Traffic

Thu, Aug 9

Vgutierrez committed rOSCCa7da6f29d309: Refactor CertCentral API (authored by Vgutierrez).
Refactor CertCentral API
Thu, Aug 9, 12:27 PM
Vgutierrez committed rOSCCea898588fd14: Refactor CertCentral API (authored by Vgutierrez).
Refactor CertCentral API
Thu, Aug 9, 12:19 PM
Vgutierrez committed rOSCC4ea6e6d0c3cd: Refactor CertCentral API (authored by Vgutierrez).
Refactor CertCentral API
Thu, Aug 9, 12:15 PM
Vgutierrez committed rOSCC01cc12221425: Move get_certs out of CertCentral class (authored by Vgutierrez).
Move get_certs out of CertCentral class
Thu, Aug 9, 12:08 PM
Vgutierrez committed rOSCCe4104be8d457: Move get_certs out of CertCentral class (authored by Vgutierrez).
Move get_certs out of CertCentral class
Thu, Aug 9, 12:05 PM
Vgutierrez added a comment to T196560: rack/setup/install LVS200[7-10].

@ayounsi interface naming in lvs2009 and lvs2010:

current namelvs2009lvs2010
nic1enp59s0f0enp59s0f0
nic2enp59s0f1d1enp59s0f1d1
nic3enp175s0f0enp175s0f0
nic4enp175s0f1d1enp175s0f1d1
Thu, Aug 9, 9:39 AM · Patch-For-Review, ops-codfw, Traffic, Operations
Vgutierrez moved T198286: Decommission acamar and achernar from Triage to Hardware on the Traffic board.
Thu, Aug 9, 9:25 AM · decommission, Traffic, Operations, ops-codfw
Vgutierrez edited projects for T198286: Decommission acamar and achernar, added: Traffic, decommission; removed Patch-For-Review.
Thu, Aug 9, 9:24 AM · decommission, Traffic, Operations, ops-codfw
Vgutierrez added a comment to T196560: rack/setup/install LVS200[7-10].

@Papaul in lvs2009 on board NICs need to be disabled in the BIOS (in lvs2010 they're already disabled):

lvs2009
root@lvs2009:~# dmesg |grep tg3
[    2.524752] tg3.c:v3.137 (May 11, 2014)
[    2.545435] tg3 0000:04:00.0 eth4: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:59:eb:ab
[    2.545440] tg3 0000:04:00.0 eth4: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.545443] tg3 0000:04:00.0 eth4: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.545447] tg3 0000:04:00.0 eth4: dma_rwctrl[00000001] dma_mask[64-bit]
[    2.564921] tg3 0000:04:00.1 eth5: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:59:eb:ac
[    2.564926] tg3 0000:04:00.1 eth5: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[    2.564929] tg3 0000:04:00.1 eth5: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[    2.564933] tg3 0000:04:00.1 eth5: dma_rwctrl[00000001] dma_mask[64-bit]
[    2.790436] tg3 0000:04:00.1 eno2: renamed from eth5
[    2.834331] tg3 0000:04:00.0 eno1: renamed from eth4
lvs2010
vgutierrez@lvs2010:~$ sudo -i dmesg |grep tg3
vgutierrez@lvs2010:~$
Thu, Aug 9, 9:11 AM · Patch-For-Review, ops-codfw, Traffic, Operations
Vgutierrez updated the task description for T201522: Decommission chromium and hydrogen.
Thu, Aug 9, 9:08 AM · decommission, Traffic, ops-eqiad, Operations

Wed, Aug 8

Vgutierrez moved T201522: Decommission chromium and hydrogen from Triage to Hardware on the Traffic board.
Wed, Aug 8, 3:41 PM · decommission, Traffic, ops-eqiad, Operations
Vgutierrez added a project to T201522: Decommission chromium and hydrogen: Traffic.
Wed, Aug 8, 3:41 PM · decommission, Traffic, ops-eqiad, Operations
Vgutierrez created T201522: Decommission chromium and hydrogen.
Wed, Aug 8, 3:40 PM · decommission, Traffic, ops-eqiad, Operations
Vgutierrez closed T201414: Use dns100[12] as ntp servers in eqiad networking equipment as Resolved.
Wed, Aug 8, 3:20 PM · netops, Traffic, Operations
Vgutierrez closed T201414: Use dns100[12] as ntp servers in eqiad networking equipment, a subtask of T196691: rack/setup/install dns100[12].wikimedia.org, as Resolved.
Wed, Aug 8, 3:20 PM · Patch-For-Review, DNS, Operations, Traffic
Vgutierrez added a comment to T201414: Use dns100[12] as ntp servers in eqiad networking equipment.

Awesome, thanks!

Wed, Aug 8, 3:20 PM · netops, Traffic, Operations
Vgutierrez committed rOSCC2ca83980698e: Move get_certs out of CertCentral class (authored by Vgutierrez).
Move get_certs out of CertCentral class
Wed, Aug 8, 3:17 PM
Vgutierrez committed rOSCC881fa2afa58d: [WIP] Move get_certs out of CertCentral class (authored by Vgutierrez).
[WIP] Move get_certs out of CertCentral class
Wed, Aug 8, 1:07 PM
Vgutierrez committed rOSCC3c0bcda2b5fa: [WIP] Move get_certs out of CertCentral class (authored by Vgutierrez).
[WIP] Move get_certs out of CertCentral class
Wed, Aug 8, 1:04 PM
Vgutierrez committed rOSCC61cc60db118f: [WIP] Move get_certs out of CertCentral class (authored by Vgutierrez).
[WIP] Move get_certs out of CertCentral class
Wed, Aug 8, 12:53 PM
Vgutierrez committed rOSCC1c435ec94feb: [WIP] Move get_certs out of CertCentral class (authored by Vgutierrez).
[WIP] Move get_certs out of CertCentral class
Wed, Aug 8, 11:21 AM
Vgutierrez committed rOSCC1f05854a58a3: [WIP] Move get_certs out of CertCentral class (authored by Vgutierrez).
[WIP] Move get_certs out of CertCentral class
Wed, Aug 8, 10:30 AM
Vgutierrez committed rOSCCaef369730f43: [WIP] Move get_certs out of CertCentral class (authored by Vgutierrez).
[WIP] Move get_certs out of CertCentral class
Wed, Aug 8, 10:15 AM
Vgutierrez committed rOSCC8ed4ce4d1090: [WIP] Move get_certs out of CertCentral class (authored by Vgutierrez).
[WIP] Move get_certs out of CertCentral class
Wed, Aug 8, 10:00 AM

Tue, Aug 7

Ottomata awarded Blog Post: Wikipedia goes 100% Forward Secret a Yellow Medal token.
Tue, Aug 7, 6:12 PM · Traffic
MarcoAurelio awarded Blog Post: Wikipedia goes 100% Forward Secret a Mountain of Wealth token.
Tue, Aug 7, 1:47 PM · Traffic
Vgutierrez triaged T201414: Use dns100[12] as ntp servers in eqiad networking equipment as Normal priority.
Tue, Aug 7, 1:16 PM · netops, Traffic, Operations
Vgutierrez updated the task description for T196691: rack/setup/install dns100[12].wikimedia.org.
Tue, Aug 7, 10:38 AM · Patch-For-Review, DNS, Operations, Traffic
phuedx awarded Blog Post: Wikipedia goes 100% Forward Secret a Mountain of Wealth token.
Tue, Aug 7, 5:21 AM · Traffic

Mon, Aug 6

Vgutierrez claimed T196691: rack/setup/install dns100[12].wikimedia.org.
Mon, Aug 6, 4:51 PM · Patch-For-Review, DNS, Operations, Traffic
Vgutierrez added a comment to T196691: rack/setup/install dns100[12].wikimedia.org.

I'll handle this from here @RobH, thanks :)

Mon, Aug 6, 4:50 PM · Patch-For-Review, DNS, Operations, Traffic
Vgutierrez committed rOSCC467534f2a2f4: Add self_signed property on Certificate class (authored by Vgutierrez).
Add self_signed property on Certificate class
Mon, Aug 6, 4:17 PM
Vgutierrez committed rOSCCd9153ba1a4b7: Allow CSR generation without Subject Alternative Names (authored by Vgutierrez).
Allow CSR generation without Subject Alternative Names
Mon, Aug 6, 3:59 PM
Vgutierrez committed rOSCC18fe4c430b39: Implement config file parsing outside CertCentral class (authored by Vgutierrez).
Implement config file parsing outside CertCentral class
Mon, Aug 6, 12:21 PM
Vgutierrez closed T199717: Pick up a suitable ACME library for certcentral as Resolved.
Mon, Aug 6, 9:22 AM · Traffic, Operations
Vgutierrez closed T199717: Pick up a suitable ACME library for certcentral, a subtask of T199711: Deploy a scalable service for ACME (LetsEncrypt) certificate management, as Resolved.
Mon, Aug 6, 9:22 AM · Traffic, Operations, Goal
Vgutierrez committed rOSCC1c714f4115a7: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
provide ACMEv2 support based on certbot/acme library
Mon, Aug 6, 9:17 AM

Fri, Aug 3

Vgutierrez published Blog Post: Wikipedia goes 100% Forward Secret.
Fri, Aug 3, 7:19 PM · Traffic
Vgutierrez committed rOSCCc627ef661ad5: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
provide ACMEv2 support based on certbot/acme library
Fri, Aug 3, 3:51 PM
Vgutierrez committed rOSCC1de5154d2032: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Fri, Aug 3, 3:41 PM
Vgutierrez committed rOSCC31eb03a4896b: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Fri, Aug 3, 3:31 PM
Vgutierrez committed rOSCC01b5182c3c0f: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Fri, Aug 3, 1:01 PM
Vgutierrez committed rOSCCd08c0b0a0bac: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Fri, Aug 3, 10:36 AM
Vgutierrez committed rOSCCe24d7286fd97: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Fri, Aug 3, 10:00 AM

Thu, Aug 2

Vgutierrez committed rOSCC1a53ffc335de: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Thu, Aug 2, 3:33 PM

Mon, Jul 30

Vgutierrez updated the task description for T192555: Begin execution of non-forward-secret ciphers deprecation.
Mon, Jul 30, 1:06 PM · Patch-For-Review, Goal, Operations, Traffic

Fri, Jul 27

Vgutierrez closed T200405: Provide a CI container with pebble as Resolved.
Fri, Jul 27, 3:56 PM · Continuous-Integration-Config, Traffic, Operations
Vgutierrez closed T200405: Provide a CI container with pebble, a subtask of T199717: Pick up a suitable ACME library for certcentral, as Resolved.
Fri, Jul 27, 3:56 PM · Traffic, Operations
Vgutierrez committed rOSCC283cae326b15: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Fri, Jul 27, 3:36 PM
Vgutierrez committed rOSCCd3d7b2e90d2e: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Fri, Jul 27, 12:53 PM

Thu, Jul 26

Vgutierrez committed rOSCCc09b4e1a79b4: Provide a valid pebble config for the integration tests (authored by Vgutierrez).
Provide a valid pebble config for the integration tests
Thu, Jul 26, 12:27 PM
Vgutierrez committed rOSCCd8e6a5404b49: Provide a valid pebble config for the integration tests (authored by Vgutierrez).
Provide a valid pebble config for the integration tests
Thu, Jul 26, 10:59 AM
Vgutierrez committed rOSCC13fae65d6e7b: Provide a valid pebble config for the integration tests (authored by Vgutierrez).
Provide a valid pebble config for the integration tests
Thu, Jul 26, 10:53 AM
Vgutierrez committed rOSCC94eded64ea2a: Provide a valid pebble config for the integration tests (authored by Vgutierrez).
Provide a valid pebble config for the integration tests
Thu, Jul 26, 10:53 AM
Vgutierrez created P7388 building pebble.
Thu, Jul 26, 8:16 AM
Vgutierrez moved T200405: Provide a CI container with pebble from Triage to TLS on the Traffic board.
Thu, Jul 26, 7:58 AM · Continuous-Integration-Config, Traffic, Operations
Vgutierrez triaged T200405: Provide a CI container with pebble as Normal priority.
Thu, Jul 26, 7:57 AM · Continuous-Integration-Config, Traffic, Operations

Wed, Jul 25

Vgutierrez committed rOSCCfb5074b89ad5: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Wed, Jul 25, 11:00 AM
Vgutierrez committed rOSCCa5f5fc80e64b: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Wed, Jul 25, 10:58 AM
Vgutierrez committed rOSCC0b003ecb8801: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Wed, Jul 25, 10:10 AM
Vgutierrez committed rOSCC12d1d6129da0: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Wed, Jul 25, 10:03 AM

Tue, Jul 24

Vgutierrez committed rOSCC6f9e8c109e15: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Tue, Jul 24, 4:03 PM
Vgutierrez committed rOSCC5ab22d97e89d: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Tue, Jul 24, 3:50 PM
Vgutierrez committed rOSCC7d7e56c9b158: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Tue, Jul 24, 3:11 PM

Mon, Jul 23

Vgutierrez committed rOSCCecda9e3ad0f5: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Mon, Jul 23, 12:55 PM
Vgutierrez committed rOSCC6f72282707b3: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Mon, Jul 23, 11:28 AM

Fri, Jul 20

Vgutierrez committed rOSCCc4facbb85de4: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Fri, Jul 20, 11:22 AM

Thu, Jul 19

Vgutierrez added a comment to T199717: Pick up a suitable ACME library for certcentral.

I've been playing a little bit with the ACME library from certbot and it looks promising. on https://gerrit.wikimedia.org/r/#/c/operations/software/certcentral/+/446618/2/acme_requests.py in the main you can see and example of how it can be used, that code has been used to successfully issue a wildcard certificate using the LE staging API using dns-01 challenges.

Thu, Jul 19, 3:58 PM · Traffic, Operations
Vgutierrez committed rOSCCfc433349f31c: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Thu, Jul 19, 1:55 PM

Wed, Jul 18

Vgutierrez committed rOSCCd2119714314d: WIP: provide ACMEv2 support based on certbot/acme library (authored by Vgutierrez).
WIP: provide ACMEv2 support based on certbot/acme library
Wed, Jul 18, 4:33 PM

Mon, Jul 16

Vgutierrez added a comment to T199717: Pick up a suitable ACME library for certcentral.

From https://letsencrypt.org/docs/client-options/, another interesting option could be free_tls_certificates library. It's a high-level library based on python3-acme, on an initial review apparently it lacks dns-01 support :(

Mon, Jul 16, 4:05 PM · Traffic, Operations
Vgutierrez added a comment to T199717: Pick up a suitable ACME library for certcentral.

I'm strongly considering python-acme, it's the ACME client implementation used by certbot. It's already available as a standalone debian package https://packages.debian.org/search?keywords=python3-acme and the version available in stretch-backports already supports ACMEv2.

Mon, Jul 16, 3:47 PM · Traffic, Operations
Vgutierrez triaged T199717: Pick up a suitable ACME library for certcentral as Normal priority.
Mon, Jul 16, 3:43 PM · Traffic, Operations
Vgutierrez added a subtask for T199711: Deploy a scalable service for ACME (LetsEncrypt) certificate management: T194962: Create and deploy a centralized letsencrypt service.
Mon, Jul 16, 3:34 PM · Traffic, Operations, Goal
Vgutierrez added a parent task for T194962: Create and deploy a centralized letsencrypt service: T199711: Deploy a scalable service for ACME (LetsEncrypt) certificate management.
Mon, Jul 16, 3:34 PM · Patch-For-Review, Wikimedia-Hackathon-2018, Operations, Traffic
Vgutierrez created T199711: Deploy a scalable service for ACME (LetsEncrypt) certificate management.
Mon, Jul 16, 3:33 PM · Traffic, Operations, Goal
Vgutierrez committed rOSCCb7566320c066: get rid of /etc/certcentral being hardcoded everywhere (authored by Vgutierrez).
get rid of /etc/certcentral being hardcoded everywhere
Mon, Jul 16, 2:43 PM
Vgutierrez committed rOSCC5db1167a8e9a: get rid of /etc/certcentral being hardcoded everywhere (authored by Vgutierrez).
get rid of /etc/certcentral being hardcoded everywhere
Mon, Jul 16, 2:36 PM
Vgutierrez committed rOSCC6f49bd1e0592: get rid of /etc/certcentral being hardcoded everywhere (authored by Vgutierrez).
get rid of /etc/certcentral being hardcoded everywhere
Mon, Jul 16, 2:25 PM
Vgutierrez triaged T199677: cp3033 unreacheable since 2018-07-15 11:47:31 as Normal priority.
Mon, Jul 16, 1:56 PM · ops-esams, Operations, Traffic
Vgutierrez added a comment to T199677: cp3033 unreacheable since 2018-07-15 11:47:31.

After a power cycle the server it's behaving properly. Since it was already depooled I'm not repooling it

Mon, Jul 16, 11:52 AM · ops-esams, Operations, Traffic
Vgutierrez moved T199677: cp3033 unreacheable since 2018-07-15 11:47:31 from Triage to Hardware on the Traffic board.
Mon, Jul 16, 11:51 AM · ops-esams, Operations, Traffic
Vgutierrez added a comment to T199675: cp5001 unreachable since 2018-07-14 17:49:21.

both kernel and server event log shows issues on DIMM B4:

3 | 07/14/2018 | 17:49:17 | Memory ECC Uncorr Err | Uncorrectable ECC (UnCorrectable ECC |  DIMMB4) | Asserted
Mon, Jul 16, 11:49 AM · Operations, ops-eqsin, Traffic
Vgutierrez added a comment to T199677: cp3033 unreacheable since 2018-07-15 11:47:31.
[10415964.660782] ------------[ cut here ]------------
[10415964.660790] WARNING: CPU: 13 PID: 34222 at /srv/kernel/linux/net/sched/sch_generic.c:316 dev_watchdog+0x226/0x230
[10415964.660793] NETDEV WATCHDOG: eth0 (bnx2x): transmit queue 6 timed out
[10415964.660793] Modules linked in: cdc_ether usbnet mii joydev hid_generic usbhid hid cpuid binfmt_misc esp6 xfrm6_mode_transport drbg ansi_cprng seqiv xfrm4_mode_transport cpufreq_conservative cpufreq_powersave cpufreq_userspace xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo 8021q garp mrp stp llc tcp_bbr sch_fq intel_rapl sb_edac ipmi_watchdog edac_core x86_pkg_temp_thermal intel_powerclamp coretemp mgag200 ttm drm_kms_helper kvm dcdbas irqbypass crct10dif_pclmul iTCO_wdt crc32_pclmul iTCO_vendor_support evdev drm ghash_clmulni_intel pcspkr i2c_algo_bit mei_me lpc_ich mei shpchp mfd_core wmi button ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 jbd2 fscrypto mbcache raid1 md_mod sg sd_mod ahci libahci aesni_intel aes_x86_64 glue_helper lrw ehci_pci
[10415964.660847]  gf128mul bnx2x ablk_helper ptp ehci_hcd cryptd libata pps_core mdio libcrc32c usbcore crc32c_generic scsi_mod usb_common crc32c_intel
[10415964.660860] CPU: 13 PID: 34222 Comm: cache-worker Not tainted 4.9.0-0.bpo.6-amd64 #1 Debian 4.9.82-1~wmf1
[10415964.660861] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 1.0.4 08/28/2014
[10415964.660863]  0000000000000000 ffffffffa67305e5 ffff8fe9bf183e38 0000000000000000
[10415964.660865]  ffffffffa6479184 0000000000000006 ffff8fe9bf183e90 ffff8fc9b136c000
[10415964.660868]  000000000000000d ffff8fc9b1377100 000000000000005b ffffffffa64791ff
[10415964.660871] Call Trace:
[10415964.660872]  <IRQ>
[10415964.660878]  [<ffffffffa67305e5>] ? dump_stack+0x5c/0x77
[10415964.660882]  [<ffffffffa6479184>] ? __warn+0xc4/0xe0
[10415964.660884]  [<ffffffffa64791ff>] ? warn_slowpath_fmt+0x5f/0x80
[10415964.660888]  [<ffffffffa696e476>] ? tcp_retransmit_timer+0x286/0x890
[10415964.660891]  [<ffffffffa69369a6>] ? dev_watchdog+0x226/0x230
[10415964.660893]  [<ffffffffa6936780>] ? dev_deactivate_queue.constprop.27+0x60/0x60
[10415964.660898]  [<ffffffffa64e85b2>] ? call_timer_fn+0x32/0x130
[10415964.660899]  [<ffffffffa64e9385>] ? run_timer_softirq+0x1e5/0x440
[10415964.660902]  [<ffffffffa67398a4>] ? timerqueue_add+0x54/0xa0
[10415964.660904]  [<ffffffffa64ea808>] ? enqueue_hrtimer+0x38/0x90
[10415964.660909]  [<ffffffffa6a1617c>] ? __do_softirq+0x10c/0x2a2
[10415964.660911]  [<ffffffffa647f4b8>] ? irq_exit+0x98/0xa0
[10415964.660913]  [<ffffffffa6a15c14>] ? smp_apic_timer_interrupt+0x44/0x50
[10415964.660915]  [<ffffffffa6a14496>] ? apic_timer_interrupt+0x96/0xa0
[10415964.660916]  <EOI>
[10415964.660920]  [<ffffffffa64c5bb3>] ? native_queued_spin_lock_slowpath+0x113/0x190
[10415964.660922]  [<ffffffffa6a1245d>] ? _raw_spin_lock+0x1d/0x20
[10415964.660924]  [<ffffffffa64fb018>] ? futex_wake+0xc8/0x170
[10415964.660926]  [<ffffffffa64fd149>] ? do_futex+0x2d9/0xb40
[10415964.660930]  [<ffffffffa64257d9>] ? __switch_to+0x2c9/0x730
[10415964.660932]  [<ffffffffa64fda33>] ? SyS_futex+0x83/0x180
[10415964.660936]  [<ffffffffa6a0dd52>] ? schedule+0x32/0x80
[10415964.660939]  [<ffffffffa6403bd3>] ? do_syscall_64+0x93/0x1a0
[10415964.660941]  [<ffffffffa6a126b8>] ? entry_SYSCALL_64_after_swapgs+0x42/0xb0
[10415964.660942] ---[ end trace 17a2f2dfd85d5ced ]---
Mon, Jul 16, 11:32 AM · ops-esams, Operations, Traffic
Vgutierrez added a comment to T199677: cp3033 unreacheable since 2018-07-15 11:47:31.
root@cp3033:/var/log# ethtool -i eth0
driver: bnx2x
version: 1.712.30-0
firmware-version: FFV7.10.17 bc 7.10.11
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@cp3033:/var/log# ethtool eth0
Settings for eth0:
	Supported ports: [ FIBRE ]
	Supported link modes:   1000baseT/Full
	                        10000baseT/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: No
	Advertised link modes:  10000baseT/Full
	Advertised pause frame use: No
	Advertised auto-negotiation: No
	Speed: Unknown!
	Duplex: Unknown! (255)
	Port: FIBRE
	PHYAD: 1
	Transceiver: internal
	Auto-negotiation: off
	Supports Wake-on: g
	Wake-on: d
	Current message level: 0x00000000 (0)
Mon, Jul 16, 11:29 AM · ops-esams, Operations, Traffic
Vgutierrez created T199677: cp3033 unreacheable since 2018-07-15 11:47:31.
Mon, Jul 16, 11:26 AM · ops-esams, Operations, Traffic
Vgutierrez moved T199675: cp5001 unreachable since 2018-07-14 17:49:21 from Triage to Hardware on the Traffic board.
Mon, Jul 16, 11:14 AM · Operations, ops-eqsin, Traffic
Vgutierrez triaged T199675: cp5001 unreachable since 2018-07-14 17:49:21 as Normal priority.
Mon, Jul 16, 11:14 AM · Operations, ops-eqsin, Traffic
Vgutierrez added a comment to T199675: cp5001 unreachable since 2018-07-14 17:49:21.

cp5001 began to complain about memory errors at Jul 14 17:39:19:

vgutierrez@cp5001:~$ fgrep "section_type: memory error" /var/log/syslog
Jul 14 17:39:19 cp5001 kernel: [4071097.227959] {1}[Hardware Error]:   section_type: memory error
Jul 14 17:40:05 cp5001 kernel: [4071143.323703] {2}[Hardware Error]:   section_type: memory error
Jul 14 17:40:32 cp5001 kernel: [4071170.926275] {3}[Hardware Error]:   section_type: memory error
Jul 14 17:42:04 cp5001 kernel: [4071262.120688] {4}[Hardware Error]:   section_type: memory error
Jul 14 17:43:07 cp5001 kernel: [4071325.427453] {5}[Hardware Error]:   section_type: memory error
Jul 14 17:47:11 cp5001 kernel: [4071569.871201] {6}[Hardware Error]:   section_type: memory error
Mon, Jul 16, 11:13 AM · Operations, ops-eqsin, Traffic
Vgutierrez created T199675: cp5001 unreachable since 2018-07-14 17:49:21.
Mon, Jul 16, 11:02 AM · Operations, ops-eqsin, Traffic