Change Details

We've getting the occasional alert from Smokeping about install2001, with the graph showing various amounts of jitter and packet loss (up to 5% at times). I've since added achernar and bast2001 to Smokeping, with bast2001 being on the same rack as install2001 (A5) — and in fact, neighboring ports. Neither achernar nor bast2001 does NOT exhibit this behavior — everything looks perfect there. Compare [[ https://smokeping.wikimedia.org/?target=codfw.Hosts.install2001 | install2001's graphs ]] with [[ https://smokeping.wikimedia.org/?target=codfw.Hosts.bast2001 | bast2001's ]]. Therefore, the problem looks localized to install2001. Neither the host nor the switch report any CRC/FCS errors or such and apart from a few RX discards on the install2001 side, everything else looks normal. Still, a bad cable would explain this and it's easy enough to replace it with a new one that we should try it. Please replace its Ethernet cable with a new one and let me know. Thanks!

We've been getting the occasional alert from Smokeping about install2001, with the graph showing various amounts of jitter and packet loss (up to 5% at times). I've since added achernar and bast2001 to Smokeping, with bast2001 being on the same rack as install2001 (A5) — and in fact, neighboring ports. Neither achernar nor bast2001 does NOT exhibit this behavior — everything looks perfect there. Compare [[ https://smokeping.wikimedia.org/?target=codfw.Hosts.install2001 | install2001's graphs ]] with [[ https://smokeping.wikimedia.org/?target=codfw.Hosts.bast2001 | bast2001's ]]. Therefore, the problem looks localized to install2001. Neither the host nor the switch report any CRC/FCS errors or such and apart from a few RX discards on the install2001 side, everything else looks normal. --- Update 2016-06-14: we started getting a *lot * more packet loss, up to 20%. At the same time, CPU graphs showed 100% utilization (system). I logged in and found multiple threads of `acpi_pad` consuming all of the machine's CPU. I `rmmod`'ed modules `acpi_pad` and `mei`, but packet loss seems to be still there, as well as other behavior I'd call odd (general slowness?) This looks like either kernel or hardware troubles at this point, I'd bet the latter. Please investigate — thanks!

We've been getting the occasional alert from Smokeping about install2001, with the graph showing various amounts of jitter and packet loss (up to 5% at times). I've since added achernar and bast2001 to Smokeping, with bast2001 being on the same rack as install2001 (A5) — and in fact, neighboring ports. Neither achernar nor bast2001 does NOT exhibit this behavior — everything looks perfect there. Compare [[ https://smokeping.wikimedia.org/?target=codfw.Hosts.install2001 | install2001's graphs ]] with [[ https://smokeping.wikimedia.org/?target=codfw.Hosts.bast2001 | bast2001's ]]. Therefore, the problem looks localized to install2001. Neither the host nor the switch report any CRC/FCS errors or such and apart from a few RX discards on the install2001 side, everything else looks normal. Still, a bad cable would explain this and it's easy enough to replace it with a new one that we should try it.--- Please replace its Ethernet cable with a new one and let me knowUpdate 2016-06-14: we started getting a *lot * more packet loss, up to 20%. At the same time, CPU graphs showed 100% utilization (system). I logged in and found multiple threads of `acpi_pad` consuming all of the machine's CPU. I `rmmod`'ed modules `acpi_pad` and `mei`, but packet loss seems to be still there, as well as other behavior I'd call odd (general slowness?) This looks like either kernel or hardware troubles at this point, I'd bet the latter. TPlease investigate — thanks!