Maniphest T197763

Packet loss en route to gerrit.wikimedia.org
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	daniel
	Jun 20 2018, 9:11 AM

Description

I'm having trouble pulling data from gerrit, probably due to packet loss en route to ahburn.

After waiting for several minutes, I'm getting:

git review -d 441065

Downloading refs/changes/65/441065/1 from gerrit
Cannot fetch patchset contents

Does specified change number belong to this project?

The following command failed with exit code 128
  "git fetch gerrit refs/changes/65/441065/1"
-----------------------
fatal: internal server error
fatal: remote error: internal server error
-----------------------

The error message seems to be misleading, my guess it's a simple TCP timeout. Mtr shows up to 50% packet loss at ash-b1-v6.telia.net

> mtr -r -c 10 gerrit.wikimedia.org
Start: Wed Jun 20 11:06:30 2018
HOST: duesenbox                   Loss%   Snt   Last   Avg  Best  Wrst StDev
 1.|-- p200300E9EF0448B1763170FF  0.0%    10    0.3   0.4   0.3   0.6   0.0
 2.|-- p200300E9EFFF044800000000  0.0%    10   15.5  20.1  15.5  58.5  13.5
 3.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 4.|-- 2003:0:1403:c001::2        0.0%    10   20.7  20.7  20.5  21.0   0.0
 5.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 6.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 7.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 8.|-- ash-b1-v6.telia.net       50.0%    10  164.6 164.4 164.0 164.6   0.0
 9.|-- wikimedia-ic-308845-ash-b 40.0%    10  164.2 165.2 164.2 169.8   2.2
10.|-- gerrit.wikimedia.org      30.0%    10  164.0 163.9 163.6 164.3   0.0

I'm located in Berlin, ISP is Deutsche Telekom. Coming in via IPv6, public IP is 2003:e9:ef04:48b1:718b:37f4:2e05:10ea resp. 80.132.229.59.

Event Timeline

daniel created this task.Jun 20 2018, 9:11 AM

Restricted Application added a project: SRE. · View Herald TranscriptJun 20 2018, 9:11 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

daniel updated the task description. (Show Details)Jun 20 2018, 9:17 AM

Paladox subscribed.Jun 20 2018, 9:19 AM

Update: it's fine now. Feel free to close or keep open, as makes more sense for you.

v4 from Prague goes well:

$ mtr text-lb.eqiad.wikimedia.org --report-wide
Start: Wed Jun 20 11:15:08 2018
HOST: laptop                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- gateway                                   0.0%    10    2.0   1.6   1.4   2.2   0.0
  2.|-- 89.235.31.89                              0.0%    10    2.4   2.4   2.2   2.6   0.0
  3.|-- 82.119.246.253                            0.0%    10    2.4   2.8   2.4   3.7   0.0
  4.|-- cz-prg-asbr1-hunge0-2-0-0.dialtelecom.cz  0.0%    10    2.4   2.6   2.2   3.2   0.0
  5.|-- prag-b3-link.telia.net                    0.0%    10    4.5   2.8   1.9   4.5   0.8
  6.|-- win-bb2-link.telia.net                    0.0%    10  108.1 109.6 107.8 119.2   3.4
  7.|-- ffm-bb4-link.telia.net                    0.0%    10  108.3 108.0 106.9 111.2   1.2
  8.|-- prs-bb4-link.telia.net                    0.0%    10  108.7 113.4 107.0 147.3  12.3
  9.|-- ash-bb4-link.telia.net                    0.0%    10  113.3 111.6 106.9 126.5   6.5
 10.|-- ash-b1-link.telia.net                     0.0%    10  111.9 109.5 106.9 122.6   4.8
 11.|-- wikimedia-ic-308845-ash-b1.c.telia.net    0.0%    10  170.2 113.6 106.7 170.2  19.9
 12.|-- text-lb.eqiad.wikimedia.org               0.0%    10  106.3 106.6 106.3 107.1   0.0

Telia finished fixing a fibercut in NYC right now:

This is to inform you that we managed to bring all the services up.
Please note that this is a temporary solution and a permanent one will be done later.

Most likely related (network convergence, etc..).

We will monitor it more closely, thanks for reporting the issue.

ayounsi closed this task as Resolved.Jun 20 2018, 9:22 AM

ayounsi claimed this task.

• Vvjjkkii renamed this task from Packet loss en route to gerrit.wikimedia.org to wlaaaaaaaa.Jul 1 2018, 1:02 AM

• Vvjjkkii reopened this task as Open.

• Vvjjkkii removed ayounsi as the assignee of this task.

• Vvjjkkii triaged this task as High priority.

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed a subscriber: Aklapper.

• ema renamed this task from wlaaaaaaaa to Packet loss en route to gerrit.wikimedia.org.Jul 2 2018, 9:31 AM

• ema closed this task as Resolved.

• ema assigned this task to ayounsi.

• ema updated the task description. (Show Details)

Vgutierrez removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.Jul 2 2018, 9:40 AM

CommunityTechBot raised the priority of this task from High to Needs Triage.Jul 5 2018, 7:03 PM

Packet loss en route to gerrit.wikimedia.orgClosed, ResolvedPublicActions

Description

Event Timeline

Packet loss en route to gerrit.wikimedia.org
Closed, ResolvedPublic
Actions