Page MenuHomePhabricator

git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist
Closed, DeclinedPublic

Description

I'm unable to fetch or pull MW core, check out patches for review, etc.

Checking around the office it seems like some are getting the same error (using https as well as ssh), while others do not. May be related to when the last pulled, and which commits need to be transferred. A fresh clone seems to not have the issue.

1# Trying to fetch MW core:
2> git remote -vv
3gerrit ssh://daniel@gerrit.wikimedia.org:29418/mediawiki/core.git (fetch)
4gerrit ssh://daniel@gerrit.wikimedia.org:29418/mediawiki/core.git (push)
5origin ssh://daniel@gerrit.wikimedia.org:29418/mediawiki/core.git (fetch)
6origin ssh://daniel@gerrit.wikimedia.org:29418/mediawiki/core.git (push)
7
8> git fetch --all -vv
9Fetching origin
10Server supports multi_ack_detailed
11Server supports side-band-64k
12Server supports ofs-delta
13Server version is JGit/4.7.1.201706071930-r
14Marking 79b4585ab41741722d4c841dc058588f3cb1dd87 as complete
15want 9e83e7f7a042feeee9986a1fb1be531427f418ac (refs/heads/fundraising/REL1_27)
16want 14cb8f992f7f9d1e1b405c200720c53f4e62b8d2 (refs/heads/master)
17want 18911f8b7d07cbaa1c66f65b3a0d36e20ad1aa07 (refs/heads/wmf/1.31.0-wmf.25)
18want 68b856377cddc7563386fbbe49ec64f68009da01 (refs/heads/wmf/1.31.0-wmf.26)
19have 79b4585ab41741722d4c841dc058588f3cb1dd87
20have 9ced9ebca28e1ce2bed455da5ad9da208013ef18
21have 3af04875626877013fe6181f727902088df85618
22have b6c4f60b0a10d7efecb5fb0d5555e0b891e24419
23have cf1021b879f1ba84ecaacf3864c8e693ee6bf9f0
24have f2f370cb31b0d2b9d88afa10161b39a1b87af093
25have 08d90fb722a0abe7bc4dfadfdf2e7841fc86b0ed
26have 3b076177b4d862412d47cc7625d38c7c7c69acfe
27have fe92778888a381311eb4606d7940b5a089684b14
28have c6ac3d182cf2d6a0090c3dc3db87be4a89e1e708
29have 6558e96a3e2e6775a1c733e33a466277c7f66914
30have b0bdfd7e9f7fab3fb2eae37da5d953d03362c15f
31have 5caebf049326344eea2878e55c9e0f38e8fcab98
32have fb56b37d67ba9b2c1b16da68beca4bdb218993f3
33have 697a5d923a16c579a8e7f3f8a8d36a82f8f95a17
34have 5acb6a339d89a9ed65b38d5ac0a0efe7072e58be
35have fd0405a6b35d767c453cb727cc26484fb15cd09f
36have 01e335d842530fc9df0922e0b6cb30b6747a2387
37have bf8947cf7d8933976a778fb803e0f482457a3eae
38have b7737ebed1e8940df4dfd4c50c1224d0edc8639d
39have 3d2395a2af962b9bb941f3bbd819578f2b6b2d65
40have 3577973107e642493a58d2a6580a99d3dff0137b
41have 4fe27aa2e3404072ed2de203268b019fd9851a1d
42have adbbf7917bbd1bcb8f81af20c867b4df536cb3ee
43have 30f5f3edfbd1276c844f5f711290487e980ba91c
44have 7a17d143f21975993b8f82adeb4c9b22a3b319f7
45have d3d60fe4d09318d10fc898477f17ba8399f9d9ec
46have 13aabef5734f892f34632a29aa762c14dc3ae5c1
47have 9c1d8a970921c4741b6c2351d5d4ecefd0be09a7
48have 89576b42a6764a2d3f03bc5e539dd38524d1ca28
49have 0821bea5a63f4e500fdee4844c79daa31f3de00d
50have 2c018e1c9c766427f8d349b3b3f08258d4a9241b
51got ack 3 79b4585ab41741722d4c841dc058588f3cb1dd87
52got ack 3 9ced9ebca28e1ce2bed455da5ad9da208013ef18
53got ack 3 b6c4f60b0a10d7efecb5fb0d5555e0b891e24419
54got ack 3 cf1021b879f1ba84ecaacf3864c8e693ee6bf9f0
55got ack 3 08d90fb722a0abe7bc4dfadfdf2e7841fc86b0ed
56got ack 3 6558e96a3e2e6775a1c733e33a466277c7f66914
57got ack 3 fb56b37d67ba9b2c1b16da68beca4bdb218993f3
58have 744d9c415a94483fb185b732918f4650e6e9d6cf
59have f58156132556860f006832fbc908b092c18af7d7
60have 4c97eee6aab471b7e52a02a26128e90e793d1165
61have 3e13d5a7efcbdf038aba639e336e738f2608efb3
62have ffda98c828dd7e7082fcc447539eb9530178caf1
63have c081e3253a27deda5162c857d777321c96b5d71c
64have 3f1b563f88167716bec327a2a73bea494fa5d8f6
65have 9528e7f74c6a268246d894d03aa6efc0bee1e432
66have fb1917af5f7595893f206585ddbab408dc46485b
67have 9668d4e0f8181806d21cc2acfe4b2b32207c5b64
68have 71dca826d4d22916e872f2f34abd1f2b68c0b919
69have 7f3283a8f4f74ea4d3dc0fb939211b19d498201e
70have 9a8cd14f4775d0a7efb86f1c7eef9c4c1f42bc5e
71have d5c267c776854c0932030cb23ff7093d0a249905
72have 18bf5e4aae6325ca4ff627fcc6cfc1c3365dcbee
73have 6158fc21d90845e26f5ff3cd4d28cfd87f1ee4ea
74have 976e24c9cdbf9de9d34935162bd73fd86235ea77
75have 4d0b340d2f4d45d13b9a10e21b51eeebd3e0372f
76have 76bab5e0ee5df7acfc1af391ea3bdf6d2184ad39
77have c56936ffc19c6d911871dc7fb96db14617a0b9d8
78have fd29ac30b061c6e0c6590eed98f700a4e829a424
79have ffd189fb80ea712c730f2b9acec4de1faf5a8b93
80have c71fd84bee173b3d17476a16017d854b34ad324a
81have a2c207ea9efc45aa7b111ec6a11d1bee1a715fd6
82have f83b96f6e93893c9fc5ce4781597e6d1249a795c
83have c06a63b392c6a7fd29621fc3305a7a1a23b17bd7
84have 94348a56085c74bd2f5c07589f88dfeb50546677
85have e878ab185b0185fe49ac2bdc03fb6298c449ebaf
86have 91d11bc65f5a71ab74628fb52e389b1615727b9b
87have ccc42b87f5d9b6d63c15ff4a01e41024f01133d9
88have 0d534c8061e4813b2b1a053b7cf008755857cc40
89have 983ebd8605c9fa778a5bb5aa5c9e227f1223674b
90got ack 3 3577973107e642493a58d2a6580a99d3dff0137b
91got ack 3 30f5f3edfbd1276c844f5f711290487e980ba91c
92got ack 3 01e335d842530fc9df0922e0b6cb30b6747a2387
93got ack 3 bf8947cf7d8933976a778fb803e0f482457a3eae
94got ack 3 4fe27aa2e3404072ed2de203268b019fd9851a1d
95got ack 3 adbbf7917bbd1bcb8f81af20c867b4df536cb3ee
96got ack 3 7a17d143f21975993b8f82adeb4c9b22a3b319f7
97got ack 3 d3d60fe4d09318d10fc898477f17ba8399f9d9ec
98got ack 3 13aabef5734f892f34632a29aa762c14dc3ae5c1
99got ack 3 9c1d8a970921c4741b6c2351d5d4ecefd0be09a7
100got ack 3 89576b42a6764a2d3f03bc5e539dd38524d1ca28
101got ack 3 2c018e1c9c766427f8d349b3b3f08258d4a9241b
102got ack 4 fd0405a6b35d767c453cb727cc26484fb15cd09f
103done
104got ack (3) 744d9c415a94483fb185b732918f4650e6e9d6cf
105got ack (3) f58156132556860f006832fbc908b092c18af7d7
106got ack (3) 4c97eee6aab471b7e52a02a26128e90e793d1165
107got ack (3) 3e13d5a7efcbdf038aba639e336e738f2608efb3
108got ack (3) 9528e7f74c6a268246d894d03aa6efc0bee1e432
109got ack (3) 9668d4e0f8181806d21cc2acfe4b2b32207c5b64
110got ack (3) 7f3283a8f4f74ea4d3dc0fb939211b19d498201e
111got ack (3) 9a8cd14f4775d0a7efb86f1c7eef9c4c1f42bc5e
112got ack (3) 18bf5e4aae6325ca4ff627fcc6cfc1c3365dcbee
113got ack (3) 6158fc21d90845e26f5ff3cd4d28cfd87f1ee4ea
114got ack (3) 976e24c9cdbf9de9d34935162bd73fd86235ea77
115got ack (3) 76bab5e0ee5df7acfc1af391ea3bdf6d2184ad39
116got ack (3) fd29ac30b061c6e0c6590eed98f700a4e829a424
117got ack (3) c71fd84bee173b3d17476a16017d854b34ad324a
118got ack (3) c06a63b392c6a7fd29621fc3305a7a1a23b17bd7
119got ack (3) e878ab185b0185fe49ac2bdc03fb6298c449ebaf
120got ack (3) 0d534c8061e4813b2b1a053b7cf008755857cc40
121got ack (4) 983ebd8605c9fa778a5bb5aa5c9e227f1223674b
122fatal: internal server error
123got ack (1) 0d534c8061e4813b2b1a053b7cf008755857cc40
124remote: internal server error
125fatal: protocol error: bad pack header
126error: Could not fetch origin
127Fetching gerrit
128Server supports multi_ack_detailed
129Server supports side-band-64k
130Server supports ofs-delta
131Server version is JGit/4.7.1.201706071930-r
132Marking 79b4585ab41741722d4c841dc058588f3cb1dd87 as complete
133want 9e83e7f7a042feeee9986a1fb1be531427f418ac (refs/heads/fundraising/REL1_27)
134want 14cb8f992f7f9d1e1b405c200720c53f4e62b8d2 (refs/heads/master)
135want 18911f8b7d07cbaa1c66f65b3a0d36e20ad1aa07 (refs/heads/wmf/1.31.0-wmf.25)
136want 68b856377cddc7563386fbbe49ec64f68009da01 (refs/heads/wmf/1.31.0-wmf.26)
137have 79b4585ab41741722d4c841dc058588f3cb1dd87
138have 9ced9ebca28e1ce2bed455da5ad9da208013ef18
139have 3af04875626877013fe6181f727902088df85618
140have b6c4f60b0a10d7efecb5fb0d5555e0b891e24419
141have cf1021b879f1ba84ecaacf3864c8e693ee6bf9f0
142have f2f370cb31b0d2b9d88afa10161b39a1b87af093
143have 08d90fb722a0abe7bc4dfadfdf2e7841fc86b0ed
144have 3b076177b4d862412d47cc7625d38c7c7c69acfe
145have fe92778888a381311eb4606d7940b5a089684b14
146have c6ac3d182cf2d6a0090c3dc3db87be4a89e1e708
147have 6558e96a3e2e6775a1c733e33a466277c7f66914
148have b0bdfd7e9f7fab3fb2eae37da5d953d03362c15f
149have 5caebf049326344eea2878e55c9e0f38e8fcab98
150have fb56b37d67ba9b2c1b16da68beca4bdb218993f3
151have 697a5d923a16c579a8e7f3f8a8d36a82f8f95a17
152have 5acb6a339d89a9ed65b38d5ac0a0efe7072e58be
153have fd0405a6b35d767c453cb727cc26484fb15cd09f
154have 01e335d842530fc9df0922e0b6cb30b6747a2387
155have bf8947cf7d8933976a778fb803e0f482457a3eae
156have b7737ebed1e8940df4dfd4c50c1224d0edc8639d
157have 3d2395a2af962b9bb941f3bbd819578f2b6b2d65
158have 3577973107e642493a58d2a6580a99d3dff0137b
159have 4fe27aa2e3404072ed2de203268b019fd9851a1d
160have adbbf7917bbd1bcb8f81af20c867b4df536cb3ee
161have 30f5f3edfbd1276c844f5f711290487e980ba91c
162have 7a17d143f21975993b8f82adeb4c9b22a3b319f7
163have d3d60fe4d09318d10fc898477f17ba8399f9d9ec
164have 13aabef5734f892f34632a29aa762c14dc3ae5c1
165have 9c1d8a970921c4741b6c2351d5d4ecefd0be09a7
166have 89576b42a6764a2d3f03bc5e539dd38524d1ca28
167have 0821bea5a63f4e500fdee4844c79daa31f3de00d
168have 2c018e1c9c766427f8d349b3b3f08258d4a9241b
169got ack 3 79b4585ab41741722d4c841dc058588f3cb1dd87
170got ack 3 9ced9ebca28e1ce2bed455da5ad9da208013ef18
171got ack 3 b6c4f60b0a10d7efecb5fb0d5555e0b891e24419
172got ack 3 cf1021b879f1ba84ecaacf3864c8e693ee6bf9f0
173got ack 3 08d90fb722a0abe7bc4dfadfdf2e7841fc86b0ed
174got ack 3 6558e96a3e2e6775a1c733e33a466277c7f66914
175got ack 3 fb56b37d67ba9b2c1b16da68beca4bdb218993f3
176have 744d9c415a94483fb185b732918f4650e6e9d6cf
177have f58156132556860f006832fbc908b092c18af7d7
178have 4c97eee6aab471b7e52a02a26128e90e793d1165
179have 3e13d5a7efcbdf038aba639e336e738f2608efb3
180have ffda98c828dd7e7082fcc447539eb9530178caf1
181have c081e3253a27deda5162c857d777321c96b5d71c
182have 3f1b563f88167716bec327a2a73bea494fa5d8f6
183have 9528e7f74c6a268246d894d03aa6efc0bee1e432
184have fb1917af5f7595893f206585ddbab408dc46485b
185have 9668d4e0f8181806d21cc2acfe4b2b32207c5b64
186have 71dca826d4d22916e872f2f34abd1f2b68c0b919
187have 7f3283a8f4f74ea4d3dc0fb939211b19d498201e
188have 9a8cd14f4775d0a7efb86f1c7eef9c4c1f42bc5e
189have d5c267c776854c0932030cb23ff7093d0a249905
190have 18bf5e4aae6325ca4ff627fcc6cfc1c3365dcbee
191have 6158fc21d90845e26f5ff3cd4d28cfd87f1ee4ea
192have 976e24c9cdbf9de9d34935162bd73fd86235ea77
193have 4d0b340d2f4d45d13b9a10e21b51eeebd3e0372f
194have 76bab5e0ee5df7acfc1af391ea3bdf6d2184ad39
195have c56936ffc19c6d911871dc7fb96db14617a0b9d8
196have fd29ac30b061c6e0c6590eed98f700a4e829a424
197have ffd189fb80ea712c730f2b9acec4de1faf5a8b93
198have c71fd84bee173b3d17476a16017d854b34ad324a
199have a2c207ea9efc45aa7b111ec6a11d1bee1a715fd6
200have f83b96f6e93893c9fc5ce4781597e6d1249a795c
201have c06a63b392c6a7fd29621fc3305a7a1a23b17bd7
202have 94348a56085c74bd2f5c07589f88dfeb50546677
203have e878ab185b0185fe49ac2bdc03fb6298c449ebaf
204have 91d11bc65f5a71ab74628fb52e389b1615727b9b
205have ccc42b87f5d9b6d63c15ff4a01e41024f01133d9
206have 0d534c8061e4813b2b1a053b7cf008755857cc40
207have 983ebd8605c9fa778a5bb5aa5c9e227f1223674b
208got ack 3 3577973107e642493a58d2a6580a99d3dff0137b
209got ack 3 30f5f3edfbd1276c844f5f711290487e980ba91c
210got ack 3 01e335d842530fc9df0922e0b6cb30b6747a2387
211got ack 3 bf8947cf7d8933976a778fb803e0f482457a3eae
212got ack 3 4fe27aa2e3404072ed2de203268b019fd9851a1d
213got ack 3 adbbf7917bbd1bcb8f81af20c867b4df536cb3ee
214got ack 3 7a17d143f21975993b8f82adeb4c9b22a3b319f7
215got ack 3 d3d60fe4d09318d10fc898477f17ba8399f9d9ec
216got ack 3 13aabef5734f892f34632a29aa762c14dc3ae5c1
217got ack 3 9c1d8a970921c4741b6c2351d5d4ecefd0be09a7
218got ack 3 89576b42a6764a2d3f03bc5e539dd38524d1ca28
219got ack 3 2c018e1c9c766427f8d349b3b3f08258d4a9241b
220got ack 4 fd0405a6b35d767c453cb727cc26484fb15cd09f
221done
222got ack (3) 744d9c415a94483fb185b732918f4650e6e9d6cf
223got ack (3) f58156132556860f006832fbc908b092c18af7d7
224got ack (3) 4c97eee6aab471b7e52a02a26128e90e793d1165
225got ack (3) 3e13d5a7efcbdf038aba639e336e738f2608efb3
226got ack (3) 9528e7f74c6a268246d894d03aa6efc0bee1e432
227got ack (3) 9668d4e0f8181806d21cc2acfe4b2b32207c5b64
228got ack (3) 7f3283a8f4f74ea4d3dc0fb939211b19d498201e
229got ack (3) 9a8cd14f4775d0a7efb86f1c7eef9c4c1f42bc5e
230got ack (3) 18bf5e4aae6325ca4ff627fcc6cfc1c3365dcbee
231got ack (3) 6158fc21d90845e26f5ff3cd4d28cfd87f1ee4ea
232got ack (3) 976e24c9cdbf9de9d34935162bd73fd86235ea77
233got ack (3) 76bab5e0ee5df7acfc1af391ea3bdf6d2184ad39
234got ack (3) fd29ac30b061c6e0c6590eed98f700a4e829a424
235got ack (3) c71fd84bee173b3d17476a16017d854b34ad324a
236got ack (3) c06a63b392c6a7fd29621fc3305a7a1a23b17bd7
237got ack (3) e878ab185b0185fe49ac2bdc03fb6298c449ebaf
238got ack (3) 0d534c8061e4813b2b1a053b7cf008755857cc40
239got ack (4) 983ebd8605c9fa778a5bb5aa5c9e227f1223674b
240fatal: internal server error
241got ack (1) 0d534c8061e4813b2b1a053b7cf008755857cc40
242remote: internal server error
243fatal: protocol error: bad pack header
244error: Could not fetch gerrit
245

Event Timeline

daniel created this task.Mar 26 2018, 12:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 26 2018, 12:47 PM
daniel triaged this task as High priority.Mar 26 2018, 12:48 PM

Bumping to high. May even be UBN. I'm trying a fresh clone now.

Ftr:

[14:49] <zhuyifei1999_> worked for me
[14:49] <zhuyifei1999_> https://www.irccloud.com/pastebin/U3V1obs5/

A fresh clone seems to fix the problem. So no UBN.

daniel updated the task description. (Show Details)Mar 26 2018, 12:54 PM
Nikerabbit added a subscriber: Nikerabbit.
Paladox added a subscriber: Paladox.EditedMar 26 2018, 12:58 PM

Hi, please run "git remote prune origin" that will fix it and was suggested by @demon on irc on friday.

Paladox closed this task as Resolved.Mar 26 2018, 12:59 PM

As long as you run "git remote prune origin" that should resolve the problem, please reopen if it dosen't.

The reason is you have local branches pointing to remote branches that no more exist. Specially, the wmf/ branches are deleted after some weeks and whenever @demon runs git gc on the Gerrit server, the objects are gone. Then your git client sends the branch it still knows about but since it is no more on Gerrit the server aborts.

A workaround on the client is to clean up remote branches:

git remote update --prune

And then delete wmf branches (with git branch -d --force).

Ah, thank you!
Is this somehow a new thing? I wonder why I'm running into this for the first time now.

I feel like filing an upstream bug. "fatal: protocol error: bad pack header" is NOT a good way to say "the branch you are tracking no longer exists on the remote repo"...

Ah, thank you!
Is this somehow a new thing? I wonder why I'm running into this for the first time now.

Nope nothing new, it's because some branches were deleted (old wmf branches that were not needed any more), so when gc ran, it deleted unreferenced objects leading to this problem, as long as you run "git remote prune origin" it should fix it's self.

You could also do "git config remote.origin.prune true" (per @demon suggestion too)

Aklapper renamed this task from git pull fails for MW core filas with "fatal: protocol error: bad pack header" to git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist.Mar 26 2018, 3:27 PM
demon added a comment.Mar 26 2018, 3:41 PM

This also isn't a Gerrit bug. Can easily replicate with Github or any other git server. Tbh, I'm not sure why this hasn't been an issue before...the workflow here hasn't changed: we prune these branches weekly and git gc I do from time to time.

In discussing this on Friday, the thinking was that the fix for T103990 may also fix this?

demon added a comment.Mar 26 2018, 3:43 PM

You could also do "git config remote.origin.prune true" (per @demon suggestion too)

Personally, I've taken to doing git config --global fetch.prune true so my ~/.gitconfig does this on all repos & remotes by default.

Ladsgroup reopened this task as Open.EditedMar 27 2018, 10:58 AM
Ladsgroup added a subscriber: Ladsgroup.

We can't tell everyone to run "git remote prune origin" when seeing fatal. This is not the right way to fix a bug.

I propose to decline this task as this is an upstream issue.

This has nothing to do with gerrit now, the problem lies in git not doing this behaviour by default, maybe create a feature / bug report with them?

At this moment there is nothing more we can do as there is no bugs, this is intended and as @demo explained you can reproduce this with github.

@Ladsgroup I agree that it's a bug and should be fixed. But if I understand correctly, it's a bug in git.

Ladsgroup closed this task as Declined.Mar 27 2018, 4:25 PM

If you think so I'm fine but please document this somewhere so people like me don't end up asking the same question in IRC

Kghbln awarded a token.Apr 1 2018, 8:59 PM
Kghbln added a subscriber: Kghbln.