Page MenuHomePhabricator

gerrit-replica is returning 502 responses when trying to git clone, breaking libup
Closed, ResolvedPublic

Description

Dec 14 11:37:08 upgrader-05 celery[317]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/NoTitle.git/': The requested URL returned error: 502
Dec 14 11:38:08 upgrader-05 celery[317]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/NotebookViewer.git/': The requested URL returned error: 502
Dec 14 11:59:25 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/OAuth.git/': The requested URL returned error: 502
Dec 14 12:39:03 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/OpenLayers.git/': The requested URL returned error: 502
Dec 14 12:42:30 upgrader-05 containerd[582]: time="2019-12-14T12:42:30.750283226Z" level=info msg="shim reaped" id=988f678f9be23436178d4e9c8ef139f2480b1c49e4b25937535d901b45764a68
Dec 14 12:45:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PCRGUIInserts.git/': The requested URL returned error: 502
Dec 14 12:47:06 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageAssessments.git/': The requested URL returned error: 502
Dec 14 12:59:18 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageCreationNotif.git/': The requested URL returned error: 502
Dec 14 13:13:08 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageImages.git/': The requested URL returned error: 502
Dec 14 13:14:08 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageInCat.git/': The requested URL returned error: 502
Dec 14 13:26:08 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 13:31:17 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageNotice.git/': The requested URL returned error: 502
Dec 14 13:32:22 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageSchemas.git/': The requested URL returned error: 502
Dec 14 13:33:23 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageTriage.git/': The requested URL returned error: 502
Dec 14 13:39:29 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PaginateText.git/': The requested URL returned error: 502
Dec 14 13:42:32 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ParentPage.git/': The requested URL returned error: 502
Dec 14 13:43:32 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ParserFun.git/': The requested URL returned error: 502
Dec 14 13:56:46 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PerPageLicense.git/': The requested URL returned error: 502
Dec 14 14:09:51 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PhpTagsFunctions.git/': The requested URL returned error: 502
Dec 14 14:14:45 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PhpTagsSPARQL.git/': The requested URL returned error: 502
Dec 14 14:19:36 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PhpTagsWidgets.git/': The requested URL returned error: 502
Dec 14 14:22:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PhpTagsWiki.git/': The requested URL returned error: 502
Dec 14 14:22:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PictureGame.git/': The requested URL returned error: 502
Dec 14 14:26:30 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PluggableSSO.git/': The requested URL returned error: 502
Dec 14 14:36:23 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PopupAlerts.git/': The requested URL returned error: 502
Dec 14 14:59:26 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PrivateDomains.git/': The requested URL returned error: 502
Dec 14 14:59:26 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Prezi.git/': The requested URL returned error: 502
Dec 14 15:42:01 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 15:46:43 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ProtectionIndicator.git/': The requested URL returned error: 502
Dec 14 15:49:54 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 15:50:41 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PurgeClickThrough.git/': The requested URL returned error: 502
Dec 14 16:01:58 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Push.git/': The requested URL returned error: 502
Dec 14 16:02:45 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PushToWatch.git/': The requested URL returned error: 502
Dec 14 16:14:03 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PwnedPasswords.git/': The requested URL returned error: 502
Dec 14 16:33:26 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 16:37:23 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 16:45:29 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/QuickResponse.git/': The requested URL returned error: 502
Dec 14 16:48:06 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/QuickSurveys.git/': The requested URL returned error: 502
Dec 14 16:53:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Quiz.git/': The requested URL returned error: 502
Dec 14 17:05:05 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 17:05:09 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/QuizGame.git/': The requested URL returned error: 502
Dec 14 17:15:42 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RSS.git/': The requested URL returned error: 502
Dec 14 17:17:09 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/QuizTabulate.git/': The requested URL returned error: 502
Dec 14 17:24:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomArea.git/': The requested URL returned error: 502
Dec 14 17:35:36 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 17:36:38 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomGameUnit.git/': The requested URL returned error: 502
Dec 14 17:47:40 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomImage.git/': The requested URL returned error: 502
Dec 14 17:48:42 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomImageByCategory.git/': The requested URL returned error: 502
Dec 14 17:52:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomPages.git/': The requested URL returned error: 502
Dec 14 17:52:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomInCategory.git/': The requested URL returned error: 502
Dec 14 18:02:01 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ReadingLists.git/': The requested URL returned error: 502
Dec 14 18:04:10 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomSelection.git/': The requested URL returned error: 502
Dec 14 18:14:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ReassignEdits.git/': The requested URL returned error: 502
Dec 14 18:16:14 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RecentActivityFeed.git/': The requested URL returned error: 502
Dec 14 18:26:08 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Recoin.git/': The requested URL returned error: 502
Dec 14 18:28:18 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RefreshSpecial.git/': The requested URL returned error: 502
Dec 14 18:38:12 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RegexBlock.git/': The requested URL returned error: 502
Dec 14 18:40:21 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RegexFun.git/': The requested URL returned error: 502
Dec 14 18:50:16 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RegexFunctions.git/': The requested URL returned error: 502
Dec 14 18:52:25 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RelatedArticles.git/': The requested URL returned error: 502
Dec 14 19:02:19 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RelatedLinks.git/': The requested URL returned error: 502
Dec 14 19:04:29 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RelatedSites.git/': The requested URL returned error: 502
Dec 14 19:14:23 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Renameuser.git/': The requested URL returned error: 502
Dec 14 19:16:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ReplaceSet.git/': The requested URL returned error: 502
Dec 14 19:26:27 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ReplaceText.git/': The requested URL returned error: 502
Dec 14 19:28:37 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RevisionCommentSupplement.git/': The requested URL returned error: 502
Dec 14 19:43:30 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SacredText.git/': The requested URL returned error: 502
Dec 14 19:44:00 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SVGEdit.git/': The requested URL returned error: 502
Dec 14 19:55:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SafeDelete.git/': The requested URL returned error: 502
Dec 14 19:56:04 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SandboxLink.git/': The requested URL returned error: 502
Dec 14 19:58:02 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Sarcasm.git/': The requested URL returned error: 502

Event Timeline

Legoktm created this task.Dec 14 2019, 7:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2019, 7:56 PM
Legoktm renamed this task from gerrit-replica is sometimes returning 502 responses when trying to git clone, affecting libup to gerrit-replica is returning 502 responses when trying to git clone, breaking libup.Dec 14 2019, 8:00 PM
Legoktm updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-cloud) [2019-12-14T20:06:40Z] <legoktm> systemctl stop libup-celery # because of T240763

Paladox triaged this task as Unbreak Now! priority.Dec 14 2019, 8:15 PM
Paladox added a project: Operations.
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptDec 14 2019, 8:15 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-14T22:50:07Z] <hashar> Restarted Gerrit on gerrit2001 # T240763

[2019-12-14 16:07:25,808] [HTTP-87043]
WARN org.eclipse.jetty.servlet.ServletHandler : Error for /r/mediawiki/extensions/DataTransfer.git/info/refs
java.lang.OutOfMemoryError: Java heap space

And that is with a 32GB heap:

-XX:+UseG1GC -Xmx32g -Xms32g -XX:G1NewSizePercent=15

There are two threads at 100% CPU.

jstack: F31475924

jmap -heap:

Garbage-First (G1) GC with 23 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 34359738368 (32768.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 20602421248 (19648.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 16777216 (16.0MB)

Heap Usage:
G1 Heap:
   regions  = 2048
   capacity = 34359738368 (32768.0MB)
   used     = 32389869888 (30889.38702392578MB)
   free     = 1969868480 (1878.6129760742188MB)
   94.26692817360163% used
G1 Young Generation:
Eden Space:
   regions  = 0
   capacity = 1962934272 (1872.0MB)
   used     = 0 (0.0MB)
   free     = 1962934272 (1872.0MB)
   0.0% used
Survivor Space:
   regions  = 0
   capacity = 0 (0.0MB)
   used     = 0 (0.0MB)
   free     = 0 (0.0MB)
   0.0% used
G1 Old Generation:
   regions  = 287
   capacity = 32396804096 (30896.0MB)
   used     = 32389869888 (30889.38702392578MB)
   free     = 6934208 (6.61297607421875MB)
   99.97859601218858% used

Thank you hashar :) Confirmed that libup is working again.

No out of memory messages for December 12th or 13rd. The one I have found from gerrit log roughly happened at:

[2019-12-14 10:49:05,536]
[2019-12-14 10:49:05,536]
[2019-12-14 10:49:05,542]
[2019-12-14 13:12:08,827]
[2019-12-14 13:12:08,827]
[2019-12-14 15:56:35,167]
[2019-12-14 16:02:30,185]
[2019-12-14 16:11:28,080]
[2019-12-14 16:07:25,808]
[2019-12-14 16:14:29,865]
[2019-12-14 16:27:21,074]
[2019-12-14 16:46:06,978]
[2019-12-14 16:45:08,071]
[2019-12-14 16:43:10,741]
[2019-12-14 16:58:58,182]
[2019-12-14 16:48:06,235]
[2019-12-14 17:24:34,533]
[2019-12-14 17:30:27,134]
[2019-12-14 18:03:58,780]
[2019-12-14 18:24:41,986]
[2019-12-14 18:56:05,281]
[2019-12-14 19:51:11,527]
[2019-12-14 19:47:15,898]
[2019-12-14 19:40:26,615]
[2019-12-14 19:31:29,080]
[2019-12-14 19:30:30,505]
[2019-12-14 19:23:32,859]
[2019-12-14 20:53:35,223]
[2019-12-14 20:41:50,299]
[2019-12-14 20:21:41,008]
[2019-12-14 19:54:08,065]

;)

Looks like the memory usage has just been slowly increasing over the past month...

You can more or less see when libup started running last night:

hashar closed this task as Resolved.Dec 14 2019, 11:02 PM
hashar claimed this task.

tldr: try to throttle libupgrader a bit ;)

Mentioned in SAL (#wikimedia-cloud) [2019-12-15T01:30:50Z] <legoktm> rolling restart of all hound instances because of gerrit-replica outage (T240776, T240763)