Page MenuHomePhabricator
Create Task
Maniphest T240763

gerrit-replica is returning 502 responses when trying to git clone, breaking libup
Closed, ResolvedPublic
Actions

Description

Dec 14 11:37:08 upgrader-05 celery[317]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/NoTitle.git/': The requested URL returned error: 502
Dec 14 11:38:08 upgrader-05 celery[317]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/NotebookViewer.git/': The requested URL returned error: 502
Dec 14 11:59:25 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/OAuth.git/': The requested URL returned error: 502
Dec 14 12:39:03 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/OpenLayers.git/': The requested URL returned error: 502
Dec 14 12:42:30 upgrader-05 containerd[582]: time="2019-12-14T12:42:30.750283226Z" level=info msg="shim reaped" id=988f678f9be23436178d4e9c8ef139f2480b1c49e4b25937535d901b45764a68
Dec 14 12:45:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PCRGUIInserts.git/': The requested URL returned error: 502
Dec 14 12:47:06 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageAssessments.git/': The requested URL returned error: 502
Dec 14 12:59:18 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageCreationNotif.git/': The requested URL returned error: 502
Dec 14 13:13:08 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageImages.git/': The requested URL returned error: 502
Dec 14 13:14:08 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageInCat.git/': The requested URL returned error: 502
Dec 14 13:26:08 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 13:31:17 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageNotice.git/': The requested URL returned error: 502
Dec 14 13:32:22 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageSchemas.git/': The requested URL returned error: 502
Dec 14 13:33:23 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PageTriage.git/': The requested URL returned error: 502
Dec 14 13:39:29 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PaginateText.git/': The requested URL returned error: 502
Dec 14 13:42:32 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ParentPage.git/': The requested URL returned error: 502
Dec 14 13:43:32 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ParserFun.git/': The requested URL returned error: 502
Dec 14 13:56:46 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PerPageLicense.git/': The requested URL returned error: 502
Dec 14 14:09:51 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PhpTagsFunctions.git/': The requested URL returned error: 502
Dec 14 14:14:45 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PhpTagsSPARQL.git/': The requested URL returned error: 502
Dec 14 14:19:36 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PhpTagsWidgets.git/': The requested URL returned error: 502
Dec 14 14:22:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PhpTagsWiki.git/': The requested URL returned error: 502
Dec 14 14:22:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PictureGame.git/': The requested URL returned error: 502
Dec 14 14:26:30 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PluggableSSO.git/': The requested URL returned error: 502
Dec 14 14:36:23 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PopupAlerts.git/': The requested URL returned error: 502
Dec 14 14:59:26 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PrivateDomains.git/': The requested URL returned error: 502
Dec 14 14:59:26 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Prezi.git/': The requested URL returned error: 502
Dec 14 15:42:01 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 15:46:43 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ProtectionIndicator.git/': The requested URL returned error: 502
Dec 14 15:49:54 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 15:50:41 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PurgeClickThrough.git/': The requested URL returned error: 502
Dec 14 16:01:58 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Push.git/': The requested URL returned error: 502
Dec 14 16:02:45 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PushToWatch.git/': The requested URL returned error: 502
Dec 14 16:14:03 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/PwnedPasswords.git/': The requested URL returned error: 502
Dec 14 16:33:26 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 16:37:23 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 16:45:29 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/QuickResponse.git/': The requested URL returned error: 502
Dec 14 16:48:06 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/QuickSurveys.git/': The requested URL returned error: 502
Dec 14 16:53:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Quiz.git/': The requested URL returned error: 502
Dec 14 17:05:05 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 17:05:09 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/QuizGame.git/': The requested URL returned error: 502
Dec 14 17:15:42 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RSS.git/': The requested URL returned error: 502
Dec 14 17:17:09 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/QuizTabulate.git/': The requested URL returned error: 502
Dec 14 17:24:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomArea.git/': The requested URL returned error: 502
Dec 14 17:35:36 upgrader-05 celery[7309]: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Proxy Error
Dec 14 17:36:38 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomGameUnit.git/': The requested URL returned error: 502
Dec 14 17:47:40 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomImage.git/': The requested URL returned error: 502
Dec 14 17:48:42 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomImageByCategory.git/': The requested URL returned error: 502
Dec 14 17:52:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomPages.git/': The requested URL returned error: 502
Dec 14 17:52:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomInCategory.git/': The requested URL returned error: 502
Dec 14 18:02:01 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ReadingLists.git/': The requested URL returned error: 502
Dec 14 18:04:10 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RandomSelection.git/': The requested URL returned error: 502
Dec 14 18:14:05 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ReassignEdits.git/': The requested URL returned error: 502
Dec 14 18:16:14 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RecentActivityFeed.git/': The requested URL returned error: 502
Dec 14 18:26:08 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Recoin.git/': The requested URL returned error: 502
Dec 14 18:28:18 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RefreshSpecial.git/': The requested URL returned error: 502
Dec 14 18:38:12 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RegexBlock.git/': The requested URL returned error: 502
Dec 14 18:40:21 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RegexFun.git/': The requested URL returned error: 502
Dec 14 18:50:16 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RegexFunctions.git/': The requested URL returned error: 502
Dec 14 18:52:25 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RelatedArticles.git/': The requested URL returned error: 502
Dec 14 19:02:19 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RelatedLinks.git/': The requested URL returned error: 502
Dec 14 19:04:29 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RelatedSites.git/': The requested URL returned error: 502
Dec 14 19:14:23 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Renameuser.git/': The requested URL returned error: 502
Dec 14 19:16:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ReplaceSet.git/': The requested URL returned error: 502
Dec 14 19:26:27 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ReplaceText.git/': The requested URL returned error: 502
Dec 14 19:28:37 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/RevisionCommentSupplement.git/': The requested URL returned error: 502
Dec 14 19:43:30 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SacredText.git/': The requested URL returned error: 502
Dec 14 19:44:00 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SVGEdit.git/': The requested URL returned error: 502
Dec 14 19:55:34 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SafeDelete.git/': The requested URL returned error: 502
Dec 14 19:56:04 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SandboxLink.git/': The requested URL returned error: 502
Dec 14 19:58:02 upgrader-05 celery[7309]: fatal: unable to access 'https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Sarcasm.git/': The requested URL returned error: 502

Related Objects

Event Timeline

Legoktm created this task.Dec 14 2019, 7:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2019, 7:56 PM
Legoktm renamed this task from gerrit-replica is sometimes returning 502 responses when trying to git clone, affecting libup to gerrit-replica is returning 502 responses when trying to git clone, breaking libup.Dec 14 2019, 8:00 PM
Legoktm updated the task description. (Show Details)
Legoktm added a subscriber: thcipriani.Dec 14 2019, 8:05 PM
Stashbot added a comment.Dec 14 2019, 8:06 PM

Mentioned in SAL (#wikimedia-cloud) [2019-12-14T20:06:40Z] <legoktm> systemctl stop libup-celery # because of T240763

Paladox triaged this task as Unbreak Now! priority.Dec 14 2019, 8:15 PM
Paladox added a project: SRE.
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptDec 14 2019, 8:15 PM
Stashbot added a comment.Dec 14 2019, 10:50 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-14T22:50:07Z] <hashar> Restarted Gerrit on gerrit2001 # T240763

hashar subscribed.Dec 14 2019, 10:51 PM

[2019-12-14 16:07:25,808] [HTTP-87043]
WARN org.eclipse.jetty.servlet.ServletHandler : Error for /r/mediawiki/extensions/DataTransfer.git/info/refs
java.lang.OutOfMemoryError: Java heap space

And that is with a 32GB heap:

-XX:+UseG1GC -Xmx32g -Xms32g -XX:G1NewSizePercent=15

There are two threads at 100% CPU.

jstack: F31475924

jmap -heap:

Garbage-First (G1) GC with 23 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 34359738368 (32768.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 20602421248 (19648.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 16777216 (16.0MB)

Heap Usage:
G1 Heap:
   regions  = 2048
   capacity = 34359738368 (32768.0MB)
   used     = 32389869888 (30889.38702392578MB)
   free     = 1969868480 (1878.6129760742188MB)
   94.26692817360163% used
G1 Young Generation:
Eden Space:
   regions  = 0
   capacity = 1962934272 (1872.0MB)
   used     = 0 (0.0MB)
   free     = 1962934272 (1872.0MB)
   0.0% used
Survivor Space:
   regions  = 0
   capacity = 0 (0.0MB)
   used     = 0 (0.0MB)
   free     = 0 (0.0MB)
   0.0% used
G1 Old Generation:
   regions  = 287
   capacity = 32396804096 (30896.0MB)
   used     = 32389869888 (30889.38702392578MB)
   free     = 6934208 (6.61297607421875MB)
   99.97859601218858% used
hashar added projects: Release-Engineering-Team (Development services), Release-Engineering-Team-TODO (201912).Dec 14 2019, 10:51 PM
Legoktm added a comment.Dec 14 2019, 10:53 PM

Thank you hashar :) Confirmed that libup is working again.

hashar added a comment.Dec 14 2019, 10:54 PM

And the trace analysis is https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMTIvMTQvLS1nZXJyaXQyMDAxLWpzdGFjay50eHQtLTIyLTUyLTE1&

hashar added a comment.Dec 14 2019, 10:56 PM

No out of memory messages for December 12th or 13rd. The one I have found from gerrit log roughly happened at:

[2019-12-14 10:49:05,536]
[2019-12-14 10:49:05,536]
[2019-12-14 10:49:05,542]
[2019-12-14 13:12:08,827]
[2019-12-14 13:12:08,827]
[2019-12-14 15:56:35,167]
[2019-12-14 16:02:30,185]
[2019-12-14 16:11:28,080]
[2019-12-14 16:07:25,808]
[2019-12-14 16:14:29,865]
[2019-12-14 16:27:21,074]
[2019-12-14 16:46:06,978]
[2019-12-14 16:45:08,071]
[2019-12-14 16:43:10,741]
[2019-12-14 16:58:58,182]
[2019-12-14 16:48:06,235]
[2019-12-14 17:24:34,533]
[2019-12-14 17:30:27,134]
[2019-12-14 18:03:58,780]
[2019-12-14 18:24:41,986]
[2019-12-14 18:56:05,281]
[2019-12-14 19:51:11,527]
[2019-12-14 19:47:15,898]
[2019-12-14 19:40:26,615]
[2019-12-14 19:31:29,080]
[2019-12-14 19:30:30,505]
[2019-12-14 19:23:32,859]
[2019-12-14 20:53:35,223]
[2019-12-14 20:41:50,299]
[2019-12-14 20:21:41,008]
[2019-12-14 19:54:08,065]

;)

Legoktm added a comment.Dec 14 2019, 11:01 PM

Looks like the memory usage has just been slowly increasing over the past month...

usedMemory.png (499×1 px, 23 KB)

You can more or less see when libup started running last night:

cpu.png (499×1 px, 23 KB)

hashar closed this task as Resolved.Dec 14 2019, 11:02 PM
hashar claimed this task.

tldr: try to throttle libupgrader a bit ;)

Stashbot mentioned this in T240776: Codesearch "Everything" is missing MediaWiki core.Dec 15 2019, 1:30 AM

Mentioned in SAL (#wikimedia-cloud) [2019-12-15T01:30:50Z] <legoktm> rolling restart of all hound instances because of gerrit-replica outage (T240776, T240763)

Jdforrester-WMF moved this task from INBOX to Completed on the Release-Engineering-Team-TODO (201912) board.Dec 17 2019, 5:41 PM
Legoktm mentioned this in T243654: gerrit-replica is OOMing again, returning 502s on git clones.Jan 25 2020, 4:37 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL