Page MenuHomePhabricator

Phabricator does not degrade properly when GitLab is unavailable
Closed, ResolvedPublic

Description

At least twice today I had Phabricator bailing out when browsing a task. The last one was while browsing https://phabricator.wikimedia.org/T325245 which gave me:

Unhandled Exception ("RuntimeException")

WARNING: Invalid argument supplied for foreach()

From logstash, it might be:

[2023-03-28 13:10:34] EXCEPTION: (RuntimeException) Invalid argument supplied for foreach() at [<arcanist>/src/error/PhutilErrorHandler.php:261]
arcanist(), ava(), phabricator(), translations(), wmf-ext-misc()
  #0 <#2> PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<wmf-ext-misc>/src/customfields/GitLabPatchesCustomField.php:113]
  #1 <#2> GitLabPatchesCustomField::getMergeRequestsForTask(string) called at [<wmf-ext-misc>/src/customfields/GitLabPatchesCustomField.php:63]
  #2 <#2> GitLabPatchesCustomField::shouldAppearInPropertyView() called at [<phabricator>/src/infrastructure/customfield/field/PhabricatorCustomField.php:299]
  #3 <#2> PhabricatorCustomField::shouldEnableForRole(string) called at [<phabricator>/src/infrastructure/customfield/field/PhabricatorCustomField.php:93]
  #4 <#2> PhabricatorCustomField::getObjectFields(ManiphestTask, string) called at [<phabricator>/src/applications/maniphest/controller/ManiphestTaskDetailController.php:24]
  #5 <#2> ManiphestTaskDetailController::handleRequest(AphrontRequest) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:284]
  #6 phlog(RuntimeException) called at [<phabricator>/src/aphront/handler/PhabricatorDefaultRequestExceptionHandler.php:41]
  #7 PhabricatorDefaultRequestExceptionHandler::handleRequestThrowable(AphrontRequest, RuntimeException) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:751]
  #8 AphrontApplicationConfiguration::handleThrowable(RuntimeException) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:296]
  #9 AphrontApplicationConfiguration::processRequest(AphrontRequest, PhutilDeferredLog, AphrontPHPHTTPSink, MultimeterControl) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:204]
  #10 AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phabricator>/webroot/index.php:35]'

Maybe due to the widget added to Phabricator and fetching Gitlab patches ( T324149)

Details

TitleReferenceAuthorSource BranchDest Branch
mr widget: exercise greater paranoia about gitlab responsesrepos/phabricator/extensions!6brennenbrennen/mr-widget-status-checkwmf/stable
Customize query in GitLab

Event Timeline

hashar renamed this task from Phabicator task gives "Invalid argument supplied for foreach()" to Phabricator task gives "Invalid argument supplied for foreach()".Mar 28 2023, 1:25 PM

I too see this consistently for this task: T334970 and whenever I go to the home page.

Also happened today during a GitLab maintenance (which was subsequently abandoned) so looks like the widget doesn't fail gracefully.

taavi renamed this task from Phabricator task gives "Invalid argument supplied for foreach()" to Phabricator does not degrade properly when GitLab is unavailable.May 2 2023, 9:36 AM

I thought I had this handled, but clearly I was mistaken. I'll put a more robust guard around the call to the GitLab API.

brennen changed the task status from Open to In Progress.May 2 2023, 3:01 PM
brennen triaged this task as High priority.

Noting here where I messed this up.

This extension was originally a copypasta of the equivalent GerritPatchesCustomField, but instead of using HTTPSFuture::loadContent() as at:

https://gitlab.wikimedia.org/repos/phabricator/extensions/-/blob/wmf/stable/src/customfields/GerritPatchesCustomField.php#L100

I used the resolve() method, probably because I needed to set a header for the API token. What I failed to do was take into account that loadContent() returns false for a failed request, while here I needed to check the status.

@brennen thanks for the quick fix! During the next GitLab maintenance I'll keep a closer look on phabricator and report here what happened.

brennen moved this task from Backlog to Done or Declined on the User-brennen board.

Fix deployed. This should now be robust against an unavailable GitLab. Please re-open if further issues are seen.

Fix deployed. This should now be robust against an unavailable GitLab. Please re-open if further issues are seen.

Thanks @brennen for the fix! I can confirm, during the GitLab upgrades in T336085 Phabricator was available all the time.

Thanks @brennen for the fix! I can confirm, during the GitLab upgrades in T336085 Phabricator was available all the time.

Thanks for the check on this - always good to confirm in practice.