Page MenuHomePhabricator

Some Phabricator boards do not load cards anymore in Chrome 77
Closed, ResolvedPublic

Description

Cannot open Language-Team (Language-2019-July-September) board on Chrome 77. I updated browser version yesterday and cannot load board ever since.
Everything else opens nicely. Firefox opens the board without problems.

The board loads with no tickets in columns. Loading indicator keeps spinning. No errors shown.
Browser asks to kill the tab after some time of inactivity.

Event Timeline

RHo added a subscriber: RHo.Wed, Sep 11, 2:09 PM

Same issue occurring for me on Growth-Team and the project view Growth-Team (Current Sprint) using Chrome Beta Version 77.0.3865.65. Loads fine on Safari and Firefox.

Video showing the error occurring only on these boards:
https://youtu.be/UrGPPAFs-40

Console is also showing this bug if that helps:

Happens to me too (works on safari but breaks on chrome).

Paladox triaged this task as High priority.Wed, Sep 11, 2:10 PM
Aklapper renamed this task from Cannot load Phabricator board on Chrome 77 to Phabricator boards do not load cards in Chrome 77.Wed, Sep 11, 2:14 PM
Aklapper renamed this task from Phabricator boards do not load cards in Chrome 77 to Some Phabricator boards do not load cards anymore in Chrome 77.Wed, Sep 11, 2:20 PM

In the Developer Tools, could you take a look at the "network" panel please if there are any errors, and provide a screenshot of that?
For more information please see https://developer.chrome.com/devtools - thanks!

For me it locks up the tab so badly that I can't even get developer tools to work.

I don't see cards on Growth-Team board, just like @RHo reported.

For me it locks up the tab so badly that I can't even get developer tools to work.

I cannot open DevTools as well. But if I open DevTools while some other site is open and then go to URL for Language-Team (Language-2019-July-September) board, I see some requests are pending forever. Screenshot below. Could not export HAR file.

RHo added a comment.Wed, Sep 11, 2:38 PM

Same issues as @Petar.petkovic - unable to export a HAR file and can only get screenshots showing a bunch of requests pending:


I managed to get script execution to break by setting a "script first statement" breakpoint in the sources debugger. So far I haven't been able to figure out where the problem occurs though.

This loads okay for me in Chrome 77 on macOS 10.14:

Since the board is public, one possible test: does it work in an incognito / logged-out window? If yes, that at least gives us some kind of hint about what's going on (like, possibly some hidden card has some weird behavior). If no, that's very spooky.

RHo added a comment.Wed, Sep 11, 3:33 PM

This loads okay for me in Chrome 77 on macOS 10.14:



Since the board is public, one possible test: does it work in an incognito / logged-out window? If yes, that at least gives us some kind of hint about what's going on (like, possibly some hidden card has some weird behavior). If no, that's very spooky.

Spooky territory – incognito doesn't work for me either:

Spooky territory – incognito doesn't work for me either:

Incognito not working for me. Checked that before reporting the ticket, but forgot to write it down.

My Chrome version is 77.0.3865.75 (Official Build) (64-bit).
OS is Windows 7

Aha! I can reproduce on "Growth-Team", just not the original "Language-Team" project.

I'm not totally sure how to best debug this since all the developer tools seem to pretty much freeze up. I'll try @mmodell's approach of breakpointing the debugger and see if I can guess what's going on, since that seems like the most promising avenue. This might take me a bit to figure out, but I'll see what I can dig up.

@epriestley: It may take several tries to get debugger: At least it did for me: I did several iterations of killing the hung task, setting various breakpoints and refreshing the page before I finally got the debugger to work (Mac OSX / Chrome 77)

Good luck!

Yeah, this is wildly difficult to debug with Chrome's tools. The profiler freezes and I can't get it to unfreeze, stopping execution freezes without breaking anywhere, and when the window freezes the entire window locks up so you have to close it. This discards all your debugger state, and you have to open a new window, reset all your debugger state, then load the page. This isn't even very helpful since breaking on DOMContentLoaded doesn't get anywhere.

So you breakpoint DOMContentLoaded and then guess where other useful breakpoints might be. If you guess wrong, you resume execution into the freeze and have to start all over again. So far, I've been guessing wrong a lot.

One observation is that loading subsets of the board (like "High priority tasks matching query 'a e'") works fine. Other subsets, including small subsets like "High priority tasks matching 'a'", do not. So another attack I can try here might be to find a minimal subquery which reproduces the freeze, although this is also fairly tedious, even though the ids constraint can likely be used to simplify it somewhat.

So far, the freeze doesn't appear to be in page initialization or the project-board behavior. I'm going to guess a few more times and then work on narrowing the reproduction case if I don't get lucky.

When I click "continue" from the beginning of WorkboardDropEffect.js it immediately freezes. Not sure if that's just the last js file or that's the culprit.

I think that's just "the freeze is after WorkboardDropEffect.js" is loaded. All the class definitions load first without actually executing anything or causing side effects (like JX.WorkboardDropEffect). We seem to make it through this part okay, and I think that's what you're seeing.

After that, "behaviors" activate. These actually set the page up and cause side effects and run code. I breakpointed a couple of the most likely ones without finding the freeze (i.e., execution exited out the end, then froze), so I'm going to try breakpointing JX.initBehaviors() and see if I can identify one that's freezing.

These behaviors seem OK:

refresh-csrf
history-install

We then exit initialization and re-enter it later:

phabricator-tooltips
project-boards
...

Uhhh.. this one actually hangs, even though execution previously ran off the end of it. This is extremely perplexing. Maybe I missed something earlier...

Here's what I'm observing:

  • The "project-boards" behavior executes and completes.
  • The debugger visually advances to the next step. That is, the debugger UI redraws and shows that execution has made it through the behavior (for instance, reaching the next ++ in the enclosing for loop).
  • Then, the Chrome UI immediately freezes.

My theory is that the debugger stop is triggering a redraw, and the redraw itself is freezing Chrome. That implies this might be a bug in Chrome itself, since Javascript shouldn't be able to do anything to the DOM that puts it in a state Chrome can't render.

Since the hang doesn't appear to be in Javascript, at least according to what the debugger appears to be reporting, I'm going to try to identify a minimal set of cards to reproduce the issue, then try to reproduce it locally and hopefully narrow down a test case from there.

epriestley added a comment.EditedWed, Sep 11, 4:46 PM

Here's a view of the board with exactly one task that freezes in Chrome for me:

https://phabricator.wikimedia.org/project/board/1114/?filter=T4vGAN6FZuW.

There's nothing special about that task, so I'm going to look for local reproduction cases like "one task in an offscreen column" or something in that vein.

There's nothing special about that task

Actually, I think I got lucky and there is something special about this task. This board does not load in Chrome:

This board loads fine in Chrome:

So I now think this is Chrome rendering engine bug with offscreen elements inside multiple scrolling panes interacting with wordwrapping or backslashes. This is completely crazy but I seem to have a reliable local repro now so I should be able to get a simplified reproduction case out of this.

Broken in Chrome:

Broken in Chrome:

Broken in Chrome:

Works great in Chrome:

The only difference between cases (3) and (4) is that the task title has one fewer "M" so it does not wordwrap.

Here is a simplified document which freezes Chrome 77:

<html>
  <head>
    <style type="text/css">
      .AAA {
        width: 200px;
        white-space: nowrap;
        word-wrap: break-word;
      }
      .BBB {
        white-space: normal;
      }
    </style>
  </head>
  <body>
    <div class="AAA">
      <span>X</span>
      <a class="BBB">MMMMMMMMMMMMMMMMMMMMMMMMM</a>
    </div>
  </body>
</html>

Here is what this document looks like in Safari:

I'll upstream this to Chrome.

This is a pretty fundamental problem in the Chrome rendering engine so we're probably mostly at Google's mercy here. I may be able to find some kind of workaround by changing the wrapping behavior, but it's likely to come at a pretty high UI cost.

I upstreamed this here:

This is maybe a reasonable workaround in the short term, basically removing break-word from cards:

diff --git a/webroot/rsrc/css/phui/workboards/phui-workcard.css b/webroot/rsrc/css/phui/workboards/phui-workcard.css
index 3c6a798fc8..64238cc48b 100644
--- a/webroot/rsrc/css/phui/workboards/phui-workcard.css
+++ b/webroot/rsrc/css/phui/workboards/phui-workcard.css
@@ -36,6 +36,7 @@
 
 .phui-workcard .phui-oi-link {
   white-space: normal;
+  word-wrap: normal;
   font-weight: normal;
   color: {$blacktext};
   margin-left: 2px;

The cost of this change is that tasks which have a single long "word" in their title will overflow instead of wrapping:

This isn't quite as nice, but is probably a reasonable tradeoff to avoid the entire browser window freezing. I'll plan to make this change in the Phabricator upstream in the next release unless Chrome is exceptionally responsive to the upstream report.

Aklapper moved this task from Backlog to Reported Upstream on the Upstream board.
Aklapper moved this task from Backlog to Upstreamed on the Phabricator (Upstream) board.
Isarra added a subscriber: Isarra.Wed, Sep 11, 8:26 PM

Since the downsides are fairly minor, I've landed the workaround above into the Phabricator upstream in https://secure.phabricator.com/D20812.

Mentioned in SAL (#wikimedia-operations) [2019-09-13T20:00:38Z] <twentyafterfour> hotfixing T232600 due to severity of the bug and relative safety of the fix (if this breaks, yell at James_F who twisted my arm and made me do it)

mmodell closed this task as Resolved.Fri, Sep 13, 8:02 PM
mmodell claimed this task.
mmodell awarded a token.

Thank you, @epriestley, for patching over chrome's lack of proper QA. Really heroic debugging that was!

mmodell reassigned this task from mmodell to epriestley.Fri, Sep 13, 8:04 PM

Nice! Thanks for everyone's help hunting this one down, this was definitely one of the most bizarre bugs I've ever run into.