Page MenuHomePhabricator

Centrally look for flakey browser tests
Open, Needs TriagePublic

Description

We seem to constantly be fighting a battle against flakey browser tets.

We go through a few prolonged cycles in this area which just make the situation worse all around:

Assuming they are random, and rechecking

  • My CI fails for a seemingly flakey reason
  • type recheck re running ALL jobs (not only the failed on), probably expending 30-60 minutes worth of execution time on CI nodes?
  • the recheck turned it green, so the issue doesn't get flagged up investigated and fixed

Failing to report issues & save needed details

  • Report a seemingly flakey browser test in CI at a high level, linking to logs for CI runs
  • The team that needs to look at the failure may not look for a week or 2 as the ticket goes through their process
  • By time the team looks at the ticket the links to CI builds are dead and the investigation is pretty hard / not worth it at that point?
  • Wait for the process to repeat?

I propose that we experiment with looking at seemingly flaky / failing browser tests centrally in the jenkins logs.

I would hope that:

  • We can catch "trending" issues before people would normally report them
  • We can look at issues with multiple runs and logs being automatically provided to us (instead of waiting for people to report more failure in phabricator)
  • We diagnose and fix the issues faster
  • Everyone is happier, and we reduce the painful loops that I mention above,

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptMar 11 2021, 5:15 PM

I very briefly looked at the main jenkins node, for a single example job and identified one of the โœ–s that indicates a failed test in 124 logs.

addshore@contint2001:/srv/jenkins/builds/wmf-quibble-selenium-php72-docker$ cat */log | grep โœ– | wc -l
124

Looking at wider tests that run selenium seemingly

addshore@contint2001:/srv/jenkins/builds$ cat ./*-selenium-*/*/log | grep โœ– | wc -l
1616

and such a check could make some output like this?

addshore@contint2001:/srv/jenkins/builds$ cat ./*-selenium-*/*/log | grep โœ– | sort | uniq -c | sort -nr
     38 [Chrome 73.0.3683.75 linux #0-9]    โœ– Trims whitespace from Gloss
     38 [Chrome 73.0.3683.75 linux #0-9]    โœ– shows the language and value in edit mode
     38 [Chrome 73.0.3683.75 linux #0-9]    โœ– removes sense when clicked on remove
     38 [Chrome 73.0.3683.75 linux #0-9]    โœ– Removes Gloss
     38 [Chrome 73.0.3683.75 linux #0-9]    โœ– Gloss value unchanged after editing was cancelled
     38 [Chrome 73.0.3683.75 linux #0-9]    โœ– can not save senses with redundant languages
     38 [Chrome 73.0.3683.75 linux #0-9]    โœ– can edit sense and save successfully
     38 [Chrome 73.0.3683.75 linux #0-7]    โœ– can not save lemmas with redundant languages
     38 [Chrome 73.0.3683.75 linux #0-7]    โœ– can be edited multiple times
     38 [Chrome 73.0.3683.75 linux #0-7]    โœ– can be edited
     38 [Chrome 73.0.3683.75 linux #0-5]    โœ– shows id
     38 [Chrome 73.0.3683.75 linux #0-4]    โœ– can edit the lexical category of a Lexeme
     38 [Chrome 73.0.3683.75 linux #0-4]    โœ– can edit the language of a Lexeme
     38 [Chrome 73.0.3683.75 linux #0-3]    โœ– shows Forms header
     38 [Chrome 73.0.3683.75 linux #0-3]    โœ– show Forms grammatical features
     38 [Chrome 73.0.3683.75 linux #0-3]    โœ– has representation
     38 [Chrome 73.0.3683.75 linux #0-3]    โœ– has link to Form
     38 [Chrome 73.0.3683.75 linux #0-3]    โœ– has Forms container
     38 [Chrome 73.0.3683.75 linux #0-3]    โœ– has each representation having a language
     38 [Chrome 73.0.3683.75 linux #0-3]    โœ– has an ID
     38 [Chrome 73.0.3683.75 linux #0-2]    โœ– can be removed
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– trims whitespace from representation
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– has statement list
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– FormId counter is not decremented when old revision is restored
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– FormId counter is not decremented when addForm is undone
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– change multi-variant representations
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– can remove representation
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– can remove first grammatical feature
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– can prefill representation language for lexeme with one lemma
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– can not save representations with redundant languages
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– can edit statements on a new Form
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– can edit representation
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– can add representation
     38 [Chrome 73.0.3683.75 linux #0-1]    โœ– can add grammatical feature
     38 [Chrome 73.0.3683.75 linux #0-11]    โœ– can be added
     38 [Chrome 73.0.3683.75 linux #0-10]    โœ– "before all" hook for Lexeme:Senses
     38 [Chrome 73.0.3683.75 linux #0-0]    โœ– can be added
      8 [chrome 73.0.3683.75 Linux #0-4]        โœ– "before each" hook
      6 [chrome 73.0.3683.75 Linux #0-6]        โœ– show a permission error when opening bridge
      6 [Chrome 73.0.3683.75 linux #0-1]    โœ– "before all" hook
      6 [Chrome 73.0.3683.75 linux #0-0]        โœ– "before all" hook
      5 [Chrome 73.0.3683.75 linux #0-2]    โœ– page should have the button for creating a new filter
      5 [Chrome 73.0.3683.75 linux #0-0]    โœ– submits the search with the correct search terms from all the pill fields
      5 [Chrome 73.0.3683.75 linux #0-0]    โœ– submits the search on double enter when there is text in "These Words" field
      5 [Chrome 73.0.3683.75 linux #0-0]        โœ– saved data is retained (3)
      5 [Chrome 73.0.3683.75 linux #0-0]        โœ– saved data is retained (2)
      5 [Chrome 73.0.3683.75 linux #0-0]        โœ– is visible to logged-in admins
      5 [Chrome 73.0.3683.75 linux #0-0]        โœ– I should receive an error
      5 [Chrome 73.0.3683.75 linux #0-0]        โœ– even if the token is invalid, the ongoing edit is not lost
      5 [Chrome 73.0.3683.75 linux #0-0]        โœ– edit can be saved (3)
      5 [Chrome 73.0.3683.75 linux #0-0]        โœ– edit can be saved (2)
      5 [Chrome 73.0.3683.75 linux #0-0]        โœ– a CSRF token is required to save the filter
      4 [chrome 73.0.3683.75 Linux #0-3]    โœ– can save with tag
      4 [Chrome 73.0.3683.75 linux #0-0]        โœ– cannot save an empty filter
      3 [chrome 73.0.3683.75 Linux #0-5]            โœ– uses the property ID if no label in fallback chain found
      3 [chrome 73.0.3683.75 Linux #0-5]        โœ– shows the current targetReference
      3 [Chrome 73.0.3683.75 linux #0-5]    โœ– Opening a nested reference
      3 [chrome 73.0.3683.75 Linux #0-5]            โœ– is shown and leads to login page
      3 [chrome 73.0.3683.75 Linux #0-5]            โœ– is shown and can be dismissed
      3 [chrome 73.0.3683.75 Linux #0-3]            ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from the English fallback language
      3 [Chrome 73.0.3683.75 linux #0-12]        โœ– is possible to immediately see lemmaLanguageCode field
      3 [Chrome 73.0.3683.75 linux #0-12]        โœ– can create lexeme with language item not bearing language code statement and mis lemma language
      3 [Chrome 73.0.3683.75 linux #0-12]        โœ– can create lexeme with language item not bearing language code statement and en lemma language
      3 [Chrome 73.0.3683.75 linux #0-12]        โœ– "before each" hook for "is not possible to edit"
      3 [Chrome 73.0.3683.75 linux #0-12]        โœ– "before all" hook for "can create lexeme with language item bearing language code statement"
      3 [Chrome 73.0.3683.75 linux #0-12]        โœ– "after each" hook for "is not possible to edit"
      3 [Chrome 73.0.3683.75 linux #0-11]        โœ– "before each" hook for "is not possible to edit"
      3 [Chrome 73.0.3683.75 linux #0-11]        โœ– "after each" hook for "is not possible to edit"
      3 [chrome 73.0.3683.75 Linux #0-0]    โœ– unsupported datatype
      2 [Chrome 73.0.3683.75 linux #0-8]    โœ– "before all" hook for "is in reading mode when opening the item page"
      2 [Chrome 73.0.3683.75 linux #0-7]    โœ– "before each" hook for "is shown when clicking publish"
      2 [chrome 73.0.3683.75 Linux #0-6]        โœ– show the editlink
      2 [chrome 73.0.3683.75 Linux #0-6]        โœ– hide the editlink
      2 [Chrome 73.0.3683.75 linux #0-6]        โœ– "before each" hook for "is in edit mode after clicking the edit button"
      2 [chrome 73.0.3683.75 Linux #0-5]            โœ– uses the label from the page content language
      2 [chrome 73.0.3683.75 Linux #0-5]            โœ– uses the label from the English fallback language
      2 [chrome 73.0.3683.75 Linux #0-5]            โœ– uses the label from a non-English fallback language
      2 [chrome 73.0.3683.75 Linux #0-5]        โœ– shows the current targetValue
      2 [chrome 73.0.3683.75 Linux #0-5]    โœ– opens app in OOUI dialog
      2 [chrome 73.0.3683.75 Linux #0-5]        โœ– is not shown when logged in
      2 [chrome 73.0.3683.75 Linux #0-5]    โœ– indicates loading while app gathers its data
      2 [Chrome 73.0.3683.75 linux #0-5]    โœ– "before each" hook for "shows the warning overlay for anonymous users when clicking the edit button"
      2 [chrome 73.0.3683.75 Linux #0-4]    โœ– shows ErrorUnknown when launching bridge for a non-existent entity
      2 [Chrome 73.0.3683.75 linux #0-4]    โœ– should appear and disappear correctly
      2 [chrome 73.0.3683.75 Linux #0-4]    โœ– can retry saving bridge from ErrorSaving
      2 [chrome 73.0.3683.75 Linux #0-4]    โœ– can go back from a save error both on desktop and mobile
      2 [chrome 73.0.3683.75 Linux #0-4]    โœ– can be relaunched from ErrorUnknown
      2 [chrome 73.0.3683.75 Linux #0-3]            ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the property ID if no label in fallback chain found
      2 [chrome 73.0.3683.75 Linux #0-3]            ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from the page content language
      2 [chrome 73.0.3683.75 Linux #0-3]            ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from a non-English fallback language
      2 [chrome 73.0.3683.75 Linux #0-3]        ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= shows the occurence of errors
      2 [chrome 73.0.3683.75 Linux #0-3]        ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= shows the current targetValue
      2 [chrome 73.0.3683.75 Linux #0-3]    ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= opens app in OOUI dialog
      2 [chrome 73.0.3683.75 Linux #0-3]    โœ– saves an updated value
      2 [Chrome 73.0.3683.75 linux #0-3]    โœ– can expand collapsed references on a protected page as unprivileged user
      2 [chrome 73.0.3683.75 Linux #0-2]    ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= can save with tag
      2 [Chrome 73.0.3683.75 linux #0-1]    โœ– can add a statement using the keyboard
      2 [chrome 73.0.3683.75 Linux #0-1]    โœ– "before each" hook
      2 [chrome 73.0.3683.75 Linux #0-0]    ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= closes on Escape-key event
      1 [Chrome 73.0.3683.75 linux #0-7]    โœ– disappears and saves when clicking publish
      1 [Chrome 73.0.3683.75 linux #0-6]        โœ– "before each" hook for "can edit labels, descriptions, and aliases"
      1 [Chrome 73.0.3683.75 linux #0-5]    โœ– "after each" hook for "never appears for logged in users"
      1 [chrome 73.0.3683.75 Linux #0-4]        โœ– can retry saving without assertuser
      1 [Chrome 73.0.3683.75 linux #0-13]        โœ– can create lexeme with language item not bearing language code statement and mis lemma language
      1 [Chrome 73.0.3683.75 linux #0-13]        โœ– can create lexeme with language item not bearing language code statement and en lemma language
      1 [Chrome 73.0.3683.75 linux #0-13]        โœ– "before each" hook for when blocked
      1 [Chrome 73.0.3683.75 linux #0-13]        โœ– "before all" hook for with default LexemeLanguageCodePropertyId configured
      1 [Chrome 73.0.3683.75 linux #0-13]        โœ– "after each" hook for when blocked
      1 [chrome 73.0.3683.75 Linux #0-1]    โœ– matches according to cursor movement
      1 [chrome 73.0.3683.75 Linux #0-1]    โœ– highlights matching bracket
      1 [chrome 73.0.3683.75 Linux #0-1]    โœ– closes on clicking the cancel button
      1 [Chrome 73.0.3683.75 linux #0-12]        โœ– "before each" hook for when blocked
      1 [Chrome 73.0.3683.75 linux #0-12]        โœ– "after each" hook for when blocked
      1 [chrome 73.0.3683.75 Linux #0-0]    โœ– resets collapsing when changes are discarded
      1 [Chrome 73.0.3683.75 linux #0-0]    โœ– flyout for notices appears when clicked @daily

Quickly checking the top 5, 2 are never referenced on phabricator, but the other 3 are T224546#5499853 T229634 T232364

I'm not sure if โœ– is the only thing that I should be grepping for? It seems to be what I should be looking for for most of the wikibase / wikidata browser tests, but I'm not sure about others...

my bash foo isn't up to figuring out how to tie these greps back to the individual runs that had the errors to allow for deep diving into the loge though at this stage...

I collected some more data today.
We now have some non voting -apache- jobs, and I want to filter those out, so switching to the follwing command:
ls ./*-selenium-*/*/log | grep -v apache | xargs cat | grep โœ– | sort | uniq -c | sort -nr

16 [chrome 73.0.3683.75 Linux #0-4]        โœ– "before each" hook
13 [Chrome 73.0.3683.75 linux #0-3]    โœ– can expand collapsed references on a protected page as unprivileged user
 9 [Chrome 73.0.3683.75 linux #0-4]    โœ– should appear and disappear correctly
 9 [Chrome 73.0.3683.75 linux #0-1]    โœ– can add a statement using the keyboard
 8 [Chrome 73.0.3683.75 linux #0-8]    โœ– "before all" hook for "is in reading mode when opening the item page"
 8 [chrome 73.0.3683.75 Linux #0-5]            โœ– uses the property ID if no label in fallback chain found
 8 [chrome 73.0.3683.75 Linux #0-5]            โœ– uses the label from a non-English fallback language
 8 [chrome 73.0.3683.75 Linux #0-5]            โœ– is shown and leads to login page
 8 [chrome 73.0.3683.75 Linux #0-4]    โœ– can go back from a save error both on desktop and mobile
 8 [chrome 73.0.3683.75 Linux #0-3]    โœ– saves an updated value
 7 [Chrome 73.0.3683.75 linux #0-6]        โœ– "before each" hook for "is in edit mode after clicking the edit button"
 7 [chrome 73.0.3683.75 Linux #0-6]        โœ– "before each" hook
 7 [chrome 73.0.3683.75 Linux #0-5]            โœ– uses the label from the page content language
 7 [chrome 73.0.3683.75 Linux #0-5]            โœ– uses the label from the English fallback language
 7 [chrome 73.0.3683.75 Linux #0-5]        โœ– shows the current targetValue
 7 [chrome 73.0.3683.75 Linux #0-5]        โœ– shows the current targetReference
 7 [chrome 73.0.3683.75 Linux #0-5]    โœ– opens app in OOUI dialog
 7 [chrome 73.0.3683.75 Linux #0-5]            โœ– is shown and can be dismissed
 7 [chrome 73.0.3683.75 Linux #0-5]        โœ– is not shown when logged in
 7 [chrome 73.0.3683.75 Linux #0-5]    โœ– indicates loading while app gathers its data
 7 [chrome 73.0.3683.75 Linux #0-4]    โœ– shows ErrorUnknown when launching bridge for a non-existent entity
 7 [chrome 73.0.3683.75 Linux #0-4]    โœ– can retry saving bridge from ErrorSaving
 7 [chrome 73.0.3683.75 Linux #0-4]    โœ– can be relaunched from ErrorUnknown
 7 [chrome 73.0.3683.75 Linux #0-3]    โœ– can save with tag
 7 [Chrome 73.0.3683.75 linux #0-2]    โœ– the title should match
 7 [chrome 73.0.3683.75 Linux #0-1]    โœ– "before all" hook
 7 [chrome 73.0.3683.75 Linux #0-0]    โœ– unsupported datatype
 7 [Chrome 73.0.3683.75 linux #0-0]    โœ– "before all" hook for "Special:SetLabel"
 6 [Chrome 73.0.3683.75 linux #0-13]        โœ– can create lexeme with language item not bearing language code statement and mis lemma language
 6 [Chrome 73.0.3683.75 linux #0-13]        โœ– can create lexeme with language item not bearing language code statement and en lemma language
 6 [Chrome 73.0.3683.75 linux #0-13]        โœ– "before each" hook for when blocked
 6 [Chrome 73.0.3683.75 linux #0-13]        โœ– "before all" hook for with default LexemeLanguageCodePropertyId configured
 6 [Chrome 73.0.3683.75 linux #0-13]        โœ– "after each" hook for when blocked
 6 [Chrome 73.0.3683.75 linux #0-12]        โœ– "before each" hook for when blocked
 6 [Chrome 73.0.3683.75 linux #0-12]        โœ– "after each" hook for when blocked
 5 [Chrome 73.0.3683.75 linux #0-4]    โœ– can edit the lexical category of a Lexeme
 4 [Chrome 73.0.3683.75 linux #0-9]    โœ– Trims whitespace from Gloss
 4 [Chrome 73.0.3683.75 linux #0-9]    โœ– shows the language and value in edit mode
 4 [Chrome 73.0.3683.75 linux #0-9]    โœ– removes sense when clicked on remove
 4 [Chrome 73.0.3683.75 linux #0-9]    โœ– Removes Gloss
 4 [Chrome 73.0.3683.75 linux #0-9]    โœ– Gloss value unchanged after editing was cancelled
 4 [Chrome 73.0.3683.75 linux #0-9]    โœ– can not save senses with redundant languages
 4 [Chrome 73.0.3683.75 linux #0-9]    โœ– can edit sense and save successfully
 4 [Chrome 73.0.3683.75 linux #0-7]    โœ– can not save lemmas with redundant languages
 4 [Chrome 73.0.3683.75 linux #0-7]    โœ– can be edited multiple times
 4 [Chrome 73.0.3683.75 linux #0-7]    โœ– can be edited
 4 [Chrome 73.0.3683.75 linux #0-7]    โœ– "before all" hook for "is shown when clicking publish"
 4 [Chrome 73.0.3683.75 linux #0-5]    โœ– shows id
 4 [Chrome 73.0.3683.75 linux #0-5]    โœ– "before all" hook for "shows the warning overlay for anonymous users when clicking the edit button"
 4 [Chrome 73.0.3683.75 linux #0-4]    โœ– can edit the language of a Lexeme
 4 [Chrome 73.0.3683.75 linux #0-4]    โœ– "before all" hook for "should appear and disappear correctly"
 4 [Chrome 73.0.3683.75 linux #0-3]    โœ– shows Forms header
 4 [Chrome 73.0.3683.75 linux #0-3]    โœ– show Forms grammatical features
 4 [Chrome 73.0.3683.75 linux #0-3]    โœ– has representation
 4 [Chrome 73.0.3683.75 linux #0-3]    โœ– has link to Form
 4 [Chrome 73.0.3683.75 linux #0-3]    โœ– has Forms container
 4 [Chrome 73.0.3683.75 linux #0-3]    โœ– has each representation having a language
 4 [Chrome 73.0.3683.75 linux #0-3]    โœ– has an ID
 4 [Chrome 73.0.3683.75 linux #0-2]    โœ– can be removed
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– trims whitespace from representation
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– has statement list
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– FormId counter is not decremented when old revision is restored
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– FormId counter is not decremented when addForm is undone
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– change multi-variant representations
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– can remove representation
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– can remove first grammatical feature
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– can prefill representation language for lexeme with one lemma
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– can not save representations with redundant languages
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– can edit statements on a new Form
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– can edit representation
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– can add representation
 4 [Chrome 73.0.3683.75 linux #0-1]    โœ– can add grammatical feature
 4 [Chrome 73.0.3683.75 linux #0-11]    โœ– can be added
 4 [Chrome 73.0.3683.75 linux #0-10]    โœ– "before all" hook for Lexeme:Senses
 4 [Chrome 73.0.3683.75 linux #0-0]    โœ– can be added
 3 [Chrome 73.0.3683.75 linux #0-7]    โœ– disappears and saves when clicking publish
 3 [Chrome 73.0.3683.75 linux #0-7]    โœ– "before each" hook for "is shown when clicking publish"
 3 [Chrome 73.0.3683.75 linux #0-5]    โœ– "before each" hook for "shows the warning overlay for anonymous users when clicking the edit button"
 3 [chrome 73.0.3683.75 Linux #0-3]            ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from the English fallback language
 3 [Chrome 73.0.3683.75 linux #0-12]        โœ– is possible to immediately see lemmaLanguageCode field
 3 [Chrome 73.0.3683.75 linux #0-12]        โœ– can create lexeme with language item not bearing language code statement and mis lemma language
 3 [Chrome 73.0.3683.75 linux #0-12]        โœ– can create lexeme with language item not bearing language code statement and en lemma language
 3 [Chrome 73.0.3683.75 linux #0-12]        โœ– "before each" hook for "is not possible to edit"
 3 [Chrome 73.0.3683.75 linux #0-12]        โœ– "before all" hook for "can create lexeme with language item bearing language code statement"
 3 [Chrome 73.0.3683.75 linux #0-12]        โœ– "after each" hook for "is not possible to edit"
 3 [Chrome 73.0.3683.75 linux #0-11]        โœ– "before each" hook for "is not possible to edit"
 3 [Chrome 73.0.3683.75 linux #0-11]        โœ– "after each" hook for "is not possible to edit"
 3 [Chrome 73.0.3683.75 linux #0-0]    โœ– submits the search with the correct search terms from all the pill fields
 2 [Chrome 73.0.3683.75 linux #0-6]        โœ– can edit labels, descriptions, and aliases
 2 [Chrome 73.0.3683.75 linux #0-5]    โœ– Opening a nested reference
 2 [Chrome 73.0.3683.75 linux #0-4]    โœ– shows search preview in collapsed state
 2 [chrome 73.0.3683.75 Linux #0-3]            ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the property ID if no label in fallback chain found
 2 [chrome 73.0.3683.75 Linux #0-3]            ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from the page content language
 2 [chrome 73.0.3683.75 Linux #0-3]            ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from a non-English fallback language
 2 [chrome 73.0.3683.75 Linux #0-3]        ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= shows the occurence of errors
 2 [chrome 73.0.3683.75 Linux #0-3]        ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= shows the current targetValue
 2 [chrome 73.0.3683.75 Linux #0-3]    ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= opens app in OOUI dialog
 2 [Chrome 73.0.3683.75 linux #0-3]    โœ– submits the search taking into consideration all entered criteria
 2 [chrome 73.0.3683.75 Linux #0-2]    ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= can save with tag
 2 [Chrome 73.0.3683.75 linux #0-15]    โœ– "before each" hook for Special:Watchlist
 2 [Chrome 73.0.3683.75 linux #0-14]    โœ– shows lemmas in title links to lexemes
 2 [chrome 73.0.3683.75 Linux #0-0]    ha:////4Jnzv/KYMmvaFk73pZiiMWIJmramBgvmeeSOQYtaPOEPAAAAox+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDDI2RQXJOYpFJdU5qTaKoEttlJQdnYxAAJrJTsAN/bC9aUAAAA=โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= closes on Escape-key event
 1 [chrome 73.0.3683.75 Linux #0-4]        โœ– can login and retry saving
 1 [Chrome 73.0.3683.75 linux #0-15]    โœ– "before all" hook for Special:Watchlist
 1 [Chrome 73.0.3683.75 linux #0-14]    โœ– shows lemmas in title links to lexemes on Special:RecentChanges
 1 [chrome 73.0.3683.75 Linux #0-1]    โœ– stores responses for users with NONE group when survey is submitted
 1 [chrome 73.0.3683.75 Linux #0-1]    โœ– paragraph edits can be reverted
 1 [chrome 73.0.3683.75 Linux #0-1]    โœ– closes on clicking the cancel button
 1 [Chrome 73.0.3683.75 linux #0-0]    โœ– submits the search on double enter when there is text in "These Words" field
 1 [chrome 73.0.3683.75 Linux #0-0]    โœ– is enabled for new user
 1 [chrome 73.0.3683.75 Linux #0-0]    โœ– Heading shows the logged-in user's name

The output can be trimmed a bit with a regex \s+(\d+).*(โœ–.*) -> $1 $2\n.
And I'm starting to throw it in a spreadsheet for now to see what we can spot.
https://docs.google.com/spreadsheets/d/1zllaM9T7RxOF29zu5SN0KM2eqtNdWfqfnXkOYE3XpUo

These errors appear to be happening less frequently, infact all of these errors seem to go from 38 -> 4 occurrences...

  • โœ– "before all" hook for Lexeme:Senses
  • โœ– can add grammatical feature
  • โœ– can add representation
  • โœ– can be edited
  • โœ– can be edited multiple times
  • โœ– can be removed
  • โœ– can edit representation
  • โœ– can edit sense and save successfully
  • โœ– can edit statements on a new Form
  • โœ– can edit the language of a Lexeme
  • โœ– can edit the lexical category of a Lexeme
  • โœ– can not save lemmas with redundant languages
  • โœ– can not save representations with redundant languages
  • โœ– can not save senses with redundant languages
  • โœ– can prefill representation language for lexeme with one lemma
  • โœ– can remove first grammatical feature
  • โœ– can remove representation
  • โœ– change multi-variant representations
  • โœ– FormId counter is not decremented when addForm is undone
  • โœ– FormId counter is not decremented when old revision is restored
  • โœ– Gloss value unchanged after editing was cancelled
  • โœ– has an ID
  • โœ– has each representation having a language
  • โœ– has Forms container
  • โœ– has link to Form
  • โœ– has representation
  • โœ– has statement list
  • โœ– Removes Gloss
  • โœ– removes sense when clicked on remove
  • โœ– show Forms grammatical features
  • โœ– shows Forms header
  • โœ– shows id
  • โœ– shows the language and value in edit mode
  • โœ– Trims whitespace from Gloss
  • โœ– trims whitespace from representation

Some new one seem to be appearing:

  • โœ– "after each" hook for when blocked (2 -> 12)
  • โœ– "before each" hook (10->23)
  • โœ– "before each" hook for when blocked (2 -> 12)
  • โœ– can expand collapsed references on a protected page as unprivileged user (2 -> 13)
  • โœ– can go back from a save error both on desktop and mobile (2 -> 8)
  • โœ– is shown and leads to login page (3 -> 8)
  • โœ– saves an updated value (2 -> 8)
  • โœ– should appear and disappear correctly (2 -> 9)

and some more...

For the logs that currently exist on jenkins there are 500 browser test failures, in comparison to the previous dump of logs which had ~1600

Running this once more:

ls /srv/jenkins/builds/*-selenium-*/*/log | grep -v apache | xargs cat | grep โœ– | sort | uniq -c | sort -nr

The output can be trimmed a bit with a regex \s+(\d+).*(โœ–.*) -> $1 $2\n

We get:

6 โœ– can edit the lexical category of a Lexeme
5 โœ– FormId counter is not decremented when old revision is restored
4 โœ– Trims whitespace from Gloss
4 โœ– shows the language and value in edit mode
4 โœ– removes sense when clicked on remove
4 โœ– Removes Gloss
4 โœ– Gloss value unchanged after editing was cancelled
4 โœ– can not save senses with redundant languages
4 โœ– can edit sense and save successfully
4 โœ– can not save lemmas with redundant languages
4 โœ– can be edited multiple times
4 โœ– can be edited
4 โœ– shows id
4 โœ– can edit the language of a Lexeme
4 โœ– shows Forms header
4 โœ– show Forms grammatical features
4 โœ– has representation
4 โœ– has link to Form
4 โœ– has Forms container
4 โœ– has each representation having a language
4 โœ– has an ID
4 โœ– can be removed
4 โœ– trims whitespace from representation
4 โœ– has statement list
4 โœ– FormId counter is not decremented when addForm is undone
4 โœ– change multi-variant representations
4 โœ– can remove representation
4 โœ– can remove first grammatical feature
4 โœ– can prefill representation language for lexeme with one lemma
4 โœ– can not save representations with redundant languages
4 โœ– can edit statements on a new Form
4 โœ– can edit representation
4 โœ– can add representation
4 โœ– can add grammatical feature
4 โœ– can be added
4 โœ– "before all" hook for Lexeme:Senses
4 โœ– cannot save an empty filter
4 โœ– can be added
3 โœ– Opening a nested reference
3 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from the English fallback language
3 โœ– Check user page is editable
3 โœ– is possible to immediately see lemmaLanguageCode field
3 โœ– can create lexeme with language item not bearing language code statement and mis lemma language
3 โœ– can create lexeme with language item not bearing language code statement and en lemma language
3 โœ– "before each" hook for "is not possible to edit"
3 โœ– "before all" hook for "can create lexeme with language item bearing language code statement"
3 โœ– "after each" hook for "is not possible to edit"
3 โœ– "before each" hook for "is not possible to edit"
3 โœ– "after each" hook for "is not possible to edit"
2 โœ– "before each" hook for "is in edit mode after clicking the edit button"
2 โœ– indicates loading while app gathers its data
2 โœ– "before each" hook
2 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the property ID if no label in fallback chain found
2 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from the page content language
2 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= uses the label from a non-English fallback language
2 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= shows the occurence of errors
2 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= shows the current targetValue
2 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= opens app in OOUI dialog
2 โœ– can expand collapsed references on a protected page as unprivileged user
2 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= can save with tag
2 โœ–ha:////4Pwnj1riiGD98wSXevu/34pSmwxVTFrID0iRXdT+WGGeAAAAkR+LCAAAAAAAAP9b85aBtbiIQT2jNKU4P0+vIKc0PTOvWC8xrzgzOT8nv0gvODO3ICfVoyQ3xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA= closes on Escape-key event
2 โœ– should work for addition
1 โœ– "before each" hook for "is in reading mode when opening the item page"
1 โœ– disappears and saves when clicking publish
1 โœ– "before each" hook for "is shown when clicking publish"
1 โœ– can edit labels, descriptions, and aliases
1 โœ– uses the property ID if no label in fallback chain found
1 โœ– is shown and leads to login page
1 โœ– "before each" hook for "shows the warning overlay for anonymous users when clicking the edit button"
1 โœ– should appear and disappear correctly
1 โœ– should show a preview page
1 โœ– saves an updated value
1 โœ– stores responses for users with NONE group when survey is submitted
1 โœ– closes on clicking the cancel button
1 โœ– can add a statement using the keyboard
1 โœ– "before each" hook

And adding to the spreadsheet https://docs.google.com/spreadsheets/d/1zllaM9T7RxOF29zu5SN0KM2eqtNdWfqfnXkOYE3XpUo/edit

This snapshot sees ~ half the number of browser test errors as in the last snapshot! Woo!
From 500 down to 228.
(Though this is just a number, figuring this out as a % of number of runs might make more sense)

The last period saw an increase from 228 to 707.

The top offenders would be:

  • โœ– "before all" hook
  • โœ– "before each" hook

But there is a general increase across the board on the failure front.

One thing I notice for the 2 error outputs here is that they are very general and having more specific error messages might lead to a different grouping.

For example:

[Chrome 73.0.3683.75 linux #0-8] Termbox: reading
[Chrome 73.0.3683.75 linux #0-8]    โœ– "before all" hook for "is in reading mode when opening the item page"

Is a nice error message for grouping, vs

[chrome 73.0.3683.75 Linux #0-1] Spec: /workspace/src/extensions/Wikibase/client/data-bridge/tests/selenium/specs/canCancel.js
[chrome 73.0.3683.75 Linux #0-1] Running: chrome (v73.0.3683.75) on Linux
[chrome 73.0.3683.75 Linux #0-1] Session ID: a07d05924209c6f0168018801886ccdd
[chrome 73.0.3683.75 Linux #0-1]
[chrome 73.0.3683.75 Linux #0-1] App
[chrome 73.0.3683.75 Linux #0-1]    โœ– "before all" hook

and

[Chrome 88.0.4324.182 linux #0-1] Spec: /workspace/src/extensions/AbuseFilter/tests/selenium/specs/importingFilters.js
[Chrome 88.0.4324.182 linux #0-1] Running: Chrome (v88.0.4324.182) on linux
[Chrome 88.0.4324.182 linux #0-1] Session ID: e3b72217-1f08-4752-9545-0ab5d4dd82bd
[Chrome 88.0.4324.182 linux #0-1]
[Chrome 88.0.4324.182 linux #0-1] When importing a filter
[Chrome 88.0.4324.182 linux #0-1]    โœ– "before all" hook

Which are not.

So this error is unlikely to be a single test misbehaving, rather just blanket failures caused by CI being overworked?
SO I won't bother digging into this spike at all...
IT might be worth filling in some followups to make this output better though.
So some pastest of grepped output for before all https://phabricator.wikimedia.org/P15455 and before each https://phabricator.wikimedia.org/P15456 were captured

Running again today (1 week after the last check)
Things dropped from ~700 to ~350.
THis โœ– "before each" hook one remains a big failer.
I think what I said in T277205#7013062 still applies though in terms of making these failures more distinct.

I'm going to deep dive into a recent failure again in T281122 which was recently created.

Change 682705 had a related patch set uploaded (by Addshore; author: Addshore):

[mediawiki/extensions/Wikibase@master] bridge: Switch tests waiting for bridge to open from 10 -> 20s wait

https://gerrit.wikimedia.org/r/682705

Change 682705 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] bridge: Switch tests waiting for bridge to open from 10 -> 20s wait

https://gerrit.wikimedia.org/r/682705

Change 683141 had a related patch set uploaded (by Addshore; author: Addshore):

[mediawiki/extensions/Wikibase@REL1_36] bridge: Switch tests waiting for bridge to open from 10 -> 20s wait

https://gerrit.wikimedia.org/r/683141

3x the number of errors looking at all recorded logs today.

Some big culprits are:

โœ– "before each" hook 60
โœ– "before all" hook 28
โœ– unsupported datatype 18
โœ– uses the label from a non-English fallback language 22
โœ– uses the label from the English fallback language 22
โœ– uses the label from the page content language 22
โœ– uses the property ID if no label in fallback chain found 25
โœ– shows the current targetReference 18
โœ– shows the current targetValue 18
โœ– shows ErrorUnknown when launching bridge for a non-existent entity 18
โœ– saves an updated value 20
โœ– opens app in OOUI dialog 18
โœ– is shown and can be dismissed 18
โœ– is shown and leads to login page 19
โœ– is not shown when logged in 19
โœ– indicates loading while app gathers its data 23
โœ– can save with tag 20
โœ– can retry saving bridge from ErrorSaving 18
โœ– can expand collapsed references on a protected page as unprivileged user 18
โœ– can go back from a save error both on desktop 18
โœ– can be relaunched from ErrorUnknown 18
โœ– can add a statement using the keyboard 21
โœ– "before each" hook for "is in edit mode after clicking the edit button" 18

Will have to wait some more time to see if the bridge patch above had any impact.
It could also be good to deep dive into another one of these seemingly random but repeating test failures, and see if anything possibly lines up with what was happening in T281122

Change 685474 had a related patch set uploaded (by WMDE-leszek; author: Addshore):

[mediawiki/extensions/Wikibase@REL1_35] bridge: Switch tests waiting for bridge to open from 10 -> 20s wait

https://gerrit.wikimedia.org/r/685474

Change 685474 abandoned by WMDE-leszek:

[mediawiki/extensions/Wikibase@REL1_35] bridge: Switch tests waiting for bridge to open from 10 -> 20s wait

Reason:

https://gerrit.wikimedia.org/r/685474

Just looked again and we are at a pretty low point in terms of errors right now :)

The big culprits for the last slot or time are:

12 , โœ– FormId counter is not decremented when old revision is restored
12 , โœ– FormId counter is not decremented when addForm is undone
7 , โœ– can go back from a save error both on desktop and mobile
4 , โœ– checks for welcome message after signup
4 , โœ– show a permission error when opening bridge
4 , โœ– Trims whitespace from Gloss
4 , โœ– shows the language and value in edit mode
4 , โœ– removes sense when clicked on remove
4 , โœ– Removes Gloss
4 , โœ– Gloss value unchanged after editing was cancelled
4 , โœ– can not save senses with redundant languages
4 , โœ– can edit sense and save successfully
4 , โœ– can not save lemmas with redundant languages
4 , โœ– can be edited multiple times
4 , โœ– can be edited
4 , โœ– shows id
4 , โœ– can edit the lexical category of a Lexeme
4 , โœ– can edit the language of a Lexeme
4 , โœ– shows Forms header
4 , โœ– show Forms grammatical features
4 , โœ– has representation
4 , โœ– has link to Form
4 , โœ– has Forms container
4 , โœ– has each representation having a language
4 , โœ– has an ID
4 , โœ– can be removed
4 , โœ– trims whitespace from representation
4 , โœ– has statement list
4 , โœ– FormId counter is not decremented when old revision is restored
4 , โœ– FormId counter is not decremented when addForm is undone
4 , โœ– change multi-variant representations
4 , โœ– can remove representation
4 , โœ– can remove first grammatical feature
4 , โœ– can prefill representation language for lexeme with one lemma
4 , โœ– can not save representations with redundant languages
4 , โœ– can edit statements on a new Form
4 , โœ– can edit representation
4 , โœ– can add representation
4 , โœ– can add grammatical feature
4 , โœ– can be added
4 , โœ– "before all" hook for Lexeme:Senses
4 , โœ– can be added

Looks like it could make sense to dive into Lexeme failures next.
I'd want to dive into logs etc first thought before just upping timeouts..
(There are various tickets relating to lexeme browser tests that could likely be crosslinked)

Just checked this again looking at the last period and there were only 225 failures (lowest since i started looking), the peak i have seen is 1.6k
(I have updated the sheet)
The same sort of failure seem to happen, it'd be good to try and break out some more tasks from this.

Change 683141 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_36] bridge: Switch tests waiting for bridge to open from 10 -> 20s wait

https://gerrit.wikimedia.org/r/683141