Since no change has been in beta, I think a change in core was cause of this.
I think the offending change is: https://gerrit.wikimedia.org/r/#/c/328377. Too big to revert. @TTO: Can you check and fix beta features?
Change 334717 had a related patch set uploaded (by Ladsgroup):
Add user to the database
Sorry about this. Jenkins showed that these tests were failing on Wikibase, so I fixed that extension, but no other failures showed up.
Change 334717 merged by jenkins-bot:
Use MutableTestUser in tests