Something must be up with beta. Looking at the dashboard, Echo and TwoColConflict daily tests also started failing around the same time. I'm surely not going to get into the weeds of debugging the ultimate selenium + beta combination today, but you do make a good point that we no longer need the addUserToGroup call, so I'm going to drop those. It would only hide the issue, and other stuff might still fail, but at least it should make tests faster.

In T393428#10796557, @Daimona wrote:

we no longer need the addUserToGroup call, so I'm going to drop those.

I spoke too soon: I can only drop it for api-testing, and not selenium. We need to create an additional organizer account to test multiple organizers, and that new account will need to be added to a group; whether it's event-organizer or sysop shouldn't affect this.

The job was passing 2025-05-28 to 2025-06-16 but then again started to fail on 2025-06-17 and it is still failing.

I'm wondering what's up with the API request. I think addUserToGroup needs to check the response and throw an exception if it lacks the expected data (including the entire response in the output).

The job is no longer failing with this error. Could this task be resolved?

There's another problem (T398310: selenium-daily-beta jobs marked as UNSTABLE), but it's unrelated to this.

In T393428#10987385, @zeljkofilipin wrote:

The job is no longer failing with this error. Could this task be resolved?

Maybe, at least for the time being; but I wouldn't be surprised if it started failing again as it previously did. Checking the graphs, CampaignEvents and TwoColConflict show the same recovery pattern, reinforcing the theory that something is/was up with beta.

Change #1167609 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] selenium: Throw exception when API response lacks expected field

https://gerrit.wikimedia.org/r/1167609

test is failing again

TypeError: Cannot read properties of undefined (reading 'includes')
    at Object.addUserToGroup (/src/node_modules/wdio-mediawiki/Api.js:112:68)
    at async Object.createOrganizerAccount (/src/tests/selenium/EventUtils.js:30:3)
    at async Object.loginAsOrganizer (/src/tests/selenium/EventUtils.js:17:3)
    at async Context.<anonymous> (/src/tests/selenium/specs/enableEventRegistration.js:13:3)

@vaughnwalters Do you need help with this? I can ask my team if we can work on it during our next sprint Test Platform (naŭ 9) (starts next week).

@zeljkofilipin Yes I could probably use some help with this, at least with releasing a new version of wdio-mediawiki with @Daimona's patch so we can see more about what is going on in there that is causing this to fail.

Change #1167609 merged by jenkins-bot:

[mediawiki/core@master] selenium: Throw exception when API response lacks expected field

https://gerrit.wikimedia.org/r/1167609

@vaughnwalters we will release new version of wdio-mediawiki probably tomorrow. We are waiting for some other commits to merge. If Campaign Events wants to use the new version of wdio-mediawiki it will have to use ESM in Selenium tests. (See T396139: Use ESM in all repositories for selenium tests.)

wdio-mediawiki v5.1.0 is released. Let Test Platform know if you need help converting selenium tests to ESM.

In T393428#11016784, @zeljkofilipin wrote:

wdio-mediawiki v5.1.0 is released. Let Test Platform know if you need help converting selenium tests to ESM.

Okay thanks, starting on T400110 for this update.

Note that current campaign events failures occurring on the daily runs are also occurring in the same location in Mediawiki daily tests.

Error: invalidjson: No valid JSON response
    at /src/node_modules/mwbot/src/index.js:254:31
    at tryCatcher (/src/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/src/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/src/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/src/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/src/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/src/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/src/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/src/node_modules/bluebird/js/release/async.js:102:5)
    at Async.drainQueues [as _onImmediate] (/src/node_modules/bluebird/js/release/async.js:15:14)
    at process.processImmediate (node:internal/timers:483:21)

@AMarkossyan-WMF Hi, judging from https://integration.wikimedia.org/ci/job/selenium-daily-beta-CampaignEvents/, things look red as usual, sadly. Have I missed something?

Checking recent logs, the new failure mode is the one reported in T393428#11029515, which doesn't even say what HTTP request it's for. On the bright side, it confirms that something must be madly messed up with API requests in beta. But it could be mwbot being at fault, not necessarily beta itself. Nonetheless, I wanted to find the exact API requests to see if anything went wrong on the beta side. Logstash has nothing useful. Same for api.log, which seems to have no requests at all for 2025-07-29 02:18:xx (from the last daily run). And that's about it for my daily quota of chasing nonsense in E2E tests + beta.

@Daimona Hi, sorry, I misinterpreted this, because it was moved to the "done" column on our Sprint, so I assumed it was resolved.

Instead, I think it was the way @zeljkofilipin communicated that he's no longer the assignee of this ticket.

I'll reopen it now and reassign it to @vaughnwalters (based on the comment history).

Ah I see, and no problem! I don't think I'll have capacity to do a deep dive, but let me know if I can help with something specific.

I just came across T382432 that I filed a while back about SQLite deadlocks. The interesting bit is that the selenium error in that task description matches the one in T393428#11029515. This doesn't necessarily mean that beta is deadlocking, but I suppose it does mean that beta may not be responding (in time) to HTTP requests (hence the "invalid JSON").

Keeping this ticket open for now as I will not be able to confirm this error is consistently gone until after the Error: invalidjson: No valid JSON response is first fixed which is happening upstream in multiple extensions. see T372068#11075998

I merged r1177594 to print additional information and triggered a rebuild. The relevant output is:

11:42:12 [0-0]   response: '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n' +
11:42:12 [0-0]     '<html><head>\n' +
11:42:12 [0-0]     '<title>308 Permanent Redirect</title>\n' +
11:42:12 [0-0]     '</head><body>\n' +
11:42:12 [0-0]     '<h1>Permanent Redirect</h1>\n' +
11:42:12 [0-0]     '<p>The document has moved <a href="https://en.wikipedia.beta.wmcloud.org/w/api.php?format=json">here</a>.</p>\n' +
11:42:12 [0-0]     '</body></html>\n'
11:42:12 [0-0] }

The CI config uses wmflabs.org for beta, but it got moved to wmcloud.org as part of T289318. Of course! I'm going to make a patch to update that, although I wonder if mwbot should actually follow redirects.

Change #1177975 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[integration/config@master] jjb: Update beta domain for selenium-daily tests

https://gerrit.wikimedia.org/r/1177975

Change #1177975 merged by jenkins-bot:

[integration/config@master] jjb: Update beta domain for selenium-daily tests

https://gerrit.wikimedia.org/r/1177975

I triggered another rebuild with the correct beta domain. Now it fails with:

12:08:42 [chrome 136.0.7103.92 linux #0-0] Enable Event Registration @daily
12:08:42 [chrome 136.0.7103.92 linux #0-0]    ✓ is configured correctly
12:08:42 [chrome 136.0.7103.92 linux #0-0]    ✓ requires event data
12:08:42 [chrome 136.0.7103.92 linux #0-0]    ✖ can be enabled
12:08:42 [chrome 136.0.7103.92 linux #0-0]
12:08:42 [chrome 136.0.7103.92 linux #0-0] 2 passing (37.6s)
12:08:42 [chrome 136.0.7103.92 linux #0-0] 1 failing
12:08:42 [chrome 136.0.7103.92 linux #0-0]
12:08:42 [chrome 136.0.7103.92 linux #0-0] 1) Enable Event Registration @daily can be enabled
12:08:42 [chrome 136.0.7103.92 linux #0-0] Can't call getText on element with selector ".mw-page-title-main" because element wasn't found
12:08:42 [chrome 136.0.7103.92 linux #0-0] Error: Can't call getText on element with selector ".mw-page-title-main" because element wasn't found
12:08:42 [chrome 136.0.7103.92 linux #0-0]     at async condition (file:///src/node_modules/expect-webdriverio/lib/matchers/element/toHaveText.js:19:28)
12:08:42 [chrome 136.0.7103.92 linux #0-0]     at async Object.executeCommand (file:///src/node_modules/expect-webdriverio/lib/util/executeCommand.js:2:20)
12:08:42 [chrome 136.0.7103.92 linux #0-0]     at async file:///src/node_modules/expect-webdriverio/lib/matchers/element/toHaveText.js:45:24
12:08:42 [chrome 136.0.7103.92 linux #0-0]     at async waitUntil (file:///src/node_modules/expect-webdriverio/lib/utils.js:36:43)
12:08:42 [chrome 136.0.7103.92 linux #0-0]     at async Object.toHaveText (file:///src/node_modules/expect-webdriverio/lib/matchers/element/toHaveText.js:44:18)
12:08:42 [chrome 136.0.7103.92 linux #0-0]     at async Context.<anonymous> (file:///src/tests/selenium/specs/enableEventRegistration.js:31:3)

In T393428#11077696, @Daimona wrote:

I triggered another rebuild with the correct beta domain. Now it fails with [...]

The screencast shows that form submission fails due to a validation error. Screenshot:

Possibly clock skew or something like that. Maybe form submission happening before our date input field enhancements kick in. I'll look into fixing it.

I have a vague understanding of the issue:

• In PHP, the minimum allowed time is set to the current time, seconds included
• In JS, we set the field's seconds to 0 (for the default, current, min, and max values)
  • We also increase the minutes by one, so e.g. 12:18:03 becomes 12:19:00, as otherwise, 12:18:00 would be considered invalid

The issue we're observing in beta would happen if we set the seconds to 0 without increasing minutes. I reproduced it locally by commenting out that line. I don't know how that could happen in reality, though. Maybe it's just that everything happens too quickly and the field does not have time to update.

Change #1178028 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/extensions/CampaignEvents@master] Make registration form enhancements more robust for browser tests

https://gerrit.wikimedia.org/r/1178028

In T393428#11077955, @Daimona wrote:

I don't know how that could happen in reality, though. Maybe it's just that everything happens too quickly and the field does not have time to update.

I still don't know that, and my best guess is still the same as before. I fixed this by waiting for the time fields to get all the enhancements. I originally tried rounding down the minimum time in PHP, without increasing the minutes in JS. But that then conflicted with the max date, with the check for aggregated answers, and finally with the independent validation in EventFactory. So, I've decided that I won't attempt changing that at this time.

Please add this to Test Platform if you need our help.

Change #1180704 had a related patch set uploaded (by Vaughn Walters; author: Vaughn Walters):

[mediawiki/extensions/CampaignEvents@master] selenium: Fixing some test flake for local runs

https://gerrit.wikimedia.org/r/1180704

Change #1178028 merged by jenkins-bot:

[mediawiki/extensions/CampaignEvents@master] Make registration form enhancements more robust for browser tests

https://gerrit.wikimedia.org/r/1178028

Change #1180704 merged by jenkins-bot:

[mediawiki/extensions/CampaignEvents@master] selenium: Fixing some test flake for local runs

https://gerrit.wikimedia.org/r/1180704