This lines up with:

2025-07-01
13:29	<urbanecm@deploy1003>	Finished scap sync-world: Backport for [[gerrit:1164979|Growth: Configure higher impact module edit limits for english and test wiki (T341599)]] (duration: 19m 10s)

which is enwiki-specific. Looking.

The error itself comes from this section:

// We expect the event service to return an array of objects
// in the response body.
// FormatJson::decode will return `null` if the message failed to parse.
// If anything other than an array is parsed we treat it as unexpected
// behaviour, and log the response at error severity.
// See https://phabricator.wikimedia.org/T370428

// $failureInfosByKind should look like:
// {
// 	  "<failure_kind">: [
// 		{ ..., "event": {<failed event here>}, "context": {<failure context here>},
//		{ ... }
//    ],
// }
$failureInfosByKind = FormatJson::decode( $res['body'], true );
if ( is_array( $failureInfosByKind ) ) {
	foreach ( $failureInfosByKind as $failureKind => $failureInfos ) {
		// $failureInfos should not be null or empty.
		// This is just a guard against what the intake
		// service returns (or the behavior of different json parsing methods - possibly).
		// https://www.mediawiki.org/wiki/Manual:Coding_conventions/PHP#empty()
		if ( $failureInfos === null || $failureInfos === [] ) {
			continue;
		}

		// Get the events that failed from the response.
		$failedEvents = array_map(
			static function ( $failureStatus ) {
				return $failureStatus['event'] ?? null;
			},
			$failureInfos // <-- ERROR: This should be array but is int!
		);

which would seem to indicate that somehow the event service is failing to send a correct error response when, presumably, it deals with a request or response that is too large?

The link in the description "Find trace.id in Logstash" reveals that there is also a 500 error coming from EventBus, and that has the error message:

{"status":500,"type":"internal_error","title":"PayloadTooLargeError","detail":"request entity too large","method":"POST","uri":"/v1/events"}

Which would seem to confirm @Urbanecm_WMF's suspicion about what triggered this now.

We certainly do have some impact data updated for enwiki since the deployment:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select count(*) from growthexperiments_user_impact where geui_timestamp like '20250702%';
+----------+
| count(*) |
+----------+
|    23448 |
+----------+
1 row in set (0.006 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

If I'm looking at those 500er events from EventBus than we're seeing over 500 of them starting very recently:

Probably, all have this issue here as the cause.

I think that is enough for us to roll back.

In T398418#10967489, @Michael wrote:

If I'm looking at those 500er events from EventBus than we're seeing over 500 of them starting very recently:

Probably, all have this issue here as the cause.

I think that is enough for us to roll back.

That being said, if all events with an impact of say 1100 and up would be causing this, then I would expect that many more of these errors would occur. So maybe the actual threshold is closer to the upper end of 10K here?

> \MediaWiki\MediaWikiServices::getInstance()->get('GrowthExperimentsUserImpactUpdater')
= GrowthExperiments\UserImpact\GrowthExperimentsUserImpactUpdater {#5139}

> $u = \MediaWiki\MediaWikiServices::getInstance()->get('GrowthExperimentsUserImpactUpdater')
= GrowthExperiments\UserImpact\GrowthExperimentsUserImpactUpdater {#5139}

> sudo $userImpactLookup = $u->userImpactLookup
= GrowthExperiments\UserImpact\ComputedUserImpactLookup {#7737}

> $user = User::newFromId(44217690)
= MediaWiki\User\User {#7871
    +mId: 44217690,
    +mName: null,
    +mActorId: null,
    +mRealName: null,
    +mEmail: null,
    +mTouched: null,
    +mEmailAuthenticated: null,
    +mFrom: "id",
    mId: 44217690,
    mName: null,
    mActorId: null,
    mRealName: null,
    mEmail: null,
    mTouched: null,
    mEmailAuthenticated: null,
    mFrom: "id",
  }

> $impact = $userImpactLookup->getExpensiveUserImpact($user, 1, [])
= GrowthExperiments\UserImpact\ExpensiveUserImpact {#21673}

> strlen(json_encode($impact))
= 11143829

> \MediaWiki\MediaWikiServices::getInstance()->getJobQueueGroup()->push(new \MediaWiki\JobQueue\JobSpecification('refreshUserImpactJob', ['impactDataBatch' => [$user->getId() => json_encode($impact)], 'staleBefore' => \MediaWiki\Utils\MWTimestamp::time() + 1]))

   TypeError  array_map(): Argument #2 ($array) must be of type array, int given.

>

I successfully reproduced this issue for an user on enwiki, see above.

In T398418#10967531, @Urbanecm_WMF wrote:

[...]
I successfully reproduced this issue for an user on enwiki, see above.

Nice! Then let's roll it back and figure something out.

Change #1165866 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] [Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000

https://gerrit.wikimedia.org/r/1165866

Change #1165866 merged by jenkins-bot:

[operations/mediawiki-config@master] [Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000

https://gerrit.wikimedia.org/r/1165866

Mentioned in SAL (#wikimedia-operations) [2025-07-02T12:48:57Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1165865|[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866|[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]]

I'm adding Data-Engineering because this is also surfacing an issues in the EventBus code: It should probably not break with a PHP TypeError in this situation. Also, I'm not sure if the service should respond with a 500 Error in the first place.

Mentioned in SAL (#wikimedia-operations) [2025-07-02T12:51:16Z] <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1165865|[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866|[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-07-02T12:58:40Z] <urbanecm@deploy1003> Finished scap sync-world: Backport for [[gerrit:1165865|[Growth] Move Impact limit configuration to ext-GrowthExperiments (T341599)]], [[gerrit:1165866|[Growth] enwiki: Decrease wgGEUserImpactMaxEdits to 1000 (T398418 T341599)]] (duration: 09m 42s)

Filled two follow-ups:

• T398506: EventBus throws TypeError: array_map(): Argument #2 ($array) must be of type array, int given when the event is too large, for Data Platform Engineering to fix EventBus behaviour
• T398500: [timebox: 3 days] Impact module: Support larger wgGEUserImpactMaxEdits, for the Growth team to ensure we support larger values of the limit

With this, I believe this can be moved to QA, as the error should no longer happen, and follow-up work is tracked elsewhere.