New Impact module: Run backend updating logic on all Wikipedias
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Urbanecm_WMF
	Aug 14 2023, 10:20 AM

Description

In the parent task (T336203), we want to deploy new Impact module to all Wikipedias. For that to be possible, the backend updating logic needs to be available on all Wikipedias. This should be done prior to the actual deployment, so we can be sure updating impact data on each edit doesn't result in technical issues that we didn't observe on the smaller wikis (OTOH, new Impact is already enabled on frwiki or arwiki, both fairly large wikis, so the probability is low, but we should test anyway).

Checklist

Enable wgGERefreshUserImpactDataMaintenanceScriptEnabled on large Wikipedias (P50571 is my proposal)
Observe logs (Logstash + maintenance job logs), ensure impact data gets populated. Suggested duration: 1 week
Resolve T344428: refreshUserImpactJob logs mysterious fatal errors
Enable wgGERefreshUserImpactDataMaintenanceScriptEnabled everywhere
Once impact data populates on all Wikipedias, resolve the task.

Details

	Subject	Repo	Branch	Lines +/-
	Growth: Enable new Impact backend everywhere	operations/mediawiki-config	master	+1 -35
	Growth: Enable new Impact backend on large Wikipedias	operations/mediawiki-config	master	+21 -2

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		KStoller-WMF	T248330 [EPIC] Growth: Positive reinforcement
Resolved		KStoller-WMF	T222310 [EPIC] Positive reinforcement: New Impact Module
Open		KStoller-WMF	T342303 [EPIC] Growth: Deploy Positive reinforcement features to all Wikipedias
Resolved	Nov 1 2023	Urbanecm_WMF	T336203 Positive reinforcement: Deploy the new Impact module to all Wikipedias
Resolved		Urbanecm_WMF	T344143 New Impact module: Run backend updating logic on all Wikipedias
Resolved		Urbanecm_WMF	T344428 refreshUserImpactJob logs mysterious fatal errors
Resolved	PRODUCTION ERROR	Urbanecm_WMF	T344427 Wikimedia\Rdbms\DBConnectionError: Cannot access the database: could not connect to any replica DB server; Unknown error while connecting (db1221)
Resolved	PRODUCTION ERROR	Urbanecm_WMF	T341658 Error: Class 'Cdb\Exception' not found
Resolved	PRODUCTION ERROR	Urbanecm_WMF	T348614 PHP Fatal Error from line 221 of /srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php: require(): Failed opening required '/srv/mediawiki/php-1.41.0-wmf.29/includes/libs/rdbms/exception/DBConnectionError.php' (include_path=

Event Timeline

Urbanecm_WMF created this task.Aug 14 2023, 10:20 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 14 2023, 10:20 AM

Urbanecm_WMF mentioned this in T336203: Positive reinforcement: Deploy the new Impact module to all Wikipedias.Aug 14 2023, 10:21 AM

Urbanecm_WMF claimed this task.Aug 14 2023, 10:23 AM

Urbanecm_WMF triaged this task as Medium priority.

Sgs awarded a token.Aug 14 2023, 4:07 PM

Sgs subscribed.

Change 949033 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Growth: Enable new Impact backend on large Wikipedias

https://gerrit.wikimedia.org/r/949033

Change 949034 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Growth: Enable new Impact backend everywhere

https://gerrit.wikimedia.org/r/949034

Change 949033 merged by jenkins-bot:

[operations/mediawiki-config@master] Growth: Enable new Impact backend on large Wikipedias

https://gerrit.wikimedia.org/r/949033

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:52:19Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]]

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:53:55Z] <urbanecm@deploy1002> urbanecm and d3r1ck01: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessibl

Mentioned in SAL (#wikimedia-operations) [2023-08-16T14:06:32Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] (duration: 14m 13s)

Urbanecm_WMF updated the task description. (Show Details)Aug 16 2023, 2:18 PM

UserImpact jobs started for the P50571 wikis. I saw a bunch of error messages in the logs; some of them happened on the wikis we newly deployed to, some happened elsewhere. Example log entries:

Exception executing job: refreshUserImpactJob Спеціальна: impactDataBatch=array(100) staleBefore=1692174841 requestId=e75360eec015cb3c92297e77 : Wikimedia\Rdbms\DBConnectionError: Cannot access the database: could not connect to any replica DB server; Unknown error while connecting (db1148)

[4d5e3babcf44dff5854e1f00] /rpc/RunSingleJob.php   Wikimedia\Rdbms\DBConnectionError: Cannot access the database: could not connect to any replica DB server; Unknown error while connecting (db1170:3317)

Exception executing job: refreshUserImpactJob Spécial: impactDataBatch=array(100) staleBefore=1692174729 requestId=4d5e3babcf44dff5854e1f00 : Wikimedia\Rdbms\DBConnectionError: Cannot access the database: could not connect to any replica DB server; Unknown error while connecting (db1170:3317)

I'm not sure why the errors are happening. Might be that there was a lot of to do for all the projects enabled at once. Let's monitor the logs for a couple of more days to see if this reoccurs and then we can proceed. Next scheduled execution should be tomorrow UTC morning (05:15 and 07:45 respectively).

Urbanecm_WMF moved this task from Incoming to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.Aug 16 2023, 3:58 PM

The errors reoccured this morning :-(. I believe they have the same root cause as described in T341658#9042285, as connecting to a database is (also) handled in a way similar to opening files. I filled T344428 as an umbrella task.

Considering the amount of logspam, we should probably resolve T344428 and subtasks before proceeding to scale new Impact backend logic further.

Urbanecm_WMF moved this task from In Progress to Blocked / Needs work on the Growth-Team (Sprint 0 (Growth Team)) board.Sep 5 2023, 2:50 PM

KStoller-WMF edited projects, added Growth-Team (Sprint 1 (Growth Team)); removed Growth-Team (Sprint 0 (Growth Team)).Oct 17 2023, 3:08 PM

KStoller-WMF moved this task from Incoming to Blocked / Needs Work on the Growth-Team (Sprint 1 (Growth Team)) board.Oct 17 2023, 3:12 PM

Urbanecm_WMF updated the task description. (Show Details)Oct 25 2023, 12:16 PM

Urbanecm_WMF mentioned this in T344428: refreshUserImpactJob logs mysterious fatal errors.Oct 25 2023, 12:58 PM

Change 949034 merged by jenkins-bot:

[operations/mediawiki-config@master] Growth: Enable new Impact backend everywhere

https://gerrit.wikimedia.org/r/949034

Mentioned in SAL (#wikimedia-operations) [2023-10-26T08:39:24Z] <urbanecm@deploy2002> Started scap: Backport for [[gerrit:949034|Growth: Enable new Impact backend everywhere (T344143)]]

Mentioned in SAL (#wikimedia-operations) [2023-10-26T08:40:48Z] <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:949034|Growth: Enable new Impact backend everywhere (T344143)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-10-26T08:48:53Z] <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:949034|Growth: Enable new Impact backend everywhere (T344143)]] (duration: 09m 29s)

Urbanecm_WMF updated the task description. (Show Details)Oct 26 2023, 8:54 AM

I've enabled the new Impact backend globally. Last remaining step is to validate the impact data got populated. I'll do that in a couple of hours, when the first update (triggered in T344428#9283254) finishes everywhere.

Maintenance_bot removed a project: Patch-For-Review.Oct 26 2023, 9:12 AM

Urbanecm_WMF closed subtask T344428: refreshUserImpactJob logs mysterious fatal errors as Resolved.Oct 26 2023, 11:03 AM

The new Impact backend successfully executed on all Wikipedias. Resolving.

New Impact module: Run backend updating logic on all WikipediasClosed, ResolvedPublicActions