s1 done. Only s8 left.

Change 779166 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] labs: Set actor migration to read new

https://gerrit.wikimedia.org/r/779166

Change 779166 merged by jenkins-bot:

[operations/mediawiki-config@master] labs: Set actor migration to read new

https://gerrit.wikimedia.org/r/779166

Only wikidata left and it's mostly done. I'll slowly start read new in production once wikidata is done.

Change 783845 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] maintain-views: Drop views on revision_actor_temp

https://gerrit.wikimedia.org/r/783845

Change 784685 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] ActorMigration: Start reading from rev_actor field in group0

https://gerrit.wikimedia.org/r/784685

s8 is also done. Nothing left. I'm going to enable read new next week.

Change 784685 merged by jenkins-bot:

[operations/mediawiki-config@master] ActorMigration: Start reading from rev_actor field in group0

https://gerrit.wikimedia.org/r/784685

Mentioned in SAL (#wikimedia-operations) [2022-04-25T07:11:14Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784685|ActorMigration: Start reading from rev_actor field in group0 (T275246)]] (duration: 00m 50s)

Ran a double check on all dbs and confirm nothing left to do in production.

Change 785948 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] ActorMigration: Read from rev_actor field in all of small wikis

https://gerrit.wikimedia.org/r/785948

Change 785948 merged by jenkins-bot:

[operations/mediawiki-config@master] ActorMigration: Read from rev_actor field in all of small wikis

https://gerrit.wikimedia.org/r/785948

Mentioned in SAL (#wikimedia-operations) [2022-04-25T23:01:48Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785948|ActorMigration: Read from rev_actor field in all of small wikis (T275246)]] (duration: 00m 57s)

Change 786341 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Set actor migration to read new for medium wikis

https://gerrit.wikimedia.org/r/786341

Change 786341 merged by jenkins-bot:

[operations/mediawiki-config@master] Set actor migration to read new for medium wikis

https://gerrit.wikimedia.org/r/786341

Mentioned in SAL (#wikimedia-operations) [2022-04-26T16:32:09Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786341|Set actor migration to read new for medium wikis (T275246)]] (duration: 02m 01s)

Change 786936 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Set actor migration to READ NEW everywhere

https://gerrit.wikimedia.org/r/786936

Change 786936 merged by jenkins-bot:

[operations/mediawiki-config@master] Set actor migration to READ NEW everywhere

https://gerrit.wikimedia.org/r/786936

Mentioned in SAL (#wikimedia-operations) [2022-04-27T12:07:10Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786936|Set actor migration to READ NEW everywhere (T275246)]] (duration: 00m 53s)

This is now reading from rev_actor field everywhere. Next step is to remove the view and then we can start to stop writing to the temp table.

Change 783845 merged by Ladsgroup:

[operations/puppet@production] maintain-views: Drop views on revision_actor_temp

https://gerrit.wikimedia.org/r/783845

Change 789271 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] dbproxy: depool clouddb1013-1017 to update views

https://gerrit.wikimedia.org/r/789271

Change 789271 merged by Razzi:

[operations/puppet@production] dbproxy: depool clouddb1013-1017 to update views

https://gerrit.wikimedia.org/r/789271

Change 789291 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] dbproxy: Repool the old batch, Depool the new one

https://gerrit.wikimedia.org/r/789291

Change 789291 merged by Ladsgroup:

[operations/puppet@production] dbproxy: Repool the old batch, Depool the new one

https://gerrit.wikimedia.org/r/789291

Change 789308 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] dbproxy: Repool clouddb10(17|18|19|20)

https://gerrit.wikimedia.org/r/789308

Change 789308 merged by Ladsgroup:

[operations/puppet@production] dbproxy: Repool clouddb10(17|18|19|20)

https://gerrit.wikimedia.org/r/789308

We are ready to stop writing to the temp table now

Change 789314 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] labs: Set actor migration to write new only

https://gerrit.wikimedia.org/r/789314

Change 789314 merged by jenkins-bot:

[operations/mediawiki-config@master] labs: Set actor migration to write new only

https://gerrit.wikimedia.org/r/789314

confirmed it's working in beta as expected. Moving to group0!!!!!11%$^%&

Change 789558 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Stop writing to temp actor table in group0

https://gerrit.wikimedia.org/r/789558

Change 789558 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop writing to temp actor table in group0

https://gerrit.wikimedia.org/r/789558

Mentioned in SAL (#wikimedia-operations) [2022-05-05T18:34:37Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:789558|Stop writing to temp actor table in group0 (T275246)]] (duration: 00m 50s)

Working as expected:

wikiadmin@10.64.0.44(mediawikiwiki)> select * from revision order by rev_id desc limit 5;
+---------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
| rev_id  | rev_page | rev_comment_id | rev_actor | rev_timestamp  | rev_minor_edit | rev_deleted | rev_len | rev_parent_id | rev_sha1                        |
+---------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
| 5198358 |     6905 |              0 |  18548471 | 20220505183752 |              0 |           0 |    4441 |       4693495 | rpok3xa7dpfgexkz7hh572c4uyrfpb0 |
| 5198357 |   127100 |              0 |  18548471 | 20220505183751 |              0 |           0 |      50 |       4549557 | 08ycj4qw81r2hk3zm95atcfigxibz0w |
| 5198356 |  1528085 |              0 |  18365305 | 20220505183359 |              0 |           0 |     450 |       5148536 | n9ijs4rmktse5fwujot3hb0lgmkre7n |
| 5198355 |    82610 |              0 |  18365305 | 20220505183206 |              0 |           0 |    8350 |       5198353 | 7m2uk076lkt8g6b87y0rnloxs0t8ofj |
| 5198354 |  1541652 |              0 |  18365305 | 20220505183205 |              0 |           0 |     276 |             0 | aildubs9ltc6v3ct0a3b72hpuoulnwx |
+---------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
5 rows in set (0.000 sec)

wikiadmin@10.64.0.44(mediawikiwiki)> select * from revision_actor_temp order by revactor_rev desc limit 5;
+--------------+----------------+--------------------+---------------+
| revactor_rev | revactor_actor | revactor_timestamp | revactor_page |
+--------------+----------------+--------------------+---------------+
|      5198356 |       18365305 | 20220505183359     |       1528085 |
|      5198355 |       18365305 | 20220505183206     |         82610 |
|      5198354 |       18365305 | 20220505183205     |       1541652 |
|      5198353 |       18365305 | 20220505183137     |         82610 |
|      5198352 |       18365305 | 20220505183135     |       1541651 |
+--------------+----------------+--------------------+---------------+
5 rows in set (0.001 sec)

Will push it to every wiki next week if no errors show up.

Change 790020 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Stop writing to rev_actor_temp table in group1

https://gerrit.wikimedia.org/r/790020

Nothing has written to the table since we disabled it:

root@db1157:/srv/sqldata/mediawikiwiki# ls -l | grep -i temp
-rw-rw---- 1 mysql mysql       2524 Jun  7  2021 revision_actor_temp.frm
-rw-rw---- 1 mysql mysql  922746880 May  5 18:39 revision_actor_temp.ibd

Change 790020 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop writing to rev_actor_temp table in group1

https://gerrit.wikimedia.org/r/790020

Mentioned in SAL (#wikimedia-operations) [2022-05-09T04:40:02Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:790020|Stop writing to rev_actor_temp table in group1 (T275246)]] (duration: 05m 06s)

Change 790618 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Stop writing to revision_actor_temp table everywhere

https://gerrit.wikimedia.org/r/790618

Change 790618 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop writing to revision_actor_temp table everywhere

https://gerrit.wikimedia.org/r/790618

Mentioned in SAL (#wikimedia-operations) [2022-05-10T08:05:02Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:790618|Stop writing to revision_actor_temp table everywhere (T275246)]] (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2023-01-24T21:38:40Z] <zabe> running migrateRevisionCommentTemp.php on testcommonswiki (s4) with --sleep 10 # T275246

Mentioned in SAL (#wikimedia-operations) [2023-01-26T22:23:57Z] <zabe> running migrateRevisionCommentTemp.php in cebwiki in screen with --sleep 2 # T275246

Mentioned in SAL (#wikimedia-operations) [2023-01-31T20:45:52Z] <zabe> start running "foreachwikiindblist s5.dblist migrateRevisionCommentTemp.php --sleep 2" in screen # T275246

Mentioned in SAL (#wikimedia-operations) [2023-02-01T10:42:11Z] <zabe> start running migrateRevisionCommentTemp in remaining sections (for now except s3) in screens # T275246

Change 888093 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Start reading from rev_comment_id in cebwiki

https://gerrit.wikimedia.org/r/888093

Change 888093 merged by jenkins-bot:

[operations/mediawiki-config@master] Start reading from rev_comment_id in cebwiki

https://gerrit.wikimedia.org/r/888093

Mentioned in SAL (#wikimedia-operations) [2023-02-09T22:42:28Z] <zabe@deploy1002> Started scap: Backport for [[gerrit:888093|Start reading from rev_comment_id in cebwiki (T275246)]]

Mentioned in SAL (#wikimedia-operations) [2023-02-09T22:44:20Z] <zabe@deploy1002> zabe: Backport for [[gerrit:888093|Start reading from rev_comment_id in cebwiki (T275246)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-09T22:50:55Z] <zabe@deploy1002> Finished scap: Backport for [[gerrit:888093|Start reading from rev_comment_id in cebwiki (T275246)]] (duration: 08m 26s)

The deploy generated a large spike in dropped logs due to mapping errors:

mapper_parsing_exception: failed to parse field [entityId] of type [text]

Preview of field's value: '{Wikibase\\DataModel\\Entity\\ItemId=Q2482746}'
caused_by illegal_state_exception: Can't get text on a START_OBJECT at 1:622"

Seems currently limited to cebwiki.

Sigh, it's because cebwiki reads from wikidata revision table but using the new config while wikidata is not populated yet. That needs to be reverted. This edge case was mentioned before.

I should have mentioned this edge case but I forget since it was long time ago we did actor migration. My apologies.

Change 887859 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Revert "Start reading from rev_comment_id in cebwiki"

https://gerrit.wikimedia.org/r/887859

Change 887859 abandoned by Zabe:

[operations/mediawiki-config@master] Revert "Start reading from rev_comment_id in cebwiki"

Reason:

amir was faster

https://gerrit.wikimedia.org/r/887859

s5 is done

Mentioned in SAL (#wikimedia-operations) [2023-02-27T18:29:48Z] <zabe> start running "foreachwikiindblist s3.dblist migrateRevisionCommentTemp.php --sleep 2" in screen # T275246

In T275246#8650104, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2023-02-27T18:29:48Z] <zabe> start running "foreachwikiindblist s3.dblist migrateRevisionCommentTemp.php --sleep 2" in screen # T275246

--sleep 1 actually

Mentioned in SAL (#wikimedia-operations) [2023-03-01T20:43:12Z] <zabe> move rev_comment_id migration screens from mwmaint1002 to mwmaint2002 # T275246

s6 is done

s7 is done

s2 is done

I suggest running a query on every wiki to make sure there is no empty rev_comment_id left. I tried running:

ladsgroup@mwmaint1002:~$ foreachwikiindblist all mysql.php -- -e "select count(*) from revision where rev_comment_id = 0;"

But that got super slow even in medium size wikis, don't want to run it on wikidata, specially on production.

Change 923545 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/puppet@production] maintain-views: Drop views on revision_comment_temp

https://gerrit.wikimedia.org/r/923545

Change 923545 merged by Ladsgroup:

[operations/puppet@production] maintain-views: Drop views on revision_comment_temp

https://gerrit.wikimedia.org/r/923545

In T275246#8882399, @Ladsgroup wrote:

I suggest running a query on every wiki to make sure there is no empty rev_comment_id left. I tried running:

ladsgroup@mwmaint1002:~$ foreachwikiindblist all mysql.php -- -e "select count(*) from revision where rev_comment_id = 0;"

But that got super slow even in medium size wikis, don't want to run it on wikidata, specially on production.

@Ladsgroup Sorry to hijack this thread, but our local wiki instance somehow ended up with all rev_comment_id equal to 0. Based on your comment, it looks like this should not be the case. Is there are migration script already or what is our solution here? (revert back to previous versions?)

I ended up here because all pages on our wiki are no longer reachable (Error: There is currently no text in this page.) Upon investigating our DB it turns out the data is all there and I can find correct revisions. But the log shows something weird:

mw                 | [rdbms] MediaWiki\Revision\RevisionStore::fetchRevisionRowFromConds [0.001s] db: SELECT  rev_id,rev_page,rev_timestamp,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,comment_rev_comment.comment_text AS `rev_comment_text`,comment_rev_comment.comment_data AS `rev_comment_data`,comment_rev_comment.comment_id AS `rev_comment_cid`,actor_rev_user.actor_user AS `rev_user`,actor_rev_user.actor_name AS `rev_user_text`,rev_actor,page_namespace,page_title,page_id,page_latest,page_is_redirect,page_len,user_name  FROM `revision` JOIN `comment` `comment_rev_comment` ON ((comment_rev_comment.comment_id = rev_comment_id)) JOIN `actor` `actor_rev_user` ON ((actor_rev_user.actor_id = rev_actor)) JOIN `page` ON ((page_id = rev_page)) LEFT JOIN `user` ON ((actor_rev_user.actor_user != 0) AND (user_id = actor_rev_user.actor_user))   WHERE rev_id = 7  LIMIT 1
mw                 | [objectcache] fetchOrRegenerate(global:revision-row-1.29:wiki_www21:4:7): miss, new value computed
mw                 | [wikipage] Page exists but has no (visible) revisions!

The query is joining the tables revision and comment over comment_id=rev_comment_id but since all our rev_comment_id are 0 and the comment table has no 0 as comment_id the query is always empty resulting in [wikipage] Page exists but has no (visible) revisions!. I am not 100% sure this is why none of our pages is reachable but it sure looks like the culprit.

I am not really sure how we ended up in this situation either and I guess the issue is on our side and we probably just need to run a migration script. But I couldnt find which one. Is this still in the making? If this is not related, I can open another ticket.

@Andreg-p Does maintenance/migrateRevisionCommentTemp.php fix it?

@matmarex Unfortunately no. Here is part of the log, it says revision_comment_temp does not exist, so nothing to do.

[rdbms] Wikimedia\Rdbms\DatabaseMysqlBase::serverIsReadOnly [0.001s] db: SELECT @@GLOBAL.read_only AS Value
[objectcache] fetchOrRegenerate(global:rdbms-server-readonly:db): miss, new value computed
[rdbms] Wikimedia\Rdbms\DatabaseMysqlBase::fieldInfo [0s] db: SELECT * FROM `revision` LIMIT 1
[rdbms] MigrateRevisionCommentTemp::doDBUpdates [0.001s] db: SHOW TABLES LIKE 'revision\_comment\_temp'
revision_comment_temp does not exist, so nothing to do.
[rdbms] Wikimedia\Rdbms\LoadBalancer::reuseOrOpenConnectionForNewRef: reusing connection for 0/wiki_physikerwelt
[rdbms] LoggedUpdateMaintenance::execute [0s] db: INSERT IGNORE INTO `updatelog` (ul_key) VALUES ('MigrateRevisionCommentTemp')

That says the table is dropped so the data is gone, you can recover the table (revision_comment_temp) from a backup (always recommended before upgrade) , put it in the main database and run the script again.

In T275246#8993430, @Andreg-p wrote:

snip

Possibly T326071?

@Izno I thought so too at first but our rev_actor is not zeroed. That looks fine.

@Ladsgroup Not something I expected, a temporary table contains critical data that cannot be recovered without backup. I am actually not sure we ever had this table. I will try next week to take a closer look on older backups.