Over on this comment thread, we figured that the way we reconcile when a revision has both been deleted and restored may be innacurate, as the timing is important to know whether it is visible or not.
In this task we should revisa that code and change it if neccesary.
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| Scale up mw-content-history-reconcile-enrich temporarily for big reconcile. | Xcollazo | operations/deployment-charts | master | +1 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T366752 Dumps 2.0 Phase III: Production level dumps (SDS 1.2) | |||
| Resolved | APizzata-WMF | T412461 On reconcile, consider what happens when a restore and a delete happen on the same revision |
Yes indeed, in T416491 I have an example of this situation and a possible cure. We can either close this, or link the two together?
Cross posting then the findings of the other one here, and closing T416491
As stated in T410431#11534001, the page_id= 69510715 and wiki_id = 'enwiki' combination that is present in MariaDB, is completely absent in the wmf_content.mediwiki_content_history_v1.
After some digging the problem is most probably in the reconciliation query.
The absence of records in the wmf_content.mediwiki_content_history_v1 does not allow the query to select any record and therefore the pipeline does not produce any reconciliation.
Thanks, closed that as dup. Copying some context from that then.
Possibly related: we are missing some page_delete events since we switched to DomainEvents.
This RevisionAccessException was thrown 1286 times in the last month.
RevisionEntitySerializer - avoid RevisionAccessException on page delete (1191152) should fix. Is there a way we can tell if merging that has an impact on MWCH reconciliation issues?
Did some testing and can confirm the results:
Executed a full reconciliation on enwiki with both the new logic and the old logic and got the following numbers:
In the new logic results I was able to find the records for page_id = 69510715:
spark.sql("select count(*) from apizzata.inconsistent_rows_of_mediawiki_content_history_v1 where page_id=69510715").show()+--------+ |count(1)| +--------+ | 51| +--------+
Additionally in the difference between the new and the old logic I was able to find a nice case for page 290.
In the new logic we can find this number of records:
spark.sql("select count(*) from apizzata.inconsistent_rows_of_mediawiki_content_history_v1 where page_id=290 and computation_class = 'all-of-wiki-time'").show()+--------+ |count(1)| +--------+ | 5164| +--------+
And we can see that (at the time of writing, snpashot number 1085788035639961903) there are none of these in the wmf_content.mediawiki_content_history_v1 table:
spark.sql("""
select count(*)
from wmf_content.mediawiki_content_history_v1
where
page_id =290
and wiki_id='enwiki'
and revision_id in (
select revision_id
from apizzata.inconsistent_rows_of_mediawiki_content_history_v1
where page_id=290 and computation_class = 'all-of-wiki-time')
""").show()+--------+ |count(1)| +--------+ | 0| +--------+
While in the result of the old logic we have:
spark.sql("select count(*) from apizzata.inconsistent_rows_of_mediawiki_content_history_v2 where page_id=290 and computation_class = 'all-of-wiki-time'").show()+--------+ |count(1)| +--------+ | 16| +--------+
And we can see that these 16 are all present in the target table:
spark.sql("""
select count(*)
from wmf_content.mediawiki_content_history_v1
where
page_id =290
and wiki_id='enwiki'
and revision_id in (
select revision_id
from apizzata.inconsistent_rows_of_mediawiki_content_history_v2
where page_id=290 and computation_class = 'all-of-wiki-time')
""").show()+--------+ |count(1)| +--------+ | 16| +--------+
Notice the v2 extension which means 'results of the old version'.
Additionally, is implied that these 16 are not present in the results of the new logic, but here is the proof:
spark.sql("""
select count(*)
from apizzata.inconsistent_rows_of_mediawiki_content_history_v1
where page_id =290 and
revision_id in (
select
revision_id
from
apizzata.inconsistent_rows_of_mediawiki_content_history_v2
where page_id =290
)""").show()+--------+ |count(1)| +--------+ | 0| +--------+
All this being said because I feel that we can proceed with the implementation and after a chat with @xcollazo he also feels like this. So here's the MR.
Thanks to the backup @xcollazo created in T419055#11699345 we can see the results of the fix:
spark.sql("""
select
page_id,
computation_class,
reasons,
count(*)
from
wmf_content.inconsistent_rows_of_mediawiki_content_history_v1
where
page_id in( 69510715,290) and
wiki_id = 'enwiki' and
computation_class in ('all-of-wiki-time','all-of-wiki-time-first-run') and
computation_dt = '2026-03-01'
group by
page_id,
computation_class,
reasons
order by
page_id,
computation_class
""").show(truncate=False)+--------+--------------------------+----------------------------------------------------------------------------+--------+ |page_id |computation_class |reasons |count(1)| +--------+--------------------------+----------------------------------------------------------------------------+--------+ |290 |all-of-wiki-time |[missing_from_target, page_was_deleted, page_was_restored] |5164 | |290 |all-of-wiki-time-first-run|[missing_from_source, page_was_deleted, page_was_restored, mismatch_user_id]|23 | |69510715|all-of-wiki-time |[missing_from_target, page_was_deleted, page_was_restored] |51 | +--------+--------------------------+----------------------------------------------------------------------------+--------+
Everything looks as expected.
Shall wait next week once the reconciled records have been properly ingested to close the ticket.
Change #1253633 had a related patch set uploaded (by Xcollazo; author: Xcollazo):
[operations/deployment-charts@master] Scale up mw-content-history-reconcile-enrich temporarily for big reconcile.
Change #1253633 merged by jenkins-bot:
[operations/deployment-charts@master] Scale up mw-content-history-reconcile-enrich temporarily for big reconcile.
spark.sql("""
select
page_id,
'before reconciliation' as time,
count(*)
from
wmf_content.mediawiki_content_history_v1 timestamp as of '2026-03-12'
where
page_id in( 69510715,290) and
wiki_id = 'enwiki'
group by
1,2
union all
select
page_id,
'after reconciliation' as time,
count(*)
from
wmf_content.mediawiki_content_history_v1
where
page_id in( 69510715,290) and
wiki_id = 'enwiki'
group by
1,2
""").show(truncate=False)
+--------+---------------------+--------+
|page_id |time |count(1)|
+--------+---------------------+--------+
|290 |before reconciliation|23 |
|290 |after reconciliation |5190 |
|69510715|after reconciliation |51 |
+--------+---------------------+--------+The records have been ingested! closing the ticket