The root cause of the issue can be found in the slack thread.

Oh i like the /edits/v4/per_editor idea. That way clients know 6 parameters is the preferred way. But the question I have is the best place to implement backward compatibility.

Great! I would go ahead to drop it.

Also, thank you for the decision documentation!

My pleasure!

I think it might work, we'll need to make changes to the AQS service like register two routes for the 2 URL. We would still use 1 handler but update it to handle wiki_id.

For Routing, I would create a new endpoint rather than change the existing endpoint so there are breaks in existing application calling the existing API.

We decided to add wiki_id dimension to exisiting mediawiki_revision_history_reduced druid datastore.

We considered making mediawiki_history_reduced daily, by changing it's source data to wmf_mediawiki.mediawiki_history_incremental_v1 rather than wmf.mediawiki_history.
This looked attractive given that we get daily freshness since the incremental table is daily and we could get rid of the mediawiki_revision_history_reduced table.
However after reviewing we decided to not pursue this at this time due to the following problems:

I would happy with us dropping it if there are no uses for it now or in the future

In my patch I have followed this precedence for getting user_is_system value per row:

The min date for event_sanitized.mediawiki_page_change_v1 is 2024-12-30, thats more an enough coverage since the problem table, wmf_contributors.edit_per_editor_per_page_daily, starts from '2025-01-01' . We can use it to correct user_is_system. However because it's sanitized we revision.editor.user_central_id is NULL. We we can't key on user_central_id. Rev_id is available so we can key on it.

Per our policy for producing events for non-internal data products, we would not use Test Kitchen for the Thanks instrumentation, since this would be production data. Thanks events will instead be produced to Event Platform via EventBus to EventGate.

I used the wrong ticket number in my patch. here is the refinery patch for this ticket https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1342105

Sliding 30 day top pages per editor is now available

Sliding 30 day top pages per editor is now available

edits/v3/per_editor is backed by the mediawiki_history_reduced druid dataset. Druid should allow us to filter the sum by the project dimension.

Looks like the druid datasource is mediawiki_revision_history_reduced rather than mediawiki_history_reduced which means we can't filter by project dimension in druid since there the datasource has no wiki dimension at all.

The fix is merged, I'm currently rerunning pageview_per_editor_per_page_daily history data to clear duplicates. Afterwards I'll load the 2 dependent cassandra tables wmf_readership.pageviews_per_editor and wmf_readership.pageviews_top_pages_per_editor

Deleting this ticket because there is T437710: DE1.3.2: Implement "Sliding 30 Day Top-K Pages for Editor" which is addresses the same task.

@Ottomata if i use event.mediawiki_page_change_v1, We have 253,367 editors that are in the backfill covered in the events table and ~1.8M editors not covered in the events table. ,

I added an early-month grace period to snapshot selection. Any run before the 6th of the month use the already-available M-2 snapshot while runs from the 6th onward use the fresh M-1 snapshot. Dates, partitioning, and the SLA are unchanged. This would prevent the DAG from stalling,

Checked how much editors in the backfill period that's not covered in the events table:

SELECT COUNT(*) AS editors_in_history_not_in_event
FROM (
    SELECT DISTINCT event_user_central_id AS user_central_id
    FROM wmf.mediawiki_history
    WHERE snapshot = '2026-07'
      AND event_entity = 'revision' AND event_type = 'create'
      AND to_date(event_timestamp) >= DATE '2025-01-01'
      AND to_date(event_timestamp) <  DATE '2026-05-29'
      AND NOT event_user_is_temporary
      AND NOT event_user_is_anonymous
      AND event_user_id IS NOT NULL AND event_user_id > 0
      AND event_user_is_permanent = true
      AND event_user_central_id IS NOT NULL
) h
LEFT ANTI JOIN (
    SELECT DISTINCT revision.editor.user_central_id AS user_central_id
    FROM event.mediawiki_page_change_v1
    WHERE make_date(year, month, day) >= DATE '2026-05-29'
      AND revision.editor.user_central_id IS NOT NULL
) e
    ON h.user_central_id = e.user_central_id;

For the record, non-system users are defined by MediaWiki as "user is considered to exist as a non-system user if it can authenticate, or has an email set, or has a non-invalid token".

Is authenticate equivalent to being registered? Can we say can a permanent user is equivalent to a non-system user? (I was thinking if it's true we can just set user_is_system to false for backfill)

user_is_system value is copied verbatim from the source event table event.mediawiki_page_change_v1.

The root cause of the issue can be found in the slack thread.

Sure @Ahoelzl

Thanks for the suggestion, @RKemper. @Ottomata had also suggested using the internal API when I raised concerns about the rate limit and the speed of fetching page content from the public API. I’ve switched to using the internal API and will test it now and update.

QQ: are you using an internal endpoint or an external one?

oops... thats a mistake. It's 102days. 1 year was when i was working with 2.2pages/sec.

@Ottomata I was thinking, could bulk re-render to repopulate the stream. Is that something mediawiki team would be able to do for us?

@KCVelaga_WMF I just left a comment. Thanks a lot.

Currently getting the remaining pages via REST api for enwiki and WME SC wikis (cywiki,, dewiki, eswiki, frwiki, idwiki, itwiki, nlwiki, ptwiki)

Did the audit across analytics-refinery, dbt-jobs and airflow-dags. Apart from the mediacounts + mediarequest HQLs already handled by Joseph under {T431475: Data Lake - mediarequests - adapt HQL query to include counting mediarequests from webrequest_text}, there is no other HQL script that needs top be changed in analytics-refinery.

Interesting. Okay . it reduced for everyother wiki. Thanks for pointing that out. I believe we want redirects too.

spark-sql (default)> select wiki_id, count(page_id) from wmf_content.mediawiki_html_content_current_v0 where wiki_id in ("enwiki", "dewiki", "frwiki", "eswiki", "nlwiki", "itwiki", "ptwiki", "idwiki", "cywiki") group by wiki_id;
wiki_id	count(page_id)
enwiki	4291833
idwiki	690784
nlwiki	1907524
ptwiki	1036188
dewiki	2577282
frwiki	2172292
cywiki	174259
eswiki	1719846
itwiki	1561825

so actually based on the table I'm using as source of truth wmf_content.mediawiki_content_current_v1 ,for the count of all enwiki pages namespace 0, we have ~ which is more than ~8M I mentioned above.

The numbers look good to me. The table contains only page_ids not found in the event stream or in the event stream but content is null

@Htriedman thanks for the response. Closing the ticket.

Okay, we have data for enwiki now in the table wmf_content.mediawiki_html_content_current_v0 but iceberg doesn't seem to do well materializing a whole large nested struct as a single column. It doesn't successfully query the wide table. I get this error when I do a select * from ....:

java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow

I've been told to be mindful on the number of parallel downloads (IIRC 10 is the max enterprise folks told me to use but better to double-check with them when you start automating something).

Is this regardless of the type of account? and is this regardless if the size of the snapshot being downloaded

perhaps you could simply copy import_enterprise_dumps somewhere in the DE code-base and adapt it to your needs

Exactly my thoughts. So i already have a generic version of it. I started running it enwiki and some other wikis.

Hi, yes the search team has its own account wmf-search-platform

Is it a free tier account ? @dcausse

On a second thought we can move it to analytics as part of T305688

I did check that an access_token obtained is not invalidated if you request a new one so hopefully there should be no issues with concurrent jobs requesting an access token.

@dcausse looking at API (T414066). just wondering how you got access to Enterprise API. Did you get Enterprise team to create account? I'm wondering if i can use the same account for Snapshot API. I can see it's currently used for Structured Contents, can we pull full html snapshot with that account?

Thanks Isaac. It would be really helpful is we can use the Enterprise APi for commons wiki namespace 6 because that alone is around ~120M pages.

There's a total of 644877678 (~645M) pages(all wikis) to backfill and that's a lot. I dont think we can back fill all quickly with the API

Thanks a lot @Ottomata for your sharing your thoughts and insights. It's been helpful.

Why flatten?

• Simpler queries.
• we can merge on page_id. I don't think iceberg can merge on nested columns or do partition prunning on it.

Or I can simple promote only page_id to top level column

Okay @Ottomata I too a look at the stream mediawiki/page/rendering_content_change and I support modeling it as the stream. I see it captures any any re-rendering of a page. So I think we should keep the following fields:

I suppose from mediawiki_content_current_v1? In that case, you'd have revision_id and revision_dt.

unless you need more metadata about the revision from the REST API.

I'd like to get the revision_id and dt via /with_html

Will the schema be like mediawiki_content_current_v1 or more like a stripped down mediawiki_page_html_content_change_v1 (I'd prefer the latter, but it might take more bikeshedding?).

Approach update