@Htriedman I created an MR to add the DDL Scripts to your wmepageview repo. Please review. I have added it to your repo and afterwards we can update the production repository.

@Htriedman, The database is created and now I want to create the tables. I have prepared the create table statement. I just need you to confirm the column comments before i create them. See patch

To point (1) — in WME datasets, the page_id is referred to as identifier (inside a JSON object). Because this dataset is going to be used in a WME context, I plan to stick with that convention.

In that case should we rename the page_id column in the hourly table pageview_hourly_proportion to identifier for consistency?

Thanks @Htriedman for all your response. I'm about creating the tables in the newly created database. I was wondering, Do you have a folder for create table statements? I can't seem to find any in your repos.

@Htriedman, Regarding the request to move the code to a non-user repo, I'm a little confused, It seems it has already been moved because I can see a similar repo in WME directory here. Can I say that this task:

Regarding the schema @Htriedman , I took alook at the hourly job output table schemas, and I noticed the following:

1. The table htriedman.pageview_combined_analytics has a column identifier which is the page_id. This column should be named page_id to be consistent with other tables.
2. I noticed namespace_id is on neither of the tables. I would suggest it is added as it is part of a page metadata.

Hi @Htriedman . Can you help me confirm that these are all the list of output tables that need to be moved to a non-user database?
Hourly updated tables:

• htriedman.pageview_hourly_proportion
• htriedman.pageview_combined_analytics

Daily updated tables:

• htriedman.pageview_geo_distribution
• htriedman.pageview_geo_top10

Monthly updated tables:

• htriedman.pageview_associated_distribution
• htriedman.pageview_associated_top10

Thank you @mmartorana

@mmartorana Oh Please I'd like my access back if its not too late

@xcollazo . I have linked it.

Yes, for clarity, Here is the approach decided on to define our thresholds:

• Use Fix thresholds and revist them after a period of time (1 year suggested)
• Use quantiles to define thresholds.
• Current Quantiles values are gotten from using data between March 20th to Oct 15 2025.

I'm using this ticket as an opportunity to perform the following upgrades on the airflow dag:

@Hghani I think for thresholds, the question is should we use a fixed threshold or rolling threshold? I believe they both has advantages and disadvantages. Traffic changes over time. I suggested we use a mix of both.

Do the quantile values capture the spikes before May 28th? I ask because May 28th until the first week of June was the absolute peak of the May incident.

I applied the suggested thresholds above to the old incident data found in wmf.pageview_hourly_backup_2025 but unfortunately they didn't catch the changes in human-bot ratio. I think its either because the old table only has data from the time 2025-03-20 which this time the incident already started. If we had an older history data without the incident, it would help to create a more trustworthy baseline to calculate our diff.

A quick summary of 5 weeks data:
History table used: Pageview hourly
Proposed Monitory/Alerting frequency : Daily

Curent plan:

1. Understand the variability of pageviews by monitoring the delta change of total pageviews in a day against a baseline over a period of time.
2. Suggested baseline is the a floating average of pageviews over a period of 7 days.
3. We monitor this delta for a month maybe and from this pick a threshold for users and bots pageviews

@AndrewTavis_WMDE This is good news. Thank you so much for the effort put into this.

Okay @Andrew. Thank you!

To solve the missing wikis issue, we decided it's best to automate sqoop list. There are 3 source of truth in consideration:

1. Canonical_data.wikis table (from Wikimedia NOC website). Note there is ongoing work to automate this table T339928
2. Site_creation log website.
3. Project_namespace_map table.

Hi @AndrewTavis_WMDE . Thank you for the confirmation. Please feel free to reach out if you need any form of support.

@AndrewTavis_WMDE can we work with the date 28th March to finally disable the wikidata metric job in airflow?

The api metrics had been disabled. The wikidata metrics is pending,

@AndrewTavis_WMDE The plan is to go read-only (Graphite) by the end of Q3-FY24/25. We can hold off turning off the wikidata_metrics_to_graphite_daily_dag.py until towards the end of this quater.

Hi @AndrewTavis_WMDE, I'm just following up the wikidata_metrics_to_graphite_daily_dag.py. As part of T372855, Can we go ahead to turn this dag off?

There aren't any hive query or script I found using these tables. The dumps are currently used by Research and Platform Engineering teams.

I did some search on the repositories and have pulled up this analysis. It's still WIP in progress as I'm yet to get all tables/datasets that uses the mediawiki wikitext dumps.

As requested by @jwang and @mpopov, September's data for cu_log is now available in the data lake at the wmf_raw.mediawiki_private_cu_log table. The cu_log table has been added to the list of tables to be sqooped monthly so expect to get monthly data as other tables.

The switchover has been done. The gerrit repositories are deprecated(set to read-only) and the schema servers have all been updated with the gitlab urls with @BTullis support.

We plan to do the switch in 1 week time i.e 8th October, 2024.
Data-Platform-SRE We would need your support to manage merging this patch next Tuesday, on 8th October. We need to make sure the existent checkout have its git origin changed. Please help confirm availability so I can proceed with notifying everyone of this date.

Plan for EventPlatform Schema Migration.

The following documents have been updated:

• Event Platform/Schemas/Guidelines
• Event Platform/EventLogging legacy
• Metrics Platform/How to/Create a Custom Schema
• Event*
• Event Platform/Instrumentation How To
• Metrics Platform/Event Schema
• /Data Platform/Data Lake/Events
• Event Platform/Onboarding
• Event Platform/EventGate
• Gerrit/Events ingestion