User Details
User Details
- User Since
- Oct 3 2025, 9:25 AM (53 w, 3 d)
- Availability
- Available
- LDAP User
- Unknown
- MediaWiki User
- APizzata-WMF [ Global Accounts ]
Thu, Oct 8
Thu, Oct 8
APizzata-WMF added a comment to T438054: page_change SLO windows.
Hey @RLazarus, thank you for the message and have fun on your PTO!
Tue, Oct 6
Tue, Oct 6
APizzata-WMF added a comment to T437961: Calculate active UWERs on a weekly cadence.
Merged all the base models and waiting for tomorrow sqoop and run to start the back fill.
In the mean time I have also created the final metrics cube that can be reviewed here: https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/198
Thu, Oct 1
Thu, Oct 1
APizzata-WMF added a comment to T437961: Calculate active UWERs on a weekly cadence.
Merged the HQL table creation scripts and the weekly sqoop jobs.
Would appreciate a review on last week's MR: base_moderator_actions_suppression_weekly.
Wed, Sep 30
Wed, Sep 30
APizzata-WMF added a comment to T437733: User and team Hive database cleanup tracking for Fall 2026.
Tue, Sep 22
Tue, Sep 22
APizzata-WMF added a comment to T438054: page_change SLO windows.
Hey @elukey, thanks for the reply.
Unfortunately we can compute these metrics only once a month, but I get your point. Please let me know if anything else comes up!
APizzata-WMF added a comment to T437961: Calculate active UWERs on a weekly cadence.
Created the following MRs:
- https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343954 to create the weekly actor sqoop
- https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1343956 to create the weekly actor table DDL
- https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2683 to create the weekly sqoop repair job
- https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/189 to create the dbt weekly model for base_moderator_actions_suppression
APizzata-WMF added a comment to T437961: Calculate active UWERs on a weekly cadence.
Here are the results:
- 4/32 took 2h.
- 2/16 took 4:30h.
- 4/16 took 2:10h.
APizzata-WMF placed T438054: page_change SLO windows up for grabs.
Mon, Sep 21
Mon, Sep 21
APizzata-WMF moved T436664: ST6.3.2 page_change.v1 SLO from In progress to Blocked/Paused on the Data-Engineering (Q1 FS26/27 July 1st - September 30th) board.
APizzata-WMF updated subscribers of T437961: Calculate active UWERs on a weekly cadence.
The full snapshot of the wmf_raw.mediawiki_actor table weights roughly 6 Gb:
hdfs dfs -du /wmf/data/raw/mediawiki_private/tables/actor \
| awk '{ n=split($NF,a,"/"); printf "%-24s %8.2f GB\n", a[n], $1/1024/1024/1024 }' \
| sort
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
snapshot=2026-03 6.65 GB
snapshot=2026-04 6.68 GB
snapshot=2026-05 6.72 GB
snapshot=2026-06 6.75 GB
snapshot=2026-07 6.78 GB
snapshot=2026-08 6.82 GBThe full ingestion actor table with the following config:
/usr/bin/python3 /srv/deployment/analytics/refinery/bin/sqoop-mediawiki-tables \
--job-name sqoop-mediawiki-actor-$(/bin/date --date="yesterday" +'%Y-%m-%d') \
--output-dir /path-to/wmf/data/raw/mediawiki_private/tables \
--wiki-file /mnt/hdfs/wmf/data/wmf/mediawiki/database/grouped_wikis.csv \
--tables actor \
--user research \
--password-file /path-to/mysql-analytics-research-client-pw.txt \
--partition-name day \
--partition-value $(/bin/date --date="yesterday" +'%Y-%m-%d') \
--mappers 32 \
--processors 4 \
--output-format avrodata \
--yarn-queue default \
--log-file /path-to/sqoop-mediawiki-actor-test.logtook 2 hours. Will test with a smaller parallelism (2/16) to see if we can save up some resources.
Thu, Sep 17
Thu, Sep 17
Wed, Sep 16
Wed, Sep 16
APizzata-WMF added a comment to T438054: page_change SLO windows.
Following further discussions with the team, we are designing an alternative approach for the Monthly Completeness SLO: on a monthly basis, all daily values will be uploaded to Prometheus to give better visibility into completeness trends throughout the month.
Tue, Sep 15
Tue, Sep 15
APizzata-WMF updated the task description for T437733: User and team Hive database cleanup tracking for Fall 2026.
Mon, Sep 14
Mon, Sep 14
APizzata-WMF added a comment to T437118: Backfill wmf_objective_metrics.heartbeat_active_editors_weekly.
Fixed!
select dt, count(*) from wmf_objective_metrics.heartbeat_active_editors_weekly where dt = '2026-06-01' group by 1 order by 1 desc ; dt count(1) 2026-06-01 663
Sep 11 2026
Sep 11 2026
APizzata-WMF added a comment to T437118: Backfill wmf_objective_metrics.heartbeat_active_editors_weekly.
weird, from my previous comment the data is there for the 2026-06-01, can you tell me what is this dashboard checking?
Sep 8 2026
Sep 8 2026
APizzata-WMF added a comment to T437118: Backfill wmf_objective_metrics.heartbeat_active_editors_weekly.
The table is backfilled:
spark.sql("""
select dt, count(*) from wmf_objective_metrics.heartbeat_active_editors_weekly group by 1 order by 1 desc """).show(20)Sep 7 2026
Sep 7 2026
APizzata-WMF updated the task description for T429770: migrate dbt_jobs pipelines to use MWH incremental.
Sep 4 2026
Sep 4 2026
APizzata-WMF added a comment to T425734: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel.
During the first phase of the analysis @JAllemandou and I noticed the following things:
Sep 1 2026
Sep 1 2026
Aug 28 2026
Aug 28 2026
Aug 27 2026
Aug 27 2026
APizzata-WMF updated the task description for T429770: migrate dbt_jobs pipelines to use MWH incremental.
APizzata-WMF updated the task description for T429770: migrate dbt_jobs pipelines to use MWH incremental.
Aug 26 2026
Aug 26 2026
APizzata-WMF updated the task description for T432017: Move all FY2026-2027 chart data to PostgreSQL.
APizzata-WMF added a comment to T432017: Move all FY2026-2027 chart data to PostgreSQL.
The job has been merged to Airflow and is running twice a day at 4 and 12 UTC.
Aug 21 2026
Aug 21 2026
APizzata-WMF updated the task description for T432017: Move all FY2026-2027 chart data to PostgreSQL.
Aug 20 2026
Aug 20 2026
APizzata-WMF updated the task description for T429770: migrate dbt_jobs pipelines to use MWH incremental.
Aug 19 2026
Aug 19 2026
APizzata-WMF reassigned T431723: MediaWiki Content File Exports are incomplete from APizzata-WMF to xcollazo.
APizzata-WMF closed T432208: Daily sqoops for logging and cu_log tables, a subtask of T429770: migrate dbt_jobs pipelines to use MWH incremental, as Resolved.
APizzata-WMF closed T428961: Review remaining 90-day windows in MWHistoryDeltaWriter, a subtask of T424350: Incremental MediaWiki History v1 - Phase I, as Resolved.
Aug 17 2026
Aug 17 2026
APizzata-WMF added a comment to T434576: [Spike] Evaluate incremental MWH for use in GB.
yes, you would select both the snapshots and the events
APizzata-WMF added a comment to T424358: Investigate oversized driver logs for `mediawiki_history_denormalize`.
Here are the result of the most recent execution:
sudo -u analytics kerberos-run-command analytics yarn logs -appOwner analytics -applicationId application_1778834239566_1568260 -out /tmp/app_mwh
APizzata-WMF added a comment to T434576: [Spike] Evaluate incremental MWH for use in GB.
Jul 31 2026
Jul 31 2026
Jul 30 2026
Jul 30 2026
APizzata-WMF added a comment to T433458: refinery-sqoop-mediawiki-production-daily fails for wikis that lack the discussiontools_subscription table.
Why do you say this is related to T432208?
If my help is needed I am glad to take a look at the issue, otherwise this can be assigned to someone else?
APizzata-WMF added a comment to T429770: migrate dbt_jobs pipelines to use MWH incremental.
The metrics:
- constructive_edits
- constructive_edit_rate
retained_editorsretained_newcomers- constructive_activations
- stg_bad_faith_edit_reverts
- base_moderator_actions_history
- base_moderator_actions_reverts
- base_moderator_actions_checkuser
- base_moderator_actions_html
- base_moderator_actions_html_historical
- heartbeat_active_editors_mart
APizzata-WMF added a comment to T424358: Investigate oversized driver logs for `mediawiki_history_denormalize`.
After testing the usage of the logback config file I obtained the following log:
Jul 28 2026
Jul 28 2026
APizzata-WMF updated subscribers of T424358: Investigate oversized driver logs for `mediawiki_history_denormalize`.
APizzata-WMF added a comment to T424358: Investigate oversized driver logs for `mediawiki_history_denormalize`.
the original run application application_1773845446826_249016 (the 2026-03 run) is unfortunately no longer available, I used the application_1778834239566_815909 which belongs to the 2026-97 run. For this execution the driver container log was ~8.2 GB and ~88% of all lines are DEBUG.
Jul 24 2026
Jul 24 2026
APizzata-WMF updated the task description for T429770: migrate dbt_jobs pipelines to use MWH incremental.
Jul 23 2026
Jul 23 2026
APizzata-WMF added a comment to T423952: Create mediawiki.user_change event stream.
Thank you all!
Jul 22 2026
Jul 22 2026
APizzata-WMF added a comment to T432208: Daily sqoops for logging and cu_log tables.
These are ingested daily in the stream table: event.mediawiki_revision_tags_change.
APizzata-WMF added a comment to T429770: migrate dbt_jobs pipelines to use MWH incremental.
For constructive_activations and retained_newcomers need to change the logic behind user_registrations.
I have the code ready but waiting on a reply on T423952#12119709 to know how to correctly handle user_is_created_by_self which is not always populated in wmf_mediawiki.mediawiki_history_incremental_v1.
APizzata-WMF added a comment to T432208: Daily sqoops for logging and cu_log tables.
change_tag and change_tag_def was also in scope
These are ingested daily in the stream table: event.mediawiki_revision_tags_change.
Jul 20 2026
Jul 20 2026
Jul 15 2026
Jul 15 2026
APizzata-WMF added a comment to T432208: Daily sqoops for logging and cu_log tables.
The cu_log sqoop succeeded with the configuration:
/usr/bin/python3 /srv/deployment/analytics/refinery/bin/sqoop-mediawiki-tables \
--job-name sqoop-mediawiki-daily-private-$(/bin/date --date="yesterday" +'%Y-%m-%d') \
--output-dir /path/to/raw/mediawiki_private/tables \
--wiki-file /mnt/hdfs/wmf/data/wmf/mediawiki/database/grouped_wikis.csv \
--tables cu_log \
--user research \
--password-file /path/to/mysql-analytics-research-client-pw.txt \
--from-timestamp $(/bin/date --date="yesterday" +'%Y%m%d000000') \
--to-timestamp $(/bin/date +'%Y%m%d000000') \
--partition-name day \
--partition-value $(/bin/date --date="yesterday" +'%Y-%m-%d') \
--mappers 4 \
--processors 10 \
--output-format avrodata \
--yarn-queue default \
--log-file sqoop-test/log/refinery/sqoop-mediawiki-private-daily.logAPizzata-WMF added a comment to T432208: Daily sqoops for logging and cu_log tables.
The logging sqoop succeeded with the configuration:
/usr/bin/python3 /srv/deployment/analytics/refinery/bin/sqoop-mediawiki-tables \
--job-name sqoop-mediawiki-loigging-$(/bin/date --date="yesterday" +'%Y-%m-%d') \
--clouddb \
--output-dir /path/to/raw/mediawiki/tables \
--wiki-file /mnt/hdfs/wmf/data/wmf/mediawiki/database/grouped_wikis.csv \
--tables logging \
--user s53272 \
--password-file /path/to/mysql-analytics-labsdb-client-pw.txt\
--from-timestamp $(/bin/date --date="yesterday" +'%Y%m%d000000') \
--to-timestamp $(/bin/date +'%Y%m%d000000') \
--partition-name day \
--partition-value $(/bin/date --date="yesterday" +'%Y-%m-%d') \
--mappers 64 \
--processors 10 \
--yarn-queue default \
--output-format avrodata \
--log-file sqoop-test/log/refinery/sqoop-mediawiki.logAPizzata-WMF updated subscribers of T432210: Daily sqoops for logging and cu_log tables.
Jul 9 2026
Jul 9 2026
Jul 8 2026
Jul 8 2026
APizzata-WMF added a comment to T428961: Review remaining 90-day windows in MWHistoryDeltaWriter.
More results collected today with @JAllemandou :
Jul 7 2026
Jul 7 2026
APizzata-WMF added a comment to T428961: Review remaining 90-day windows in MWHistoryDeltaWriter.
By checking the output of the table without both bounds ( apizzata.mediawiki_history_incremental_test_v2) together with @JAllemandou we found a nice example that encourages us to remove the bounds:
APizzata-WMF added a comment to T428961: Review remaining 90-day windows in MWHistoryDeltaWriter.
Here are the results without the bound 2:
+--------------------------------+-----------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+ |app_id |start_time |job_type|wall_clock_min|peak_concurrent_executors|lifetime_executors|peak_heap_GB|peak_offheap_MB|gc_min|task_min|shuffle_write_GB|mem_spill_GB|disk_spill_GB|num_tasks| +--------------------------------+-----------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+ |application_1778834239566_957090|2026-07-07 09:06:44.746|monthly |38.2 |82 |105 |11.86 |193.2 |23.8 |2648.3 |2651.79 |9245.95 |2015.25 |25862 | |application_1778834239566_957758|2026-07-07 10:04:13.618|daily |30.1 |128 |222 |15.82 |224.7 |131.4 |3404.6 |2944.34 |277.81 |49.91 |171900 | |application_1778834239566_958347|2026-07-07 10:52:05.734|daily |27.6 |128 |228 |16.18 |224.2 |163.9 |3755.4 |2895.96 |45.88 |7.33 |193274 | |application_1778834239566_958553|2026-07-07 11:20:37.664|daily |30.4 |128 |222 |16.13 |223.2 |153.4 |4159.0 |3246.07 |64.63 |10.12 |202362 | |application_1778834239566_959062|2026-07-07 11:52:06.664|daily |34.5 |128 |226 |16.64 |223.7 |129.7 |3005.3 |3170.26 |24.31 |3.51 |202864 | +--------------------------------+-----------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+
Jul 6 2026
Jul 6 2026
APizzata-WMF added a comment to T428961: Review remaining 90-day windows in MWHistoryDeltaWriter.
Here are the results without the bound 2:
+--------------------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+ |app_id |job_type|wall_clock_min|peak_concurrent_executors|lifetime_executors|peak_heap_GB|peak_offheap_MB|gc_min|task_min|shuffle_write_GB|mem_spill_GB|disk_spill_GB|num_tasks| +--------------------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+ |application_1778834239566_835276|daily |34.4 |128 |221 |16.4 |221.6 |146.5 |4414.1 |4306.9 |2.88 |0.42 |235690 | |application_1778834239566_836319|daily |35.5 |128 |227 |17.41 |229.8 |182.5 |5777.9 |4466.75 |2.88 |0.41 |245942 | |application_1778834239566_837244|daily |32.5 |128 |227 |17.06 |229.0 |207.2 |5955.4 |4011.04 |2.88 |0.42 |242584 | |application_1778834239566_838026|daily |33.3 |128 |222 |15.75 |214.3 |93.7 |2945.9 |4276.34 |3.13 |0.39 |240977 | +--------------------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+
Instead here are the results without both bounds:
+--------------------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+ |app_id |job_type|wall_clock_min|peak_concurrent_executors|lifetime_executors|peak_heap_GB|peak_offheap_MB|gc_min|task_min|shuffle_write_GB|mem_spill_GB|disk_spill_GB|num_tasks| +--------------------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+ |application_1778834239566_838942|daily |35.1 |128 |228 |16.84 |229.3 |147.9 |4228.6 |4321.91 |0.0 |0.0 |253300 | |application_1778834239566_839788|daily |34.3 |128 |226 |16.56 |228.2 |169.0 |5242.3 |4314.2 |3.0 |0.42 |237549 | |application_1778834239566_840992|daily |22.1 |80 |157 |16.5 |216.1 |106.8 |1965.1 |2014.44 |0.0 |0.0 |150752 | |application_1778834239566_844064|daily |33.4 |128 |222 |16.86 |218.7 |160.1 |4584.9 |4453.9 |0.0 |0.0 |245780 | +--------------------------------+--------+--------------+-------------------------+------------------+------------+---------------+------+--------+----------------+------------+-------------+---------+
Jul 3 2026
Jul 3 2026
APizzata-WMF added a comment to T429770: migrate dbt_jobs pipelines to use MWH incremental.
@Mayakp.wiki here are the results:
constructive_edits
Jul 2 2026
Jul 2 2026
APizzata-WMF added a comment to T429770: migrate dbt_jobs pipelines to use MWH incremental.
From a conversation with @Mayakp.wiki, good candidates to start the migration are:
- constructive_edits
- constructive_edit_rate
APizzata-WMF added a comment to T424355: Accelerate sqoop landing for MediaWiki History private tables.
In this month's run there were some hiccups (more in ).
Anyway I was able to quantify the time it took to sqoop the MWH tables:
- the sqoop job: refinery-sqoop-mediawiki-production-history started at 2026-07-01T03:33:09 and ended at 2026-07-01T14:13:57 lasting: 10h 40 m
- the sqoop job: refinery-sqoop-mediawiki-history started at 2026-07-01T09:53:56 and ended at 2026-07-02T00:07:00 lasting: 14h 14m