A first set of notebooks can be found here.

• The raw features notebook generates research_dev.edit_intent_raw: The ~10.9M interpretable enwiki edits enriched with revision wikitext, the parent wikitext, the parent diff, and the structured edit-types JSON. This is the shared input for every use case below, and it uses a broadcast revision_id key-set to prune the large content tables rather than shuffle them.
• A shared library module holds the config and helpers, and an exploratory analysis notebook compute general statistics about the labelled dataset

The service has been redeployed with this docker config. Does that resolve this issue?

For the example pages, that is interesting. Is that intentional to your knowledge? For example, I noticed that in 2020 some definitions for 'user' shifted, so perhaps some corrections could be applied retroactively? The redirects could explain some of the difference, but this seems a very big difference. My best guess is that it is because this page was deleted in 2020 to make place for a new page to be moved there. This will be a set of exceptions I'll have to account for.

The s1_admission_monthly.parquet (663mb) was computed with a pyspark job (notebook attached). Instead of the wmf.pageview_hourly (the time range is too large for a single job), it uses knowledge_gaps.pageviews_daily, a daily pre-aggregated + filtered dataset built for the knowledge gaps pipeline. One caveat: that dataset starts in 2017, not 2016. Schema: wiki_db, page_id, year, month, days_ge150, max_daily, first_clear_day, last_clear_day, one row per wiki/page/month which passed 150 views on any single day. Note wiki_db uses the enwiki-style database code rather than the en.wikipedia from pageview hourly.

Final update:

• Project sketch: framework, shared language, and taxonomy of streaming-friendly trending definitions.
• Data pipelines: repeatable PySpark notebooks that replay a set of trending algorithms and configurations against the same window of history.
• Web tool: workbench to visualize and compare trend detection results, with a curated gallery of cases and observations, and an annotation feature to share observations and collect labels. code.
• Project report: connects the original sketch to the workbench, walks through the tool's functionality, and proposes next steps.

Progress this week

• Met with Halley to walk through the exploratory UI and collect feedback.
• Incorporated the feedback by splitting the UI into two views:
  • Advanced: the previous UI in full. Surfaces a lot of information that can be overwhelming, geared toward working close to the algorithms (e.g. seeing how configuration changes affect results).
  • Simple: essential information only, focused on a curated set of examples that show stakeholders the differences between algorithms and the trade-offs each entails.
  • The curated examples are produced via an annotation mechanism in the advanced UI: users can annotate specific pages and events with lines, boxes, explanations, and labels.
• Refined the algorithm set: the "compare z-scores with two moving windows" approach is now split into two configurations — dpe_mwp (streaming MWP that is work in progress, can fire within 1 minute) and the original wme (batch, snapped to the hour with 90-minute ingestion delay).

Progress this week

Progress this week

• Implemented batch pipeline producing a 1-minute-granularity pageview timeseries dataset over a one-month window (builds on WE 4.3.4 traffic-patterns work)
• Implemented the batch moments comparison algorithm (based on WME project) across several configurations (10min vs 60min, 1hour vs 4days, etc.)
• Trend detection events stored in a shared Iceberg table with algorithm and configuration metadata, enabling cross-approach comparison
• Started notebooks for visualizing results and computing evaluation metrics in an algorithm agnostic way

Data extraction notebook to

Progress

• Wrote project sketch: taxonomy, architectures, evaluation plan, product requirements matrix
• Started implementation of offline comparison pipeline (builds on WE4.3.4 timeseries work)

Lifecycle stage

• Implementation. Late start in the quarter; design sketch complete, pipeline implementation underway.

Blockers and risks

• None

Thanks for the update, can we link to the phab for the security model?

The research-datasets repo is used for research to production code; the pipelines are run on the research airflow instance

I had a look and merged the branch.

This is a notebook that can be used to generate datasets for the causal analysis, with configurable output database, snapshots and time periods.

Final update:

• The hypothesis is supported. An SRE investigating suspicious traffic can now surface actionable signals in seconds via the similarity search UI, work that previously required hours of ad hoc analysis.
• What was accomplished:
  • Hypothesis report doc (review requested from Leila and Chris before we close the hypothesis on Asana).
  • Similarity search UI for SREs to explore similar IP or JA3N fingerprints based on traffic pattern analysis (user guide)
  • Signal processing pipeline over webrequest data, storing similarity search indices on Ceph S3, scheduled hourly via Airflow (technical documentation)
  • Webrequest timeseries dataset published on DataHub, reusable as input for related OKR work (datahub)
  • Similarity indices available for IP and JA3N for last 5 hours (1 min window size) and last 24 hours (5 min window size), configurable via Airflow variable. The vector-search instance is deployed on DSE cluster.
• Key lessons: There are valuable signals in our traffic data that we can leverage to protect our infrastructure — and there is more value still to be unlocked. By building a configurable signal processing pipeline, we open multiple pathways to contribute to the OKR: the search UI delivered this quarter, the timeseries dataset as a reusable input for other OKR work, and a foundation for detection (not just search — active discovery of bad actors such as IPs in a botnet).

There is a notebooks folder in research-datasets, you could put a notebook there too.

For the streaming pipeline (no CR yet, but working chart):

• processes webrequest kafka streams. There are different variants (e.g. for pageviews, ddos detection, etc), the variant I want to deploy is the prototype for streaming trend detection for pages.
• uses Ceph S3 for persistence (read & write from service), no persistence otherwise
• requires the maxmind dbs (T405509), currently uses a temporary pvc for that
• regarding sizing: greatly depends on the configuration ~8cpu/20GB of ram

For the vector-search service, which is the main priority (with CR)

• Deployment chart review in progress; addressing reviewer comments.
• Met with @CDanis ; he was able to use the tool and explore the data, and gave valuable feedback. Also aligned on remaining deliverables for this hypothesis and discussed potential next steps in Q4
• Scoped wrap-up deliverables:
  • change UI to make nearest neighbors results links to perform a new search showing the nn of the clicked result.
  • user guide documentation for the similarity search UI, supported similarity types, and what each is looking for in the data.
  • interactive python notebook for SRE to run the pipeline end-to-end, including spinning up a local vector-search instance.
  • Generate similarity indices for specific configurations for the Feb 20th DDoS incident for further manual evaluation by SRE, deployed on the production instance. (groupings: ip and ja3n, time granularity: 1min and 5min, time window, 1 hour, 3 hours, 1 day)

All open gitlab issues have been migrated to phabricator and closed. The issues can now be disabled on gitlab.

This has been completed, thank you Martin & Isaac.

• All gitlab issues have been migrated to the relevant column in the research-engineering board
• The gitlab issues have been closed with a link to the new phabricator tasks, there are no open issues left.

Comment from @MGerlach:
We could probably re-use the code to count the number of occurrences of abbreviations in the text of Wikipedia articles https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/blob/main/notebooks/AbbreviationFilter.ipynb

Comment from @AKhatun_WMF:

From @Isaac :
Notes as I explore this:

• If not careful, this will make the library much much slower. The trees are quite large if you fully expand sections into paragraphs with sentence children, each of which have word+punctuation+whitespace children.
• Maybe need to drop punctuation/whitespace here or keep it as the simpler counting approach.
• Probably should consider moving Text out of first pass of tree differ then to counteract the slowdown.

Update from @Isaac:
A* implemented in !2 (merged) and 33% speed-up on tests seen! This makes the "impossible" exploration moot as A* skips exploring those paths (only considers changes of the same type of node). I thought more about the more fine-grained pruning and am rejecting because I think it might make certain actions look "cheaper" than they actually are. One possibility though I guess is pruning but giving each node a "weight" that reflects how many subnodes it has. In theory this should prevent A* from thinking it's cheap to insert/remove these large subtrees.

• Added Wikimedia deployment pipeline and Blubber integration for the vector-search repo; production images are now published to the WMF Docker registry.
• Manually deployed vector-search service on DSE Kubernetes cluster, including the traffic pattern similarity search UI.
• For the 5-hour window around the Feb 20th incident, built similarity search tables for request count, response size, and time to first byte. The UI is available on port 8012 on deployment.eqiad.wmnet (requires SSH tunnel, ui: http://localhost:8012/traffic-patterns/neighbours).
• Investigated CAS authentication for the service — this is an even bigger stretch goal than anticipated; the authentication layer with the IdP would need to be handled by the service itself, on top of a security review and config changes from SRE.

Weekly update

Weekly update

• Merged code in research-datasets to build similarity-search indices on Ceph S3 and run large-scale evaluation with PySpark.
• Merged a vector search service code exposing similarity-search endpoints over Ceph-backed indices; currently running on a stat machine.
• Met with DPE SRE and confirmed a path to deploy the traffic pattern similarity search work. I’ll also write up Ceph S3 use cases & feedback to support upcoming security-model and Hadoop to Ceph/K8s migration design work.

For reference, there are also offline components to this:

• research.article_features table contains the same features for all historical revisions, updated via airflow dag daily
• research.article_quality_scores contains the quality score and the alternative "standard quality" (either 1 or 0) used for the knowledge gaps and movement insights reporting
• pipeline code in research-datasets.

I added a deadline for the end of the Q1, and will coordinate with Isaac&Martin.

The content diff pipeline uses diff-match-patch from google, see the pyspark code to use it here.

@brouberol has reverted the change for the research instance manually, and the failing tasks were restarted and completed successfully. This does not solve the issue, as merging a new change will overwrite the manual intervention.

Weekly update:

• Backfilled 1 month of webrequest times-series for the analysis. the data pipeline is an hourly job generating an incremental iceberg table (partitioned by day). The backfill job is configured to run daily batches (<1hr to run, generating <200GB of data, compression to webrequest ~4x)
• Started experimentation with configuration for fft analysis, length of window and timebucket. For the previous research poc I used 1 day of data in 5min buckets. There will be similarity search indices for different configurations. Initially similarity search will be an exploratory data tool; though we should work on evaluation methods (labelled data from the analysis Hamid&KC are doing for WE5.4.11 High-Confidence Signals for Automated DDoS Mitigation), or using spur data as label).

@fkaelin comments on whether we need additional internal checks for using VitePress.

Thanks. The needed data is salvaged - the directories can be removed.

After discussing with KC we can close this, we will capture related future work under a new task.

Updates:

• The previously unused common_utils module is revamped into a shared python project. A gitlab CI job can publish a versioned wheel to the package registry. Other projects in the ml-pipelines can depend on the published wheel, or via a local path for development. This allows projects to depend on different version of common_utils if needed. The shared code from research-datasets repo was moved with no changes, including the unit tests.
• The add-a-link project in ml-pipelines now depends on the new common_utils project. Previously that shared code was copied to the add-a-link project directly, the duplicate code was removed.
• There is a new development blubber variant that defines a base docker image for development. This development image can be used to run tests locally, or to start a local jupyter server for development. Documented in the readme.
• A development dependency to wmfing was added to facilitate development of spark based pipelines, with example usage for add-a-link documented here.
• After closer inspection, it didn't make sense to adopt the Command API from research-datasets for ml-pipelines. As ml-pipelines is a mono-repo with many independent sub-projects, the unified Command API (that automatically creates dataclasses for all "entry-point" methods) would require undesired dependencies. The added complexity is not worth the benefit.

Thank you and apologies, I neglected to specify my shell user name: fab ; could you rerun please?

I would be interested to see the code that creates the structured representation from the html of a page, can someone point me to the repository?

• After the initial experiment, we have not used gitlab issues for planning and project tracking. Using phabricator as single tool for this is perferable.
• Gitlab issue was convenient for external collaborators, and some research scientists continued to use gitlab issues to track code related tasks in projects that started as such collaborations.
• There seems to be a preference to disable gitlab issues for all projects; we will need some time to migrate the open tasks to a phabricator (we will likely create a single "Research-Gitlab" board with lanes for each project). I created T413791 to track this.

This html stream (or rather the events table that will be gobblined to the datalake) will be the first step towards a production html dataset in the datalake, followed by other challenges such as reconciliation and backfilling. Since the complexity of this stream is limited (almost identical to the page content change), having this initial building block in place is in my opinion a high priority from an essential work perspective. We have to get started somewhere, and once this stream is in place it will also be easier to scope and plan the other pieces that need to follow.

Great to hear. @GGoncalves-WMF your educated guess is indeed such. Here some additional context

It would be great if you pick this up @JMonton-WMF .

Resolving this - feedback and next steps will be discussed in T398071#11362486

The dataset has been updated with data from September and October 25, available in table fab.content_diff_edit_types_index. Resolving this.

Weekly updates

• started implementation of commons-utils as a project in ml-pipeline. Initial focus is on ci integration to publish a wheel, and adding it is a dependency to another project.

List of tables with a non-fully qualified uri location (leaving out the tables in users personal databases)

wmde.campaign_banner_impressions_quarter_hourly
wmde.tmp_mwh_wiki_editor_activity_flags_monthly
wmde.tmp_wdqs_normalized_queries_and_metadata
wmde.tmp_wdqs_query_segments
wmde.wd_action_api_metrics_monthly
wmde.wd_action_api_request_metadata_monthly
wmde.wd_article_placeholder_metrics_daily
wmde.wd_changes_preference_usage_by_wiki_monthly
wmde.wd_changes_preference_usage_distinct_monthly
wmde.wd_coeditors_by_wiki_monthly
wmde.wd_coeditors_distinct_monthly
wmde.wd_device_type_edits_monthly
wmde.wd_dump_metrics_monthly
wmde.wd_dump_request_metadata_monthly
wmde.wd_entity_schema_namespace_metrics_daily
wmde.wd_entity_usage_by_wiki_monthly
wmde.wd_entity_usage_distinct_monthly
wmde.wd_item_sitelink_segments_weekly
wmde.wd_query_segments_daily
wmde.wd_reliability_metrics_daily
wmde.wd_rest_api_metrics_monthly
wmde.wd_rest_api_request_metadata_monthly
wmde.wd_rollback_editors_monthly
wmde.wd_special_entity_data_metrics_daily
wmde.wd_special_entity_schema_text_metrics_daily
wmde.wdqs_metrics_daily
wmde.wdqs_metrics_monthly
wmde.wiki_editor_activity_levels_monthly
wmde.wiki_page_wd_entity_usage_monthly
wmde.wit_docs_pageview_metrics_monthly
wmde.wlb_commons_video_metrics_daily
wmde.wlb_commons_video_metrics_monthly
wmf_content.inconsistent_rows_of_mediawiki_content_history_v1
wmf_content.mediawiki_content_history_v1_old
wmf_contributors.commons_category_metrics_snapshot
wmf_contributors.commons_edits
wmf_contributors.commons_media_file_metrics_snapshot
wmf_contributors.commons_pageviews_per_category_monthly
wmf_contributors.commons_pageviews_per_media_file_monthly
wmf_contributors.editor_month
wmf_contributors.new_editor
wmf_data_ops.data_quality_alerts
wmf_data_ops.data_quality_metrics
wmf_experiments.experiment_results_v1
wmf_experiments.experiments_registry_v1
wmf_experiments.metrics_catalog_v1
wmf_product.automoderator_activity_snapshot_monthly
wmf_product.automoderator_config
wmf_product.automoderator_monitoring_snapshot_daily
wmf_product.automoderator_potential_vandalism_reverted
wmf_product.citation_needed_clickthroughs_daily
wmf_product.citation_needed_searches_daily
wmf_product.commons_deletions_monthly
wmf_product.commons_uploads_monthly
wmf_product.commons_uploadwizard_deletions_monthly
wmf_product.cx_abuse_filter_daily
wmf_product.cx_corpora
wmf_product.cx_deletion_stats_monthly
wmf_product.cx_draft_translations_daily
wmf_product.cx_exclude_users
wmf_product.cx_key_metrics_monthly
wmf_product.cx_mt_default_service_comparison_monthly
wmf_product.cx_mt_service_availability
wmf_product.cx_mt_service_usage_monthly
wmf_product.cx_published_translations_daily
wmf_product.cx_suggestions_menu_interactions_daily
wmf_product.cx_translations
wmf_product.cx_translators
wmf_product.moderation_flagged_revisions_pending_hourly
wmf_product.moderation_patrolled_recentchanges_daily
wmf_product.moderation_unpatrolled_recentchanges_daily
wmf_product.moderation_vandal_pageviews_monthly
wmf_product.trust_safety_admin_action_daily
wmf_product.trust_safety_admin_action_monthly
wmf_product.trust_safety_admin_monthly
wmf_product.trust_safety_admin_request_monthly
wmf_product.trust_safety_block_daily
wmf_product.trust_safety_block_monthly
wmf_product.trust_safety_new_admin_monthly
wmf_readership.unique_devices_per_domain_daily
wmf_readership.unique_devices_per_domain_monthly
wmf_readership.unique_devices_per_project_family_daily
wmf_readership.unique_devices_per_project_family_monthly
wmf_traffic.aqs_hourly
wmf_traffic.browser_general
wmf_traffic.interlanguage_navigation
wmf_traffic.referrer_daily
wmf_traffic.session_length

@EBernhardson Thank you for the background!

This is fixed.

The fix has been deployed, the job have been backfilled, and the schema updated with the following commands.

ALTER TABLE research.mediawiki_content_diff DROP COLUMN revision_text_sha1;
ALTER TABLE research.mediawiki_content_diff ADD COLUMN user_central_id BIGINT COMMENT 'Global cross-wiki user ID. See: https://
www.mediawiki.org/wiki/Manual:Central_ID' AFTER user_id;

Related question regarding flow of data, based on the comment from the thread you linked.

The wikitext -> html happens inside the mediawiki application using the default mediawiki parser. I'm not sure what exactly happens under the hood, i expect it's a full php parser that runs in-process but i haven't paid enough attention to exactly what they do. This is indeed quite expensive, we are running hundreds of pages a second through the parser. Part of the reason i suggest we could do this is because we already parse this flow of data. Even at this high rate, it still takes a long time to get through everything. We have a loop that re-renders everything even if not edited, but it works on 16 week cycles.

Weekly updates

• The suggested contributions are described in this doc
• Ongoing discussion regarding ML dev tooling (with notebook support). Prototype repo is wmfing, replacing the archived research-commons repo.

This work has concluded.