Page MenuHomePhabricator

Write a script to filter scraper output and compile dewiki articles with similar refs
Closed, ResolvedPublic

Description

In T367177: Compile a list of articles with "similar" references and hand-categorize use cases we left some notes on how to sample articles with "similar" refs. Integrate these commands into a script (Python / Elixir / Shell, your choice) which produces the data defined in the parent ticket: T401338: Create a list of dewiki articles with similar references.

To develop the script ahead of our next scraper run, test it on a file from older scraper outputs. Use a page-summary file such as dewiki-20240501-page-summary.ndjson.gz

Output formatted as CSV: spreadsheet

For review: Technical documentation

Event Timeline

We made a group review of the used code and its outcomes. Here are some notes and todos:

  • Wikipages with large number of total refs can be overwhelming.
  • The current lists of random low 100 is helpful.
  • We want to add two more random 300 lists with 1 - 5 and 10 - 30 similar refs.
  • We also want to use the info "ref_reuse_count" for our analysis and add it to the output.
  • To make it easier for users to also find the subreferences in the list we want to have at least 1/2 of the (total - reuse) references be similar.
  • We noticed that we need to add a round of manual checking.

I made some adjustments according to our review. Find the outcomes in the parent ticket. T401338