For the record, this data is now being automatically loaded into wmf_raw.google_search_console_{site|url}_impression (but note T434625). The data for day D is available in the Data Lake on either D+4 or D+5. The data is generally available in BigQuery on D+2, with the remaining delay being an effort to account for Google possibly revising the data.

Last week, I:

• Investigated recent Google Search trends
• Wrote the trends brief

Last week, I:

• Wrote the trends brief
• Finished and documented the impact of the contributors monthly metric backfill (T431647)

This seems like the main ticket for the work we did in 2020 to detect undeclared bot traffic ("automated") and remove it from our likely-human traffic bucket ("user").

The original goal of this ticket was to start detecting bots which did not self-identify in their user-agent header ("spider" in our bot data).

See decline of parent task: T359689#12176043

We will hopefully soon move the metric calculations to Dbt, and there's no major value in automatically generating the charts, because doing follow-up analysis and writing the report is always going to be manual.

Canonical wiki generation updated in commit 5f312023.

Report done in commit 93ddd3de.

Done while I was writing the June report.

The backfill is complete (commit fd3a7735).

After the WikimediaMessages patch is merged, I will then need to update the canonical wiki generation notebook to remove our manual addition of these names.

Thank you @KCVelaga_WMF for reviewing!

I had to do a cleanup commit after the MR because:

• somehow, while incorporating review suggestions or rebasing, I output a malformatted TSV
• the updated script output fields in a different order, and I forgot to update the fieldspec in the upload script to match

FWIW, I've confirmed that the new version of the script using the SiteMatrix address the problem of missing data for new wikis:

• bolwiki was created yesterday (T429921)
• The sites table does not have its data (because T429955 has not yet been done), so the old version of the script produces partial data
• The SiteMatrix API does have its data, so the new version of the script produces full data

Thank you for fixing those, @amastilovic!

Last week, I:

• Wrote the trends brief
• Worked with Aleksandr Mastilovic to finish backfilling base_account_registration and editor_month (T430602)
• Movement metrics query updates (T431647)
  • Improved the movement metrics SQL machinery to make backfilling easier
  • Rewrote queries
  • Spent a long time investigating why the changes caused 10-20% decreases in the calculated retained newcomers at Wikidata and Commons

For the record, the fixes were:

1. producing the month field in editor_month as a date rather than a datetime
2. dropping the previous editor_month before backfilling

The query changes in are in MR 41 and have been approved by @Mayakp.wiki.

@KCVelaga_WMF would you like to be the one to review the MR? I'm always happy to have you working on canonical data. But no worries if not!

Both MRs are now merged, so everything's in place for the backfill.