This is definitely a good idea, but although it's not that much work, it's big enough that we should wait for proper prioritization before doing it.

These suggestions are moot (because anaconda-wmf environments are long gone), and any case I think ended up implementing most of them myself.

Last week, I:

• Prepared a second draft of the metrics funnel: conceptual diagram, detailed inventory
• Monthly metrics report
  • Prepared and released the February report
  • Developed and implemented a new publication workflow that prevents permissions errors and sync conflicts
  • Completely overhauled and updated the instructions

Sharing what I've heard from @GGoncalves-WMF and others working on this: we have devised a new bot detection rule that captures most or all of this spike. In addition to applying it to new data from the start of April, we have decided to retroactively apply it December–March data: T421735 (we can't apply it to November as we no longer have the source data for that period).

While this would be very useful for Movement-Insights, from our perspective it's not top priority (unlike, for example, T418032).

Last week, I:

• Drafted the trends brief
• Improved Google Search monitoring notebook so that it automatically fetches latest data from BigQuery
• Came up with proposed priorities in preparation for Monday prioritization meeting
• Started learning about Dbt

High priority because, once the refactor finishes, we will be blocked from adding new wikis to the dataset because the generation script will fail.

Moving over from the conversation on Slack:

This week, I:

• Did the weekly monitoring
• Explored and discussed infrastructure needs for this work (e.g. incremental mediawiki_history, general availability of Dbt)
• Worked on getting service account access to exported search console data in Google Cloud (via round after round of getting permissions and then finding that they weren’t correctly applied or were insufficient)

Folding this into T419304.

I don't see a reason to have a permanent tracking task for the wiki comparison tool; if there was enough work that we needed to tie it together, a tag would be a better choice. Free free to reopen if you disagree!

This week, I:

• Paired with Maya to do the weekly monitoring
• Clarified leadership requirements around distribution of the fortnightly trends brief and the continuation of the monthly metrics report
• Made visual improvements to the Google Search Console monitoring
• Set up a basic monitoring notebook for daily editor counts
• Made a plan for fetching Google Search Console data through an API rather than through tedious manual data exports

The script that updates the MediaWiki project-namespace map uses the SiteMatrix API.

@Bewfip the syntax for unidirectional and bidirectional conversion are documented on the advanced syntax sub-page. Is that what you're referring to?

Last week, I:

• Drafted the trends brief
• Discussed needs and concerns around monthly reporting with Maya and Sam
• Improved the Google Search Console data workflow and monitoring notebook
• Documented monitoring notebooks and walked Maya through them
• Helped Maya get dependencies for the monitoring notebooks installed, dealing with a bunch of stubborn environment issues in the process

Great work, @Diskdance! I think you've completed this 17-year-old task 🎉

In T416963#11609154, @brennen wrote:

Created: https://gitlab.wikimedia.org/admin/projects/repos/movement-insights/trend-monitoring

Last week, I:

• Updated and refined the traffic statistics to be shared with Google to illustrate the Big English fundraising problems
• Did weekly monitoring and noted points for the next brief
• Ran monthly metrics calculations and generated charts
• Backfilled a few emerging wiki contributor metrics that weren’t done when Wikidata and Commons were removed from the category

In T365387#11589734, @xcollazo wrote:

mediawiki_wikitext_history doesn't exist anymore. Should we close this?

Upgrading to Spark 3.5 should allow us to remove the version specs and pins for:

• Pandas (T370705, T370707)
• Numpy (T370710)
• PyArrow (since I believe that its pin is for the current Pandas version)

It seems pretty clear that we're not going to continue this, but I do want to close the loop properly with Fundraising, so I'm going to reopen to track that work.

Last week, I:

• Drafted and shared first “official” trends brief
• Responded to follow-up questions on brief
• Discussed plans with Sam NeSmith

In T21044#11573638, @Diskdance wrote:

For everyone's information, I've created https://www.mediawiki.org/wiki/User:Diskdance/Overview_of_Language_Converter as a general overview document of LC.

We discussed this at a team meeting today and decided there might be better ways to approach this, so we'll hold on this until we've had a chance to discuss with the full set of stakeholders in the process.

Last week, I:

• Continued discussions about reporting venue, audience, and frequency
• Worked through several different Superset issues
• Dug into new Comscore data and compared it with our unique device data
• Answered lots of data questions from Suman

Last week, I:

• Worked on the monitoring runbook and dashboard
• Discussed the trends catalog with Maya
• Dug back into data on the fundraising issues
• Produced updated charts of traffic from Google

In the WMF Slack, @Michael just asked about getting wiki creation dates for some newer wikis to understand why they're missing a particular table (T414600).

There's actually a wikiBirthday maintenance script that uses the "timestamp of minimum rev ID" method which I found to be the best option despite it being mostly wrong for UseModWiki-first wikis.

In T378253#10381138, @nshahquinn-wmf wrote:

@fkaelin has prepared an MR implementing this, so I should review it relatively soon.

In T403437#11421301, @nshahquinn-wmf wrote:

Last week, I:

• Investigated the correlation between referral traffic from Google and from other external referrers
• Finished slides for the board meeting presentation

Barring any last minute requests for changes to the slides, this work is done and the hypothesis will be closed shortly.

Last week, I:

• Investigated the correlation between referral traffic from Google and from other external referrers
• Finished slides for the board meeting presentation

In T403437#11399293, @nshahquinn-wmf wrote:

Last week, I:

• Wrapped up small wiki investigation and chose proposed example wikis
• Started work on slides for the board meeting presentation

Last week, I:

• Wrapped up small wiki investigation and chose proposed example wikis
• Started work on slides for the board meeting presentation

One additional point I've thought of: if you look in MediaWiki history and find that a group of revisions have the same exact author, timestamp, and page title, this is (almost?) certainly a group with one original and the remainder imported duplicates of that original.

I just worked on a Wikipedia 25-related request from the WMF Communications department for:

• The monthly article count for each Wikipedia during its history
• The creation data for each Wikipedia
• The first article created at each Wikipedia

In T278467#8892637, @xcollazo wrote:

Agreed that the Hive tables can stay as they are, and the new Iceberg tables can do proper DATEs and TIMESTAMPs. When inserting into Iceberg, we can cast accordingly. See T335305 for an example conversion from year, month, day INTs to a DATE.

@Niharika what kind of input are you looking for?

@JAllemandou absolutely, I think the rule improvements are in great shape! By specifying TLDs for each search engine, you have already gone well above and beyond the requirements 😊

In T406531#11335567, @JAllemandou wrote:

I have copied some values where referer has changed grouped by referer-domain and ordered by number of hists desc here (note: this list doesn't include IP-referer that have now been categorized as unknown instead of external).

In T403437#11308848, @nshahquinn-wmf wrote:

Since my last update, I:

• Produced lots more visuals and analysis
• I'm on track to have this analysis and visualization work largely completed by the end of the day Tue, 28 Oct (since I'll be on vacation Wed-Fri)

Since my last update, I:

• Produced lots more visuals and analysis
• I'm on track to have this analysis and visualization work largely completed by the end of the day Tue, 28 Oct (since I'll be on vacation Wed-Fri)

In T403437#11288192, @nshahquinn-wmf wrote:

Since my last update, I:

• Did a ton of analysis and data visualization in preparation for public communications about recent declines in pageviews
• Dug into trends in referrers
• Tested and found support for the the hypothesis that iOS traffic declined less than Android traffic (suggesting that pageviews from people with high socio-economic status declined less)

Since my last update, I:

• Did a ton of analysis and data visualization in preparation for public communications about recent declines in pageviews
• Dug into trends in referrers
• Tested and found support for the the hypothesis that iOS traffic declined less than Android traffic (suggesting that pageviews from people with high socio-economic status declined less)

Random suggestions:

• Consider including Kagi as a search engine
• Consider addressing T383088 by dropping the requirement that the referrer start with "http:// or "https://"

In T403437#11243456, @nshahquinn-wmf wrote:

This week, I:

• Analyzed Comscore and unique device data
  • Lots of contradictory signals, plus noise due to the ongoing traffic data backfill, which should finish by Tue, Oct 7.
• Analyzed Google-reported clickthrough and Google-referred pageview data

This week, I:

• Analyzed Comscore and unique device data
  • Lots of contradictory signals, plus noise due to the ongoing traffic data backfill, which should finish by Tue, Oct 7.
• Analyzed Google-reported clickthrough and Google-referred pageview data

In T405533#11235980, @JAllemandou wrote:

I'm very sorry for the misunderstanding here. I made a vocabulary mistake with the French false-friend "demander". I was really not meaning that you were forcing for the change, I'll be careful when using the word "demand" in future.

In T402324#11237257, @Mayakp.wiki wrote:

speaking for our needs, I think that's totally fine! :)