Cool! Moving to "next 2 weeks" since I will have to wait a week or so for the Jan 2023 mediawiki_history snapshot.

I think the neatest way to deal with the pageview data loss is to wait a few more days until the start of February. Then we can do a Jan 2023 snapshot which will look at data going back to Feb 2022. The data loss ended on 27 Jan, so this will avoid it entirely without the need for any special casing.

In T317171#8562493, @SBisson wrote:

@nshahquinn-wmf are you suggesting that we completely replace wppw1 and wppw1t with wppw2 and wppw2t everywhere or we just use wppw2 for that special case where we alter the url of the link on the page?

In T327221#8559665, @nshahquinn-wmf wrote:

Hmm, no, on reflection, I don't think it's worth making these improvements now. I'm planning to just update it as-is.

@Mayakp.wiki oh, we already have a task for that: T324135. I think that's useful since the source and potential responses to the warnings are different.

Just updated the canonical wiki dataset, which added 4 new wikis.

In T327221#8555613, @nshahquinn-wmf wrote:

Some improvements I could potentially make in this round:

• Fix the content page count to be based on AQS or mediawiki_history so it's actually the value at the snapshot time rather than at query time.
• Add external referrer pageviews proportion
• Add Global South traffic percentage
• Add monthly new content pages
• Add whether the project uses language variants

@SBisson I'm confident it doesn't need approval. It's just a minor tweak to our instrumentation and doesn't change the scope of our data collection.

Some improvements I could potentially make in this round:

• Fix the content page count to be based on AQS or mediawiki_history so it's actually the value at the snapshot time rather than at query time.
• Add external referrer pageviews proportion
• Add Global South traffic percentage
• Add whether the project uses language variants

The PR to review is here: https://github.com/wikimedia-research/canonical-data/pull/3.

This review led me to focus on the idea of detecting interpersonal conflict on-wiki by looking at signals such as mutual reverts. If successful, that could lead to applications like:

• quantifying the incidence of on-wiki conflict for use in high-level metrics and comparison with the number of conflict reports
• early detection and automated alerting of user conflict.

This is stalled and it's not clear when we'll be able to finish it, let alone who will do it at that point.

This shouldn't be assigned to me; I've never had a concrete plan to work on it.

Apart from being blocked on T316049, Wikistories is still in its infancy, so we should avoid making a major investment like an ETL job. For now, I am manually running and sharing metrics using the Wikistories dashboard.

I've consolidated my ad-hoc reporting into a spreadsheet dashboard [WMF only]. I think that's enough to count as an initial dashboard!

Drafting ongoing in this Google doc [WMF only].

@ntsako, @JAnstee_WMF: here's an offhand thought, in case it's not too late.

I've filed T325611: Add TikTok's in-app browser to ua-parser library and consolidated a bunch of the information we have about referrers (the large majority of it from @Isaac! 😄) onto Research:Referrer on Meta-Wiki.

The results are available in this Google doc (WMF only). Some of the results are publicly available in my research notebook on GitHub.

In T324126#8432904, @xcollazo wrote:

Adding steps for a non-admin user to verify that they do not see the 'Admin' tab:

1. Connect to JupyterHub in the usual way, say: ssh -N stat1007.eqiad.wmnet -L 8880:127.0.0.1:8880
2. On a browser, go to http://localhost:8880/hub/home
3. Confirm whether the 'Admin' tab is there or not

I still kind of like this idea, but it would be significant amount of work for a pretty marginal benefit.

The simpler base environment is definitely real now, and in any case I've created a lot of new stacked environments in the past several months without encountering this issue.

In a conda-analytics environment, pip install -e . works just fine, so there's no need for an install script.

In T321960#8432515, @BTullis wrote:

Here is the same query after the configuration change has been deployed.

presto> SELECT contribution_attempt_id, COUNT(*) AS frequency FROM event.mediawiki_wikistories_contribution_event WHERE year = 2022 AND ( month < 10 OR month = 10 AND day < 17 ) GROUP BY contribution_attempt_id; 
 contribution_attempt_id | frequency 
-------------------------+-----------
 NULL                    |      2205 
(1 row)

Updated the description to note:

In addition, analytics-mysql is not available on an-test-client1001, which complicates the process of testing Wmfdata.

For the most part, the dependency doesn't matter.

Thanks to T273210, Wmfdata now has the ability to recreate Spark sessions in the same notebook, which should give it the ability to easily recover from a crashed Spark session.

Cool, thank you @xcollazo! 🎉

Okay, I've merged the documentation improvements and version 2.0.0 changes to main and sent a pre-announcement to several Slack channels and analytics-announce@lists.wikimedia.org.

Merged in PR40.

In T321088#8411016, @xcollazo wrote:

I could add the following for you on the global condarc:

# With strict channel priority, packages in lower priority channels are not considered
# if a package with the same name appears in a higher priority channel.
channel_priority: strict

channels:
  - conda-forge
  - defaults

@xcollazo a month ago, I suggested changing the default source of Conda packages in conda-analytics. Let me re-up this here so you can consider doing this before the migration. For context, I think this would be a minor improvement, so it's fine to ignore if you think it's not worth the effort.

The pull request has been merged!

The pull request has been merged!

The removals have been merged. This will stay open until we actually release version 2.0, likely late this week or early next.

I've verified that import pyspark just works in the new conda-analytics environment. Coincidentally, my changes for T273210 have ended up making PySpark available as wmfdata.spark.pyspark. So this is doubly solved.

Soon, we are going to be moving from anaconda-wmf to conda-analytics as the base for new Conda environments (T321088). That will contain Wmfdata-Python 2.0, so we can skip directly to that.

@xcollazo's code has been merged, so I think this is done. Work continues on T300442: Release Wmfdata-Python 2.0.