In T324126#8432904, @xcollazo wrote:

Adding steps for a non-admin user to verify that they do not see the 'Admin' tab:

1. Connect to JupyterHub in the usual way, say: ssh -N stat1007.eqiad.wmnet -L 8880:127.0.0.1:8880
2. On a browser, go to http://localhost:8880/hub/home
3. Confirm whether the 'Admin' tab is there or not

I still kind of like this idea, but it would be significant amount of work for a pretty marginal benefit. It's a reasonable expectation that an analyst who makes a mistake and overwrites their source data with an incorrect transformation just has to rerun the query, even if takes a while.

The simpler base environment is definitely real now, and in any case I've created a lot of new stacked environments in the past several months without encountering this issue.

In a conda-analytics environment, pip install -e . works just fine, so there's no need for an install script.

In T321960#8432515, @BTullis wrote:

Here is the same query after the configuration change has been deployed.

presto> SELECT contribution_attempt_id, COUNT(*) AS frequency FROM event.mediawiki_wikistories_contribution_event WHERE year = 2022 AND ( month < 10 OR month = 10 AND day < 17 ) GROUP BY contribution_attempt_id;
contribution_attempt_id | frequency
-------------------------+-----------
NULL | 2205
(1 row)

Updated the description to note:

In addition, analytics-mysql is not available on an-test-client1001, which complicates the process of testing Wmfdata.

For the most part, the dependency doesn't matter.

Thanks to T273210, Wmfdata now has the ability to recreate Spark sessions in the same notebook, which should give it the ability to easily recover from a crashed Spark session.

Cool, thank you @xcollazo! 🎉

Okay, I've merged the documentation improvements and version 2.0.0 changes to main and sent a pre-announcement to several Slack channels and analytics-announce@lists.wikimedia.org.

Merged in PR40.

In T321088#8411016, @xcollazo wrote:

I could add the following for you on the global condarc:

# With strict channel priority, packages in lower priority channels are not considered
# if a package with the same name appears in a higher priority channel.
channel_priority: strict

channels:
  - conda-forge
  - defaults

@xcollazo a month ago, I suggested changing the default source of Conda packages in conda-analytics. Let me re-up this here so you can consider doing this before the migration. For context, I think this would be a minor improvement, so it's fine to ignore if you think it's not worth the effort.

The pull request has been merged!

The pull request has been merged!

The removals have been merged. This will stay open until we actually release version 2.0, likely late this week or early next.

I've verified that import pyspark just works in the new conda-analytics environment. Coincidentally, T273210 will end up making PySpark available as wmfdata.spark.pyspark. So this is doubly solved.

Soon, we are going to be moving from anaconda-wmf to conda-analytics as the base for new Conda environments (T321088). That will contain Wmfdata-Python 2.0, so we can skip directly to that.

@xcollazo's code has been merged, so I think this is done. Work continues on T300442: Release Wmfdata-Python 2.0.

Waiting for @xcollazo's review.

Currently up for review in PR36.

Waiting for @xcollazo's review.

This is currently up for review in PR36.

In T298179#8378525, @Antoine_Quhen wrote:

If the timeout is removed, it could be possible to detect and alert when not production yarn applications are running for more than a week, for safety.

The bulk of the removals up are for review in https://github.com/wikimedia/wmfdata-python/pull/35, although I just realized I still need to update the change log in that PR.

I will be doing all of this with the exception of T318587. It's a long list but it's all pretty simple.

Waiting on review by Data Engineering.

I need to add this new field to the sanitization allowlist.

I will handle retaining contribution_attempt_id in T317934, so at long last, this task is done!

I've merged the improved documentation, although the generated schemas have some weird formatting (like this). If it's not too much work, I should tidy that up.

In T321960#8356472, @Ottomata wrote:

If you select this data using Hive or Spark, it returns NULL for that column in old data.

@phuedx interested in your thoughts on this 😊

I spoke too soon: while I was updating the documentation, I thought of one more thing to check and ended up finding T322094.

Given that, I'm going to boldly close this.

I happen to be doing checks on the same data again and this problem is clearly fixed.

The example given in the description (https://en.wikipedia.org/api/rest_v1/page/html/9) now renders correctly:

In T312262#8322503, @nshahquinn-wmf wrote:

• According to the consumption schema, story_open_time is the cumulative amount of time the user has had any story open during the current page view. I suspect that it is instead limited to the story_view event type and contains the amount of time that story was open before being closed. I should figure out which is the case and, if my suspicions are correct, decide whether we should adjust the instrumentation or adjust the definition.

Just verified the fix for T318706: Story builder instrumentation has story_already_exists hardcoded to false.

Sorry for the delay! I've just checked the data we've received for this and there are a number of events where story_already_exists. I checked a random sample and the logged values were all correct. We can safely close this.

I checked the result of T317934: Add a edit attempt identifier to the Wikistories contributor data stream and noted T321960: Presto returns incorrect data for an added field.

Sorry for the delay!

This has been released in Wmfdata 1.4.0.

I collected the data in this document and shared it earlier today.

In T221482#8330593, @Milimetric wrote:

then just mark all the revisions that have much larger revision ids than their parent (via rev_parent_id as revision_is_probably_imported

In T319360#8332953, @xcollazo wrote:

https://github.com/wikimedia/wmfdata-python/pull/31 was merged, and a new release of wmfdata has been done: https://github.com/wikimedia/wmfdata-python/releases/tag/v1.4.0.