Once this is installed on the servers, will it automatically take effect within users' environments? Or will users have to do something like restart their Jupyter servers or create new Conda environments?

In T275233#7629084, @Milimetric wrote:

This is sort of blocked on T292699: Conda's CPPFLAGS may not be correct when pip installing a package that needs c/cpp compilation

This was declined rather than resolved.

@Isaac Hmm...I was thinking it belong somewhere in the Research namespace on Meta, since it might be more interesting to researchers than to infrastructure developers. If you go with Meta, I think you'd need to create a new page (Research:Referrers?).

I spent a significant amount of time developing a theory of change diagram for Wikistories. I got quite a lot of positive feedback, but in the end it didn't seem to be useful for actually making decisions, especially considering the amount of effort it required. We are not planning to pursue it any further.

Thanks, @Isaac! That's very helpful context. It would be great if you put that information on a wiki page somewhere since in general I think we don't understand that much about our referrer data.

I've also updated the Superset dashboard with a graph of weekly previews by access method, accompanied by a note explaining this issue and when it was resolved.

I've checked that the update produces correct data (see this notebook on GitHub for details) and deployed it with a start date of 4 Jan.

Since the release, our webrequest data does show some requests with the new wppw1t parameter coming from third-party sites:

time           wppw1t_requests
2022-01-04     9
2022-01-05     2
2022-01-06     3
2022-01-08    14
2022-01-09     2

I've confirmed using the app simulator that recommendations are no longer being shown.

I've done my work here. I've turned off the experiment on the config page, so the clients should no longer be displaying any recommendations, and shut down the job that produces those recommendations.

But this also means that a user could potentially get a session, save the handle in a variable, begin using it, use run, go back to using the session directly without calling get_session again, and then unexpectedly have their session die in the middle of using it because of the timeout set by run.

My latest thinking is that we should just have a single create_session function which can also be used to recreate the session using new settings (by passing, say, recreate=True). Calling create_session when a session is already running without passing recreate would trigger a warning.

I happened to ask @Mayakp.wiki what new features she would find useful in Wmfdata-Python. One thing she mentioned was a way to get syntax highlighting on queries to be run with Wmfdata-Python.

@jwang @ovasileva Andrew is the expert, but I believe this is happening because dt has switched to meaning the time according to the client. So if someone, say, has the time on their phone set to 2008 or 2022, dt would reflect that. On the other hand, meta.dt (which is used to set the partition fields) is the time our server received the event, which would still be 2021.

I've updated all the metrics on the dashboard to weekly granularity.

I've put up a draft pull request on GitHub. I still need to make some tweaks, so I haven't requested review yet.

Okay, @Milimetric tested and approved, so I've gone ahead and merged it. This is done!

@Arrbee could you approve this access for Mary?

Mary still needs to sign L3; that should happen shortly.

In T297172#7567161, @Krinkle wrote:

In short: It is an HTTP response header, and most attributes like title, special, wprov, etc cannot be set client-side. There is an exemption specifically for "preview=1" (unused, appears ineffective) and "pageview=1" (in use).

I triggered another series of requests myself, and I was able to find them in webrequest without a problem. I'm now comfortable saying this is consistently working, so this will be resolved as soon as one of Inuka's engineers releases the code.

@jsn.sherman sure! I believe EventLogging works differently on the Beta Cluster, so I'll have to figure out how before I can check the data. I'll do that today.

I wasn't able to find those requests, but I was finally able to find a duplicate set of requests that Stephane made on Friday. Because of how weird this has been, I'm planning to repeat the test with another engineer this week just to confirm that everything is working before we release this new code.

Data Engineering folks, this ticket needs some input from you 😊

I just tried broadening my search to all requests for the summary of the ivory page during those two hours (with or without a wprov parameter). I still don't see anything that looks like you. Maybe your browser had cached those requests?

@SBisson I don't see any requests with that referer, those articles, or the new parameter when I search from 18:00 to 20:00 UTC today (13:00 to 15:00 EST). I do see 46 other requests with the old parameter, though. I'm not sure what's going on.

In T247261#7556444, @Milimetric wrote:

I had, thanks, the regex looks good. I'm not able to test for merging this week, so I hope someone else can get to it. If not, I'll merge next week.

@SBisson I can't see the data for that hour yet since there's a delay of a couple hours in refining it. I should be able to do the check after another hour.

@SBisson thanks for this! The pull request looks good to me.

In T269625#7544510, @SLien_WMF wrote:

@SNowick_WMF Hi again Shay! As we update our metrics in comms, I wanted to ask about the use of the registered editors metric versus anonymous plus registered editors - is there one metric over the other you recommend for the stats we use regularly in public comms? We've historically used registered editors in our public comms stats, but I wanted to better understand the reasoning here. Thanks!

@Milimetric in case you missed it, I made some changes to the PR in response to your comment. Let me know what you think!

In T297172#7553721, @SBisson wrote:

@nshahquinn-wmf when I add the header, strict-origin-when-cross-origin kicks in and the browser blocks the requests. Do you know anything about that?

I've resolved this on the main branch; it will be released with the next batch of fixes.

Okay, I've reworked the pull request and solved the outstanding issues (particularly the database permissions issue).

Thanks, @sbassett!

I already have a open pull request for this in the wmfdata-python repo (although it hasn't been touched in about 10 months). I'll work on getting that finished in the other ticket!

Not really related to analytics.wikimedia.org 😊

Very cool! What's the process for making resolved security issues like this public? Is that something we should do now?

Thanks @jbond! I edited the section you wrote and included references to it in the appropriate places in the instructions. If you agree with the changes, I think we might be able finally close this task!