I would like to announce the new stewardship of this project as a followup to this email:
And what names would you like me to use for you?

Theoretically we can also answer most of these questions with Superset. However, Superset can be slow with webrequest and is prone to timeout. Something that could be helpful in the long term is T413952: [Request] Sample of refined webrequest table in data lake. It reduces dependency on fields being added to Turnilo each time.

1. Analytics support provided during Q4

1. Analytics support provided during Q3

All looks good to me. The maintenance DAGs have been running successfully. Thank you

@brennen hi! thanks for taking up the other request, wondering if you can create this one as well? Just wanted to check as it's been a few days, totally okay if not.

Thanks, all good now.

Hi @brennen, thanks for creating. I think I am not yet added to the repo, I am getting a 404 error when trying to access.

Just adding another usecase to have this data in Turnilo.

The request has been created for documentation, the request has already been completed by @JAllemandou.

I tried to copy over the query for that graph to Quarry, but I was getting an error. Despite the error, I have the impression that Quarry seems quite SQL-centric which may make it hard to serve usecases where communities may need an intuitive overview they can easily adjust without SQL or database knowledge. I'm less familiar with Quarry and maybe the error is preventing me from making a fair comparison. So maybe someone else may be able to illustrate how to support this better with Quarry.

In T416373#11665536, @Andrew wrote:

If it is shutting down, it would be good to understand. which will be the available alternatives to provide communities with that kind of support.

Quarry accesses the same datasets as the superset install, so any query that works in superset should be available there as well.

If you want to make a pitch for preserving superset, please detail the specific use case that's not supported on Quarry and which current users will be affected by the deprecation. Thanks!

My understanding is that this is not currently a high priority in WE5, so we have a little bit of freedom to schedule it in the coming weeks. Let me know if this is wrong, or it changes.

@JAllemandou @GGoncalves-WMF I put together a initial list on this spreadsheet.

@GGoncalves-WMF the updated task description captures all the details well, thank you!

Thanks @GGoncalves-WMF for creating this.

If this is high priority, let us know?

In T416373#11588290, @Samwalton9-WMF wrote:

Might it be possible to enable superset.wikimedia.org dashboards to be publicly viewable?

@bd808 I believe having a public instance is still useful for the usecases I mentioned above, even if that was not the originally intended goal. We can try Cloud VPS, but recreating dashboard, configuring DB etc. will be take some time, and not sure how much effort it would be.

@elukey didn't realize the newly labelled levels, nice! Gerard would want analytics-privatedata-users level 1 for now, as he needs to access Superset and also the datasets restricted to analytics-privatedata-users (no need for Kerberos, and SSH for now). Thanks.

Hi @taavi! While I understand the original intention of having a public Superset instance is to replace Quarry, which didn't happen. We have been using Quarry to develop and run queries against Wiki replicas, while Superset is really intended for visualization and dashboard. Unless someone has the need for latter, they don't have the need to go to Superset instance - which may have been the case with many users of Quarry.

@CDanis Please share if you have any other notes / details that would be useful from the SRE's approach. I think round-robin per CDN hostname across the stream would be better. Also, proposed a higher sampling rate as Spark should be able to handle it without much issues.

Marking this as completed as the first version is functional. Additional tasks can be filed for additions/improvements.

Seems like it coincides, the jobs were scheduled to run on 5th of every month.

I re-ran manually, everything is back up again.

Okay, I don't know what happened, I re-ran the scripts manually again, they ran fine. Not sure if this needs to be further investigated, or probably a one-off connection issue at the time of the monthly update.

@bd808 Any thoughts on this? As the errors seem similar to T189386 & T183644

Triaging this as high as it affecting at least two public facing dashboards.

The data is now updated and the charts are working.

@santhosh The Airflow pipeline is fixed, I will re-run the Toolforge update scripts.

I have updated the dashboard to include rowiki.

Thanks Miriam. I will update this task with more concrete support items we might need for Q2.

The analysis and the outputs have been shared with Halley.

cc @cchen @OSefu-WMF

Yes, this is not relevant. Reportupdater is no longer (I think). But all the pipelines are based on Airflow

@GGoncalves-WMF I have been able to work with the scripts Marcel and Jennifer developed for processes the logs - super helpful! A quick question, the logs at /srv/log/webrequest/archive/dumps.wikimedia.org/ and then processed. I am curious about what process publishes logs to that location - is it solely from webrequests or some other source?

I don't think it is possible with any of the available contextual attributes. I have to check with Exp Platform on this one.

Hi @Miriam! I can definitely say until end of Q2. At the moment, I don't know yet what needs might evolve for Q3 and beyond. If needed, I make a new request or update this one - will that work?

cc @OSefu-WMF

@MartinRulsch sorry, I completely missed your question, sorry for about that. In future, feel free to ping again or send an email if something hasn't been replied for long.

@GGoncalves-WMF thank you!

Hi all, as this analysis will inform a decision within the WE5 scope of work, so I can take this up (confirmed with @HCoplin-WMF on Slack).

Posting the A/A test and A/B test experiment flow here as well, and when a pagevisit event should be logged.

Tagging @cchen for QA.

For the current Synthetic A/A test, are we ONLY interested in tracking the page visit event, and not the entry points to visit MinT wiki readers (which would be handled in a later A/B test)

Do we interested if user leave the page and possible because of page loading time?
if yes then we need to log the "duration" as well when page unload.
Note: Currently, there is no "unload" log event

@hueitan thanks for pointing that. Please use fy25-26-we-3-1-5-mint-readers

The dashboard is functional at: https://superset.wmcloud.org/superset/dashboard/unified-cx-metrics/

no schema changes are required, just updating the ID is fine. 1.4.2 includes the experiment related fields, which will be used when the experiment is live.