Planning- if airflow dag is used, estimation goes to an 8

Just a small status update before I am off for a long weekend: the base query is built, will just need to finalise the main metric calculation and then update / test the DAGs.

catherinekelsey opened https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/61

T417763_prep_staging_layers

Created a MR, for an initial internal review as well to check the logic. The linting check is failing, even though the errors shown do not exist... Bringing to our Analytics meeting on 28/07.

catherinekelsey merged https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/61

T417763_prep_staging_layers

Hi @Ifrahkhanyaree_WMDE & @ECohen_WMDE - I just wanted to give a quick update on this, as tomorrow I go on leave for a few days - will be back in on 14th August.

I had wanted to have merged the monthly DAG for this today, however, due to some updates to make the testing process run faster - I have some jobs failing (that have nothing to do with what I'm working on), but can't quite resolve them in order for the overall pipeline to pass.

I have booked a pair session, for the 14th, to go through this and resolve.

The full status:

1. Daily DAG / task for preparing the data: completed and merged :)
2. Monthly metrics DAG/task for us to then compute the metrics in the dashboard: in progress - the task / code is written and merged, it's as per above the issue with testing the DAG that is holding me up.
3. Dashboard build: will be completed once step 2 is resolved.

Please also be aware, I have combined the ticket T417760 also into this work, as the additional dimensions of location, device type and OS - made sense to incorporate.

In the meantime, you can take a look at some of the testing output for steps 1 and 2, which hopefully can give an idea of what we are building ahead of the dashboard.

Please let me know any questions in the meantime!

Hi @Ifrahkhanyaree_WMDE and @ECohen_WMDE - have created a dashboard for the monthly metrics as discussed. Please see here. It's quite dense, so please let me know if you would like anything removed, especially for tabs 1 &2.

The top charts on tabs 1&2, will show the trend and monthly % change. Please let me know if a monthly %change calculation would be valuable anywhere else? Am also happy to have a call / meet to go through, in case anything is unclear. Then once we are aligned - can add data for June / close the ticket.

Hi @catherine.kelsey.wmde the dashboard looks amazing, thank you for all the work you've put in! I'm waiting for Elisha to come back and get her input before we can resolve this ticket. Is that fine?

Hey @Ifrahkhanyaree_WMDE - sounds great!

Hi @catherine.kelsey.wmde - thanks for setting up a meeting to discuss. Overall it looks great and we're really happy to be able to see these numbers. Super helpful!

@Ifrahkhanyaree_WMDE and I collected our comments ahead of time and will put them here so you can review before we chat:

• Currently filtering by month breaks the dashboard
• Additional pages requested here are missing: T417762#12128571
• Having the individual referrers as filtering options isn’t ideal, it would be good to have that breakdown within the referrer tab. Note: we only need to see internal individual referrers.
• It looks like you’ve added redirects as separate pages - it is possible to add in the page titles that they are redirects? For example, "Wikidata:Reuse (redirect > Wikidata:Data access)"
• You’ve added one talk page (Wikidata_talk:For_developers/Getting_support), do we want to do this consistently? I would add all or none (for example, add Wikidata talk:Data access)

Thanks and talk soon!

Hey @ECohen_WMDE - thanks a lot for going through and these comments!

• Fixed - apologies, had the wrong value type parsing, but now works :)
• Added - apologies, I missed this comment from the previous ticket!
• Updates, have added 2 new charts on the referrer tab to now help you view this more easily
• I'm not sure what you mean by this, as this is likely just how our pages are defined... lets try and use a bit of our meeting today to discuss
• I think we are only seeing this because this is the only page with views, but I could be wrong. Let me know which other pages you require and can check them :)

@catherine.kelsey.wmde Looks great!! Thanks for already updating.

Yes, let's discuss the last two points in the meeting.

As discussed in meeting:

1. @ECohen_WMDE to provide example of the pages per wikidata access so we can then see if classigication is possible
2. @catherine.kelsey.wmde to exclude talk pages. Updated here

Hey @ECohen_WMDE - just a small reminder regarding the classifications :)

@catherine.kelsey.wmde - thanks for the reminder! Sorry, this slipped through.

Here are some example redirects:

• Wikidata:I
• Wikidata:INTRO
• Wikidata:HERE
• Wikidata:Purpose
• Wikidata:Welcome
• Wikidata:Wikidata

All redirect to > Wikidata:Introduction

Hey @ECohen_WMDE - thank you very much for this, have added a CASE condition on the dimension for the above. If you're happy with this, let us know and then close the task :)

@ECohen_WMDE, are the redirects listed above all of the ones that we need to include, or are there others? @catherine.kelsey.wmde, checking in as we may need to actually derive the redirects for the pages via the redirect field in certain page tables :)

Thanks @AndrewTavis_WMDE - The above list is just an example of some of the redirects so that the right field can be identified to then tag all the redirects included. It is not exhaustive.

@catherine.kelsey.wmde I'm not sure what adding the dimension did? I don't see them in the table anymore. They don't need to disappear; I just wanted to be able to easily differentiate between which are real pages and which are redirects. If possible, it would also be great to show which page it redirects to, then we could look at all of the data for the same page together. Let me know if that makes sense, happy to also have a quick chat about this

Hey @ECohen_WMDE - ah sorry, I misunderstood! I thought you wanted them classified to the introduction page (as this was where the re-direct goes).

Will take a look at the other table for how we can identify the re-directs. I think though, this would need to be a separate ticket, as if I need to update the code to read from another table, is a bit more involved than just updating in Superset. I've created a task here T436786 :)

I hope that's ok for you, but do let me know if not!

Hi @catherine.kelsey.wmde - all good! Thanks for making the follow up ticket. Didn't realize this required going into another table. I'm glad it's documented for the future but would say that this last step is not urgent/shouldn't be prioritized. We have the data we need for decision making and I can always manually check if a page is a redirect or not.