Resolving given the above :)

FYI @Arian_Bozorg and @Lydia_Pintscher: The decision was made to not include subclasses of entity schema classes in the process at this time as this was causing the query to explode in resource requirements. I tried to quadruple the normal resources that we give to queries and it still wasn't enough, and reworking it so that it could work will be a lot of time and will increase the complexity significantly. As this isn't being included in the WDQM at this time, I think it's best that we finalize this and move on with other priorities.

@Lina_Farid_WMDE: Note that the table has now been updated, and Superset as well. We're not getting nearly as many of the flows where the user doesn't add an automatic citation as we expected. It's around the same as failures. Likely a lot of what's happening is the user exits after the auto-generation.

@Lina_Farid_WMDE, can you confirm the kind of citation that you want to track and send along an example page?

@Lina_Farid_WMDE, we still want to do a one off task for this as and then you can give us some feedback on the initial numbers and we can build a pipeline from there if need be? Let us know if this works!

Thank you so much, @atsuko! Looking forward to hearing about the next steps 😊

Thanks so much for the context, @Leif_WMDE! Looking forward to when we can pick this up :)

Hey @atsuko 👋 Just wanting to check in here on this as I've been quite busy and haven't been able to look into T435487 yet. Do you have a more concrete understanding of when that should be done by? I can then look to prioritize it 😊

Note: There's a discussion that @Lina_Farid_WMDE could be better served if this is a pipeline that generates a prep table. Effort for this would be similar to that of the one off investigation as the query for the investigation would just need to be instrumented 😊

@Arian_Bozorg and @Lydia_Pintscher: This task is now finalized and I've scheduled a check in call for us to look at WDQM late in the day on Tuesday :) You can see the results at wd_quality_metrics, with the current value for EntitySchema fulfillment being a proportion of 0.67 (~67%).

Happy to, @xcollazo! Just did the following edit: https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FSystems%2FAirflow%2FDeveloper_guide&diff=2465731&oldid=2452021 :)

I took a moment just now to test this and can confirm that adding --set config.airflow.config.email.email_backend=airflow.utils.email.send_email_smtp does allow for the devenv to send an email. We have a MR above to add this to our docs, which resolves this 😊 Thanks @BTullis and @brouberol!

Subscribed @danshick-wmde as well as we'll be getting a copy review for this when we're done to make sure it's accessible to a wide audience. Exciting! 🎉

Note that @catherine.kelsey.wmde did the WD changes preference for September just now. Thanks, @catherine.kelsey.wmde!

Note: We'll merge this with https://phabricator.wikimedia.org/T411872

Same for here, @Lina_Farid_WMDE! Please feel free to add any further information that you feel is needed, but we can also sync when this task is picked up!

Just wrote a quick description here, @Lina_Farid_WMDE! Please feel free to add any further information that you feel is needed, but we can also sync when this task is picked up!

@awight: Could we sync with you on whether or not we're able to derive Wikidata QID citation usage in the coming days? Happy to set up a quick sync for us!

Let's similarly close this task on Thursday after our check in, @Lina_Farid_WMDE!

Note here, @Lina_Farid_WMDE: Let's be sure to confirm that this task is done in our meeting on Thursday :)

Hi @catherine.kelsey.wmde 👋 Looking at the tables, I'm not sure how useful wmf.webrequest will be, but maybe I'm misunderstanding the context of the task. I'd use wmf.webrequest if we know some specific user agents for WikiCite that we want to track the behavior of. For the presence of WikiCite related entities on Wikipedia pages, then we might need to look into wmf_raw.mediawiki_private_linktarget and other related tables where the connection between WD entities and Wikipedia pages are tracked.

@ItamarWMDE, tagging you here as it'd be nice to collaborate here and get your feedback on tmux. I've been kicking this down the road for some time. Goal here is documenting how to properly set up working productively in the stat servers, which you appear to have a lot of insights on. Let me know if it's of interest, and maybe we can schedule a call to work through things and document them!

Note that all stakeholders for this process have been included on the task now. In finalizing the metrics framework we have the goal to align the way we talk about data across the following areas of WMDE work:

Thanks, @brouberol! We'll give it a go in the next DAG testing and report back 😊

Thanks for tagging everyone and the intuition on where this might be coming from, @BTullis!

Based on a conversation with @mahmoud.abdelsattar.wmde, I wanted to comment to bring up a definite need for WikibaseQualityConstraint data being stored persistently. WMDE is using the proportion of mandatory constraints that are consistent as a part of the broader Wikidata Quality Metric (WDQM, epic is T427512). Work for this was originally based on a sample of such constraint statements and checking them via wbcheckconstraints and a bot flagged user (simplest way we could think to get initial results).