A question on this as I'm writing the basics while I'm waiting on info for the testing of the other DAGs:

• Do we want only the data for the previous month, or should we also include the data from the last two months as well?
  • Checking to see if it was helpful to have the only May and not April user agents as well
  • I think that current month and previous month is the best easy backdate we can do as if we do more there could be more than 90 days (March, April and May together are 92 days)
  • Happy to include this as it's really not much more effort!

I'll write the DAG and we can see how access to the data works if it's in a table in the data lake. Very happy to send this info along on a monthly basis, and we can just have the habit to delete the last one when the new one is sent.

Good point, @AndrewTavis_WMDE, monthly iterations make most sense considering the 90 day limit. Let's go with what you suggested.

@Lydia_Pintscher, could you please still confirm if the comparison with the previous month helped to get added insights or not, so we know if to include it this table in future iterations or not.

MR is already open with the basic structure of the DAG (basically just the first REST API one with minor modifications). I'll finalize this once the REST API metrics one and the sitelinks one are finalized as far as exporting to the published datasets. Only question on those now is where I should be sending test data to (HDFS or the stat machines).

Hi @Lydia_Pintscher, fyi, I just decided to keep the comparison with the previous month and asked Andrew to create a "new this month" column to represent it. Cheers!

The DAG its job at this point are ready. The merge request has been updated and can be seen here: gitlab.wikimedia.org/repos/data-engineering/airflow-dags/merge_requests/738. I don't have time now to do the local testing as I tried earlier today and the query was taking too long, but this will be finished quickly after I'm back.

Took a quick moment to figure out what's going on with this DAG job query so that it can be included in all of the deployments we're doing in the coming days. Specifically the job was not finishing with the setup that we had before where we wanted user agents, their total requests, and whether they had also made a REST API request in the previous month. Optimized the query as much as I could, but it was not finishing over a monthly period. I've now edited the query so that we're just getting user agents and their total requests within the last month ordered in a decreasing manner based on the total requests. The query for this finishes in 15 minutes as the data ingested still means that query planning cannot be done in an optimized manner.

Will send along the create table statement for this so that we'll be ready for DAG testing with the rest of the ones we're working on :)