Change Details

==== Context The current scraper did download the per wiki dumps from the public directory. This is not kept up to date anymore so we might have to look for alternatives. ==== Task [x] See if we're still able to retrieve dump files when the scraper is executed on the WMF servers - No, we will have to use the API, there is no filesystem access. [] Check if there's a change in format that we need to consider when working with the data & apply fixes if feasible [] Rewrite the application to retrieve dumps through the API. [] Set up our team's private Enterprise credentials on the analytics cluster and document internally. See https://enterprise.wikimedia.com/api/

==== Context The current scraper did download the per wiki dumps from the public directory. This is not kept up to date anymore so we might have to look for alternatives. ==== Task [x] See if we're still able to retrieve dump files when the scraper is executed on the WMF servers - No, we will have to use the API, there is no filesystem access. [] Check if there's a change in format that we need to consider when working with the data & apply fixes if feasible [] Rewrite the application to retrieve dumps through the API. [] Set up our team's private Enterprise credentials on the analytics cluster and document internally. See https://enterprise.wikimedia.com/api/ == Implementation notes - This probably belongs in the mediawiki_client library, with the generalized goal of accessing "Enterprise API" resources. It's fine to prototype inside of the scraper and move the logic later, but note that the client library is already set up with API fixtures and tests.

==== Context The current scraper did download the per wiki dumps from the public directory. This is not kept up to date anymore so we might have to look for alternatives. ==== Task [x] See if we're still able to retrieve dump files when the scraper is executed on the WMF servers - No, we will have to use the API, there is no filesystem access. [] Check if there's a change in format that we need to consider when working with the data & apply fixes if feasible [] Rewrite the application to retrieve dumps through the API. [] Set up our team's private Enterprise credentials on the analytics cluster and document internally. See https://enterprise.wikimedia.com/api/ == Implementation notes - This probably belongs in the mediawiki_client library, with the generalized goal of accessing "Enterprise API" resources. It's fine to prototype inside of the scraper and move the logic later, but note that the client library is already set up with API fixtures and tests.