Change Details

==== Context The current scraper did download the per wiki dumps from the public directory. This is not kept up to date anymore so we might have to look for alternatives. ==== Task [x] See if we're still able to retrieve dump files when the scraper is executed on the WMF servers - No, we will have to use the API, there is no filesystem access. [] Implement Enterprise API module in https://gitlab.com/wmde/technical-wishes/mediawiki_client_ex , with verbs for the dump actions we need: - GET https://api.enterprise.wikimedia.com/v2/snapshots - GET https://api.enterprise.wikimedia.com/v2/snapshots/{snapshot_identifier}/chunks - GET https://api.enterprise.wikimedia.com/v2/snapshots/{snapshot_identifier}/chunks/{identifier}/download [] Check if there's a change in format that we need to consider when working with the data & apply fixes if feasible [] Rewrite the application to retrieve dumps through the API. [] Set up our team's private Enterprise credentials on the analytics cluster and document internally. [] Implement unauthenticated mode once the analytics cluster is allow-listed. Api docs: - https://enterprise.wikimedia.com/api/ - https://enterprise.wikimedia.com/docs/snapshot/ == Implementation notes - This probably belongs in the mediawiki_client library, with the generalized goal of accessing "Enterprise API" resources. It's fine to prototype inside of the scraper and move the logic later, but note that the client library is already set up with API fixtures and tests.

==== Context The current scraper did download the per wiki dumps from the public directory. This is not kept up to date anymore so we might have to look for alternatives. ==== Task [x] See if we're still able to retrieve dump files when the scraper is executed on the WMF servers - No, we will have to use the API, there is no filesystem access. [] Implement Enterprise API module in https://gitlab.com/wmde/technical-wishes/mediawiki_client_ex , with verbs for the dump actions we need: - GET https://api.enterprise.wikimedia.com/v2/snapshots - GET https://api.enterprise.wikimedia.com/v2/snapshots/{snapshot_identifier}/chunks - GET https://api.enterprise.wikimedia.com/v2/snapshots/{snapshot_identifier}/chunks/{identifier}/download [] Check if there's a change in format that we need to consider when working with the data & apply fixes if feasible [] Rewrite the application to retrieve dumps through the API. [] Set up our team's private Enterprise credentials on the analytics cluster and document internally. [] Implement unauthenticated mode once the analytics cluster is allow-listed. Api docs: - https://enterprise.wikimedia.com/api/ - https://enterprise.wikimedia.com/docs/snapshot/ == Implementation notes Work in progress branch: https://gitlab.com/wmde/technical-wishes/mediawiki_client_ex/-/merge_requests/22 - This probably belongs in the mediawiki_client library, with the generalized goal of accessing "Enterprise API" resources. It's fine to prototype inside of the scraper and move the logic later, but note that the client library is already set up with API fixtures and tests.

==== Context The current scraper did download the per wiki dumps from the public directory. This is not kept up to date anymore so we might have to look for alternatives. ==== Task [x] See if we're still able to retrieve dump files when the scraper is executed on the WMF servers - No, we will have to use the API, there is no filesystem access. [] Implement Enterprise API module in https://gitlab.com/wmde/technical-wishes/mediawiki_client_ex , with verbs for the dump actions we need: - GET https://api.enterprise.wikimedia.com/v2/snapshots - GET https://api.enterprise.wikimedia.com/v2/snapshots/{snapshot_identifier}/chunks - GET https://api.enterprise.wikimedia.com/v2/snapshots/{snapshot_identifier}/chunks/{identifier}/download [] Check if there's a change in format that we need to consider when working with the data & apply fixes if feasible [] Rewrite the application to retrieve dumps through the API. [] Set up our team's private Enterprise credentials on the analytics cluster and document internally. [] Implement unauthenticated mode once the analytics cluster is allow-listed. Api docs: - https://enterprise.wikimedia.com/api/ - https://enterprise.wikimedia.com/docs/snapshot/ == Implementation notes Work in progress branch: https://gitlab.com/wmde/technical-wishes/mediawiki_client_ex/-/merge_requests/22 - This probably belongs in the mediawiki_client library, with the generalized goal of accessing "Enterprise API" resources. It's fine to prototype inside of the scraper and move the logic later, but note that the client library is already set up with API fixtures and tests.