Change Details

`https://dumps.wikimedia.org/` does not have a robots.txt. We added Terms of Use for dumps: https://phabricator.wikimedia.org/T408881 Adding a complementary robots.txt file makes sense. As a basis we should use `https://en.wikipedia.org/robots.txt` stripped of the non-relevant Wikipedia paths.

`https://dumps.wikimedia.org/` does not have a `robots.txt`. We added Terms of Use for dumps: {T408881} Adding a complementary robots.txt file makes sense. As a basis, we should use [[ https://en.wikipedia.org/robots.txt | https://en.wikipedia.org/robots.txt ]], stripped of the irrelevant Wikipedia paths.

`https://dumps.wikimedia.org/` does not have a `robots.txt`. We added Terms of Use for dumps: https://phabricator.wikimedia.org/{T408881} Adding a complementary robots.txt file makes sense. As a basis, we should use `se [[ https://en.wikipedia.org/robots.txt | https://en.wikipedia.org/robots.txt` txt ]], stripped of the non-irrelevant Wikipedia paths.