Blocking bots (July 21 / Oct 14 2020, and Jan 18 / March 29, 2021)

The bots blocked on July 21, 2020 may have made a noticeable difference, as we saw a ~30% decrease in exports:

Data

Here's a daily report of number of exports since December 1, 2020, with average export times starting January 14:

MariaDB [s52561__wsexport_p]> SELECT DATE(time), COUNT(id), AVG(duration) FROM books_generated WHERE YEAR(time) = 2021 OR (YEAR(time) = 2020 AND MONTH(time) = 12) GROUP BY DATE(time);

Visualization

The blue bars are number of exports, and the red line is the average export time. The straight lines are the trends, which at face value clearly show the number of exports on an upward trend, and the average export time on a downward trend. The red line incorrectly is accounting for nonexistent data prior to January 14, which I can't seem to correct in Google Sheets, but there is in fact a downward trend starting January 14.

One interesting pattern is that more exports seem to equate to a lower average export time. This might be explained by repeated and rapid exports of the same book, which pull from the cache and export much faster, hence bringing down the average.

Conclusions

It is difficult to draw any conclusions since there are many factors that can skew the data, particularly bots which can come and go at any time.

In mid January we started seeing an increase in exports, which might be attributed to moving the download links to the Wikisource extension. This meant the HTML for the links was present on page load, rather than it being added by JavaScript, so more web crawlers and bots that don't understand JavaScript might have started clicking on those links. This is just a theory, though.

The other variances I am not able to explain without significant and tedious effort.

Note that all wikis were stuck on wmf.27 for a few weeks. On February 16, all wikis were promoted to wmf.30 (T271344#6833578) which brought both the download button to all Wikisources as well as replacing Electron PDF with WS Export. Within days, we saw a dramatic increase (+600%) in downloads, as well as spikes in CPU usage: https://grafana-labs.wikimedia.org/goto/WG9qUSlMk. I cannot conclusively tie these events together, but it seems probable they are related.

Caching HTTP requests should by all accounts decrease the download times for subsequent exports in short succession (which is not uncommon), but the number of other one-off exports where caching isn't beneficial is probably enough to skew the numbers, and I suspect that's why it appears not much improved after February 17. However the trend of average export times has clearly decreased.

The drop in downloads starting March 29 might be explained by the disruptive bot that was blocked.

Thank you so much for this analysis, @MusikAnimal!

One quick observation & question: Yup, the blocked bot definitely correlates, timeline-wise, to the drop in downloads on March 29th. Do we know if this bot was also affecting the numbers *before* our changes, or is it possible that it only impacted export data after our changes (since export links may have been easier for web crawlers/other cause) after our changes? Or is it hard to say?

Either way, the export numbers are still larger after our changes, but any additional context on the bot would be helpful. Thank you!

In T278439#6972573, @ifried wrote:

One quick observation & question: Yup, the blocked bot definitely correlates, timeline-wise, to the drop in downloads on March 29th. Do we know if this bot was also affecting the numbers *before* our changes, or is it possible that it only impacted export data after our changes (since export links may have been easier for web crawlers/other cause) after our changes? Or is it hard to say?

I should have checked more when I did this analysis... The logs only go back two weeks. I checked the oldest ones we have at the time of writing, dated March 23 and March 24, and there were no instances of that particular user agent. I highly suspect other bots were involved (perhaps the same one too, just using a different user agent), because the massive increase in exports starting February 23 seems to be too much to attribute to human activity, though as I said the February 16 release probably influenced it some, too. Indeed it is hard to say :(

What will be really interesting is to see the numbers once we implemented IP-based throttling, which will stop the vast majority of bots once and for all. That's a 10% project I plan on implementing in the near future, assuming our request with Cloud Services is approved (T279111).