This other work should be done before adding the generation-time: T266190: Wikisource Export: Migrate existing DB system to use Doctrine ORM

In T267079#6605718, @Samwilson wrote:

This other work should be done before adding the generation-time: T266190: Wikisource Export: Migrate existing DB system to use Doctrine ORM

Yes, good point

PR ready for review: https://github.com/wsexport/tool/pull/298

Also, there's nothing in this task about actually displaying the duration information anywhere. Is that okay? Do we want to add something to the statistics page?

The first part of this is merged and is ready for QA. There's not much to look at other than directly in the database.

@Samwilson Yes, great question about actually displaying the duration! Perhaps I can talk with @jwang (after our holiday break) about creating a table to display the calculated average time per download over a period of time. I think a statistics page would be great, but it can be handled in a separate ticket. I have created a placeholder ticket to separately explore this question: T270770. Thanks!

We now record the number of seconds it took to generate an ebook, rounded up to the nearest second.

It is recorded in the books_generated mysql table as the duration column. See example below.

I compared the duration recorded in the database with timings I recorded myself (in the browser and python requests library). The timings seemed to agree.

Example

SELECT time, lang, title, format, duration FROM books_generated WHERE title="Life_in_a_Thousand_Worlds" AND duration IS NOT NULL ORDER BY time;

+------------------------+------+---------------------------+--------+----------+
| time                   | lang | title                     | format | duration |
+------------------------+------+---------------------------+--------+----------+
| 2020-12-25 07:05:22.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        2 |
| 2020-12-26 12:53:37.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        1 |
| 2021-01-04 09:53:53.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        2 |
| 2021-01-04 09:56:28.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        1 |
| 2021-01-04 09:56:44.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        1 |
| 2021-01-04 13:08:19.00 | en   | Life_in_a_Thousand_Worlds | pdf-a4 |       43 |
| 2021-01-04 13:09:00.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        1 |
| 2021-01-05 15:34:18.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        2 |
| 2021-01-06 09:17:29.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        1 |
| 2021-01-06 09:17:52.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        1 |
| 2021-01-06 09:29:13.00 | en   | Life_in_a_Thousand_Worlds | pdf-a4 |       34 |
| 2021-01-06 09:31:08.00 | en   | Life_in_a_Thousand_Worlds | pdf-a4 |       34 |
| 2021-01-06 11:12:41.00 | en   | Life_in_a_Thousand_Worlds | epub-3 |        1 |
+------------------------+------+---------------------------+--------+----------+

@ifried @jwang It has just occured to me. When the generation of an ebook times out (after 300 seconds on production, I think) we don't record it in the books_generated database. So, we won't have any performance data for those ebooks.

I don't know if this will give us misleading data, e.g. we only have data for smaller books, and no data for books which are causing us problems. I don't know what we can do about it from a technical perspective, though.

Books which don't generate successfully for other reasons (i.e. due to an exception in the code) are not recorded in the database either, although I don't know if this is as much of an issue.

@dom_walden Thank you for providing this information! It's a really great point.

My thoughts so far: I still think there would be value in charting the data that we can get, even if it is not representative of the worst cases that pass 300 seconds on production. This is because we can still identify trends (even if they are based on a smaller sample size rather than all downloads), such as average times for generating ebooks, general trends in duration over time, trends related to file formats, etc. This data can help us better understand the impact or changes over time, even if it doesn't show the worst cases.

Just to clarify, @Samwilson: There's no way we can get data on the ebook exports that time out? Or is there a way? Just want to understand the options for handling this. Thanks in advance!

In T267079#6729305, @ifried wrote:

Just to clarify, @Samwilson: There's no way we can get data on the ebook exports that time out? Or is there a way? Just want to understand the options for handling this. Thanks in advance!

FWIW, we could store all downloads (even if it fails) in the db if that is information we want to keep around. Currently, we log the failures but I'm not sure if those logs are being rotated or for how long we keep them.

Okay, thanks, @dmaza. I've added this information to the follow-up ticket (T271336), so it doesn't get lost. We'll discuss it in a future meeting.