Page MenuHomePhabricator

Very large campaigns fail to load images
Open, Needs TriagePublic

Description

One campaign failed to load images after T226303: Save campaign images to the database, namely https://isa.toolforge.org/campaigns/153. This campaign includes Category:Summer with a depth of 5, which in turn means a lot of categories. For instance there are categories following the pattern "August 2019 in Lyon" for a lot of month, year and location combinations. At some point the process crashes.

Event Timeline

I tried running the script on this campaign again and it failed. It runs up until adding the images to the database, before committing. Possibly something is overloaded at this point, there is no error output. Not sure if all the images are added, but there are at least 698,497 log entries for adding images, which is a few times larger than the next largest campaign (which works).

I started an update on this with the script, with the WIP code for T300984 deployed. It has not failed yet after running for about 24 h. It has added 119,294 images and according to Petscan, there are a total of 1,440,971 images in the campaign categories. I'm going to let it run for now and see if it gets done (unless I need to restart the server for some reason).

The script stopped at 156,130 images. There was no error in the log. It ran for about 38 hours.

Since there are campaigns that have nearly as many images as were added (106, 81) I suspect that it's the amount of categories that is the issue. 1,442,496 for the campaign that fails compared to 2,470 and 44 respectively for the two other campaigns.