Change Details

Hey, i'm getting out of memory errors with the [Depictor](https://hay.toolforge.org/depictor/) tool. This is mainly due to the unforeseen popularity of the tool and a system that's not particularly well designed for the >250.000 files it now needs to check for the 'depicted' status'. This will probably mean that i'll need to rewrite parts of the tool, which will take time. In the meantime i think a raise of the memory limit might fix a couple of these issues. Depictor is part of my regular set of tools, hosted under the `hay` project.

I'm getting out of memory errors with the [Depictor](https://hay.toolforge.org/depictor/) tool. This is mainly due to the unforeseen popularity of the tool and a system that's not particularly well designed for the >250.000 files it now needs to check for the 'depicted' status'. This will probably mean that i'll need to rewrite parts of the tool, which will take time. In the meantime i think a raise of the memory limit might fix a couple of these issues. Depictor is part of my regular set of tools, hosted under the `hay` project. #### Some more background for those interested (and for documenting this bug for other people) Originally i got notified about this issue by user PMG (who is the top contributor for Depictor). He noticed he was getting crashes with the tool with the [cultural heritage monuments in Poland challenge](https://hay.toolforge.org/depictor/?challenge=24). What happens at that moment of the 'crash' is that the system checks if the files of the next item are already depicted. Depictor doesn't know in advance which items it's going to get, so it needs to do this every time you load a new item. While developing i noticed that those database calls get *really* slow when you're checking for lots of them (some categories might have hundreds or even thousands of files). So, instead of checking for each file it just fetches all already depicted files *and* the ones that you as a user have skipped (because you don't want to be bothered with those). It then needs to combine the results of those two queries. At this point the script fails, because there are now close to a quarter of a million files in the database, and PMG as the most active user also has a huge list of skipped files. The fetching of those two giant lists and trying to combine them gives an out of memory error. That's what you're seeing when the next image is stuck: the script has halted and the whole system fails. The reason why this is not reproducible i believe is because Toolforge is a shared hosting environment, so memory might simple be used by other processes. When i was testing out your heritage challenge i didn't see any problems at all, but that's probably just because not that many processes were running at the same time. Obviously the main problem here is that this system worked fine when Depictor was still working with just a couple of thousand files, but now that it has gotten so popular the system is no longer capable to cope. Unfortunately i don't think there are any quick fixes for this problem, except for redesigning parts of the system (which will take up time). The monuments in Poland challenge also takes ages to load because it needs to check for the existence of so many items.

Hey, i'mI'm getting out of memory errors with the [Depictor](https://hay.toolforge.org/depictor/) tool. This is mainly due to the unforeseen popularity of the tool and a system that's not particularly well designed for the >250.000 files it now needs to check for the 'depicted' status'. This will probably mean that i'll need to rewrite parts of the tool, which will take time. In the meantime i think a raise of the memory limit might fix a couple of these issues. Depictor is part of my regular set of tools, hosted under the `hay` project. #### Some more background for those interested (and for documenting this bug for other people) Originally i got notified about this issue by user PMG (who is the top contributor for Depictor). He noticed he was getting crashes with the tool with the [cultural heritage monuments in Poland challenge](https://hay.toolforge.org/depictor/?challenge=24). What happens at that moment of the 'crash' is that the system checks if the files of the next item are already depicted. Depictor doesn't know in advance which items it's going to get, so it needs to do this every time you load a new item. While developing i noticed that those database calls get *really* slow when you're checking for lots of them (some categories might have hundreds or even thousands of files). So, instead of checking for each file it just fetches all already depicted files *and* the ones that you as a user have skipped (because you don't want to be bothered with those). It then needs to combine the results of those two queries. At this point the script fails, because there are now close to a quarter of a million files in the database, and PMG as the most active user also has a huge list of skipped files. The fetching of those two giant lists and trying to combine them gives an out of memory error. That's what you're seeing when the next image is stuck: the script has halted and the whole system fails. The reason why this is not reproducible i believe is because Toolforge is a shared hosting environment, so memory might simple be used by other processes. When i was testing out your heritage challenge i didn't see any problems at all, but that's probably just because not that many processes were running at the same time. Obviously the main problem here is that this system worked fine when Depictor was still working with just a couple of thousand files, but now that it has gotten so popular the system is no longer capable to cope. Unfortunately i don't think there are any quick fixes for this problem, except for redesigning parts of the system (which will take up time). The monuments in Poland challenge also takes ages to load because it needs to check for the existence of so many items.