Page MenuHomePhabricator

Raise spacemedia tool memory limit
Closed, InvalidPublic

Description

I have started to deploy my new "spacemedia" tool and just hit the memory limit:

2019-08-10 22:05:57.076 ERROR 9 --- [pool-2-thread-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task.

java.lang.OutOfMemoryError: Java heap space
        at java.awt.image.DataBufferByte.<init>(DataBufferByte.java:92) ~[na:1.8.0_212]
        at java.awt.image.ComponentSampleModel.createDataBuffer(ComponentSampleModel.java:445) ~[na:1.8.0_212]
        at java.awt.image.Raster.createWritableRaster(Raster.java:941) ~[na:1.8.0_212]
        at javax.imageio.ImageTypeSpecifier.createBufferedImage(ImageTypeSpecifier.java:1074) ~[na:1.8.0_212]
        at javax.imageio.ImageReader.getDestination(ImageReader.java:2892) ~[na:1.8.0_212]
        at com.sun.imageio.plugins.jpeg.JPEGImageReader.readInternal(JPEGImageReader.java:1082) ~[na:1.8.0_212]
        at com.sun.imageio.plugins.jpeg.JPEGImageReader.read(JPEGImageReader.java:1050) ~[na:1.8.0_212]
        at org.wikimedia.commons.donvip.spacemedia.utils.Utils.readImage(Utils.java:78) ~[classes!/:0.0.1-SNAPSHOT]
        at org.wikimedia.commons.donvip.spacemedia.utils.Utils.readImage(Utils.java:100) ~[classes!/:0.0.1-SNAPSHOT]
        at org.wikimedia.commons.donvip.spacemedia.service.EsaService.updateMissingImages(EsaService.java:291) ~[classes!/:0.0.1-SNAPSHOT]
        at org.wikimedia.commons.donvip.spacemedia.service.EsaService.updateImages(EsaService.java:314) ~[classes!/:0.0.1-SNAPSHOT]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.9.RELEASE.jar!/:5.1.
9.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]

Can you please increase it? The tool computes SHA-1 hashes of free media released by space agencies in order to detect those missing on Wikimedia Commons. For large images, more memory is needed.

Event Timeline

Don-vip created this task.Aug 10 2019, 10:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 10 2019, 10:30 PM

Is this running on Kubernetes or on Grid Engine?

It's running on Kubernetes / jdk8.

bd808 added a subscriber: bd808.Aug 12 2019, 3:06 AM

This tool has a 4G limit. We do not currently have a method for raising the memory limit for webservices running on the Toolforge Kubernetes cluster (T183436: Add memory limit configuration for Kubernetes pods).

I have not looked deeply at the source code for the tool, but the stacktrace show seems to imply that the SHA1 hash is computed using a java.awt.image.BufferedImage object. This seems like a really heavy way to compute a file's SHA1 value. Is there maybe a technique you could use to compute the SHA1 from a stream of bytes without keeping the entire decoded image in RAM?

My bad. I wrote this code some weeks ago and did not remember that image loading code was here to ensure the files are indeed valid images (to avoid uploading corrupted files to Wikimedia Commons). Some ESA files are corrupted. The SHA-1 hashing code does not require to read the image.

This is not a blocking issue as my tool seems robust to it. A scheduled task that fails because of a memory error will likely succeed on the next try. So I can wait for the other task to be completed :)

I fixed a potential memory leak, let's see if OOM errors are gone.

Don-vip closed this task as Invalid.Thu, Sep 5, 8:22 PM

After some weeks of runtime it appears my tool was able to handle all files. So I don't really need more memory, I cancel this request.

Don-vip moved this task from Backlog to Done on the Tool-spacemedia board.Wed, Sep 11, 7:43 PM