It seems quite stable now, thank you @Amdrel !

There's no way to limit the memory usage to a predictable amount when calling ffmpeg?

Yes, this is this key that currently allows you to see all the tasks, I added you directly in Redis. I still need to add you in toolsadmin as a maintainer so that you gain SSH access to the instances :)

One thing that should help a lot is a better dispatch of tasks across instances. It seems to me that the new tasks are often picked up by encoding01, even if one of its workers is currently trnscoding a video while other instances are chilling. I'm not 100% sure about that, but we need to check that the load is evenly distributed across the instances.

Right now the number of workers was above our needs I think. I doubled the number of instances when I performed the last Debian migration, and kept this number since then, as the platform was previously often overloaded. Let's see how it goes when reducing from 4 to 3 workers per instance: https://github.com/toolforge/video2commons/pull/267

Yes, it was more stable before, so it's probably linked to this change. I'll try to reduce the number of workers per instance.

And everything crashed again :( Looks like a memory issue?

Everything should be up and running again.

instance encoding04 didn't retart, I'm trying again. Other have restarted and I can see there is a lot of stale output files in /srv/v2c/output:

All instances crashed today, I'm restarting them manually in Horizon.

@Amdrel are you registered to https://toolsadmin.wikimedia.org ? I don't find you when I try to grant you maintainer role.

The system is still unstable, three out of six instances became unresponsive this week and I had to restart them manually. I have no idea what happened, the instances stopped reporting metrics to prometheus/grafana and I couldn't ssh to them, so I restarted them from Horizon.

Thanks a lot! I was able to upload all the videos, problem solved :)

Until a decision is made, could you please at least disable authentication for requests coming from Toolforge? Right now this authentication prevents video2commons to check for YouTube videos already imported.

Yes sure! Thank you :)

Since the creation of this ticket, I made several software updates on v2c backend so maybe it has been solved meanwhile. If you still have the transcoded videos on your laptop, would it be possible for you to upload them by overriding my files?

@Cparle I think we should arrange a presentation of everyone involved in v2c :)

Fix applied on all encoder instances, let's see in the next months if it works: https://github.com/toolforge/video2commons/pull/252

Nice, thank you!

Woops no, I didn't try as I genuinely thought I wasn't able to do it. Thank you Bryan!

@dcaro Is it possible to test this new heroku stack? I'm trying to migrate to openjdk 25, it's supposed to work with heroku-22 but it doesn't, so I'm hoping it would work with the new one.

Ah, now it's working: https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/jobs/621764

Thank you David! I've requested a quota increase in T403602. I'm surprised to not be able to see this information in Horizon, is it planned in a future version?

In T400119#11135537, @Vgutierrez wrote:

It looks like that test for some reason is using the default UA of the HttpClient library: Apache-HttpClient/5.5 (Java/24.0.2)

I still have an error in my unit tests from Gitlab CI when trying to access https://upload.wikimedia.org/wikipedia/commons/d/d2/Epichlorhydrin_vzorec.webp

A new one: https://commons.wikimedia.org/wiki/File:ECDM_20250826_SAIO_Preparedness_actions.pdf

Thank you!

It seems fixed now.

OK I no longer have the error, now it works as expected. I submitted two batches, it works very fine. Thank you for this very useful work, it will save me a lot of time in the future!

Changes deployed. I guess this is fixed now, please reopen if not.

I found a bug, I selected all files of a category that only contains 4 pictures, and I got this popup:

Wow this looks awesome, thank you! I'll try it as soon as I have the need to process an entire category and let you know.

Ah yes, ok.

I try the new version and have a weird bug with gadget being displayed twice:

Hi @GFontenelle_WMF, thank you very much! I agree with your decision, and I'm super happy to have a new monitoring tool at disposal :) I'll read/watch the resources and let you know if I have questions. Thank you again!

It works again! Thank you so much Andrew!

Confirmed also here: https://commons.wikimedia.org/wiki/Commons:Village_pump/Technical#User:FlickreviewR_2_appears_to_be_down

Flickr apparently now blocks all requests with a 403 error (API calls, web page retrievals, everything). Can you please check with Flickr if Wikimedia IP address has been blocked?

Yay, thank you! It works now:

@Raymond_Ndibe I think I have found the problem, can you please take a look at https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/70/diffs?

Hi @Raymond_Ndibe,
To reproduce you can

become spacemedia

and trigger a new build using

./build.sh

which performs the following:

cd /data/project/spacemedia && cd spacemedia && git pull && cd - && \
toolforge build clean -y && \
toolforge build start https://gitlab.wikimedia.org/toolforge-repos/spacemedia --ref main \
                -e MAVEN_JAVA_OPTS='--enable-native-access=ALL-UNNAMED' \
                -e MAVEN_CUSTOM_OPTS='-Pweb -Pversion -ntp -DskipTests -Dspring.profiles.active=toolforge'

Some info from the latest build:

$ toolforge build list
build_id                                 status    start_time            end_time              source_url                                               ref    envvars                                                                                destination_image
spacemedia-buildpacks-pipelinerun-xp7dl  ok        2025-04-13T12:24:33Z  2025-04-13T12:26:58Z  https://gitlab.wikimedia.org/toolforge-repos/spacemedia  main   MAVEN_CUSTOM_OPTS=-Pweb -Pversion -ntp -DskipTests -Dspring.profiles.active=toolforge  tools-harbor.wmcloud.org/tool-spacemedia/tool-spacemedia:latest
                                                                                                                                                               MAVEN_JAVA_OPTS=--enable-native-access=ALL-UNNAMED

$ toolforge build logs spacemedia-buildpacks-pipelinerun-xp7dl | grep -i "maven\|mvn\|env"
[step-prepare] 2025-04-13T12:24:48.869253465Z -> Parsing env variables...
[step-prepare] 2025-04-13T12:24:48.870162580Z > Processing any environment variables...
[step-prepare] 2025-04-13T12:24:48.870211288Z --> Creating 'env' directory: /platform/env
[step-prepare] 2025-04-13T12:24:48.872633629Z --> Writing /platform/env/MAVEN_JAVA_OPTS...
[step-prepare] 2025-04-13T12:24:48.873826517Z --> Writing /platform/env/MAVEN_CUSTOM_OPTS...
[step-inject-buildpacks] 2025-04-13T12:24:52.932925673Z    creating: locale-buildpack-move_to_api_0.10-d55ffc9f67d4305cde8cd95073c5019077316d30/target/fixtures/env/
[step-inject-buildpacks] 2025-04-13T12:24:52.932928088Z   inflating: locale-buildpack-move_to_api_0.10-d55ffc9f67d4305cde8cd95073c5019077316d30/target/fixtures/env/LANG
[step-detect] 2025-04-13T12:24:55.828015664Z heroku/maven    4.0.2
[step-build] 2025-04-13T12:25:00.020397146Z [Installing Maven]
[step-build] 2025-04-13T12:25:00.020410412Z Maven wrapper detected, skipping installation.
[step-build] 2025-04-13T12:25:00.020430465Z [Executing Maven]
[step-build] 2025-04-13T12:25:00.020441231Z $ ./mvnw -Pweb -Pversion -ntp -DskipTests -Dspring.profiles.active clean install
[step-build] 2025-04-13T12:25:00.740236219Z WARNING: java.lang.System::load has been called by org.fusesource.jansi.internal.JansiLoader in an unnamed module (file:/tekton/home/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1/lib/jansi-2.4.1.jar)
[step-build] 2025-04-13T12:25:01.159032647Z WARNING: sun.misc.Unsafe::objectFieldOffset has been called by com.google.common.util.concurrent.AbstractFuture$UnsafeAtomicHelper (file:/tekton/home/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1/lib/guava-33.2.1-jre.jar)
[step-build] 2025-04-13T12:25:46.324080593Z [INFO] --- enforcer:3.5.0:enforce (enforce-maven) @ spacemedia ---
[step-build] 2025-04-13T12:25:46.700182729Z [INFO] Rule 0: org.apache.maven.enforcer.rules.version.RequireMavenVersion passed
[step-build] 2025-04-13T12:25:47.180861421Z [INFO] argLine set to -javaagent:/layers/heroku_maven/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/workspace/target/jacoco.exec
[step-build] 2025-04-13T12:26:01.247204436Z [INFO] argLine set to -javaagent:/layers/heroku_maven/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/workspace/target/jacoco-it.exec
[step-build] 2025-04-13T12:26:01.387510033Z [INFO] Installing /workspace/pom.xml to /layers/heroku_maven/repository/org/wikimedia/commons/donvip/spacemedia/0.5.0-SNAPSHOT/spacemedia-0.5.0-SNAPSHOT.pom
[step-build] 2025-04-13T12:26:01.389395947Z [INFO] Installing /workspace/target/web/spacemedia-0.5.0-SNAPSHOT.jar to /layers/heroku_maven/repository/org/wikimedia/commons/donvip/spacemedia/0.5.0-SNAPSHOT/spacemedia-0.5.0-SNAPSHOT.jar
[step-build] 2025-04-13T12:26:02.066904397Z WARNING: java.lang.System::load has been called by org.fusesource.jansi.internal.JansiLoader in an unnamed module (file:/tekton/home/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1/lib/jansi-2.4.1.jar)
[step-build] 2025-04-13T12:26:02.420717834Z WARNING: sun.misc.Unsafe::objectFieldOffset has been called by com.google.common.util.concurrent.AbstractFuture$UnsafeAtomicHelper (file:/tekton/home/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1/lib/guava-33.2.1-jre.jar)
[step-fix-nested-procfile-launcher] 2025-04-13T12:26:09.813309345Z --> Writing /layers/heroku_procfile/nested_launcher_fix/env/CNB_PLATFORM_API= ...
[step-fix-nested-procfile-launcher] 2025-04-13T12:26:09.813772984Z --> Writing /layers/heroku_procfile/nested_launcher_fix/env/PATH.prepend...
[step-export] 2025-04-13T12:26:57.227896084Z Adding cache layer 'heroku/maven:repository'

@Yann I am not an admin. I remind that I am a volunteer and unwillingly maintain V2C because no one else stepped up when it was completely broken. I'm still hoping that the WMF proposes an official video upload feature someday so we don't rely on exhausted volunteers for this complex infrastructure.

Sorry it's me who deleted the files to ensure they were not filling the disks and making V2C completely unavailable like in May 2024. I was unaware they were related to this ticket. Large files are not supposed to remain this long. If no one actually hanlde these requests in a reasonible delay, I'll have to delete the functionality from V2C to avoid false hopes.

Another one: https://commons.wikimedia.org/wiki/File:ECDM_20250404_Ukraine_Moldova_ECHO(1).pdf

Thank you! It works, I was able to upload the entire collection.

In T384468#10552651, @Andrew wrote:

I would very much like to follow up with flickr support about this. Can I please get some feedback?

Is there a specific user-agent to use? I'm not using a particular one.

Hello,
The problem occurs again for my tool:

Same for me, I tried to login to reboot my VMs as requested but can't login to do it.

OK, it's working fine for my tool as well.

Can we please wait a few days? FlickreviewR 2 was not the only one impacted, I completely stopped Flickr activities on my bot to stop overloading the review category. I just have resumed my uploads, I would like to wait at least 24 hours to see if I face errors.

@Andrew, video2commons is broken and I wonder if it's caused by the NFS incident. Should I wait for the maintenance before doing anything on the video encoder instances, as they are in your list?

It works now, thank you @Ladsgroup!

I can't login on wikitech and don't understand what I need to do, can anyone please help me?

Yes it was discussed and fixed on IRC:

Hi @Slst2020, no problem (and happy new year!)

The error is not always the same, for example with this one I get a different one:

If it helps, I still face problems, last one three minutes ago with https://commons.wikimedia.org/wiki/File:Telline_(Donax_trunculus)_(Ifremer_00673-78543).jpg

It's becoming more and more frequent

Same problem with 6 new files:

• https://commons.wikimedia.org/wiki/File:Vagues_sur_le_littoral_s%C3%A9tois_(Ifremer_00631-74358_-_32014).jpg
• https://commons.wikimedia.org/wiki/File:Vagues_sur_le_littoral_s%C3%A9tois_(Ifremer_00631-74358_-_32015).jpg
• https://commons.wikimedia.org/wiki/File:Vagues_sur_le_littoral_s%C3%A9tois_(Ifremer_00631-74358_-_32017).jpg
• https://commons.wikimedia.org/wiki/File:Vagues_sur_le_littoral_s%C3%A9tois_(Ifremer_00631-74358_-_32018).jpg
• https://commons.wikimedia.org/wiki/File:Vagues_sur_le_littoral_s%C3%A9tois_(Ifremer_00631-74358_-_32019).jpg
• https://commons.wikimedia.org/wiki/File:Vagues_sur_le_littoral_s%C3%A9tois_(Ifremer_00631-74358_-_32020).jpg

Hi @Slst2020! I'm sorry I didn't see your last reply. The Maven part of the build currently takes less than a minute, and the whole build less than 2 minutes, it's perfect:

@dcaro no it happened only once for me. Feel free to close the ticket :)

Yes sorry I just seen it. FYI I just relaunched the job and it worked :)

Would per-format canaries allow to allocate different hardware resources per file format? TIFF files would benefit to get more memory, for example.

With the power of buildpacks I was even able to update very easily to Java 23 :) https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/commit/c00aee97338ec2cb0af4051e2d82555aa8033e8f

@dcaro @Slst2020 have you enabled the cache already? I switched to Toolforge Build Service tonight in order to get Java 21 and the build is really fast! I don't need anything more to increase build performance, I'm very happy with the current situation :)

I managed tonight to:

• donwload jdk21 from Adoptium at build time on GitLab CI and cache it using GitLab cache mechanism. It's fast enough for me
• use jdk21 with toolforge build service / buildpacks. It's also pretty fast

For me the errors are gone (toolforge job service works, I was able to build and deploy my tool. No more DNS errors, everything looks fine).