@dancy it worked! Thanks a lot!

It didn't help, sadly.

My jobs are systematically being OOMKilled for three weeks now. My latest successful run was on 2026-04-23.
Since then I was unable to have a job not being OOMKilled.
This completely breaks the CI I put in place for my Toolforge project, is there something I can do, or should I have to wait for T389936 to be done?

It looks solved now, thanks to these fixes.

The error triggers when refreshing a running task, so you must either have a running task owned by your account, or be a video2commons admin to see all tasks.

Another one: https://commons.wikimedia.org/wiki/File:20260410_Ukraine_MEDEVAC_DM.pdf

Yes it's fixed, I was just curious :) Thank you so much, team!

@Scott_French out of curiosity, why a very few number of videos were still available, I don't really understand the error?

Yeeeees videos are back, thank you so much @Scott_French !

I wonder if it's the same problem as T422074 which also points to a video on Commons.

In T422071#11780221, @Kallichore wrote:

It appears to me that all videos are affected.

Screenshot in private mode / English:

For the point 3, if you need volunteers to beta-test the upgrade of a mariadb instance, I'm available. I have a single instance of 250GB in the spacemedia project. I am quite interested in this upgrade because Hibernate warns me that the 10.5 version is unsupported for quite some time now.

Hi David, thank you so much! I was able to update to java 25 very smoothly:
https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/commit/81c1e6a5481dc39bae564fecd6fec4889412b593

@dcaro is this update live for us? https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1175/diffs

My bot didn't face any particular error from the API when the file has been uploaded then its SDC modified:

Is there a problem with this particular file? (it has been uploaded by my bot)

The issue is solved for me, thanks a lot @matmarex !

Thank you @Andrew for approving the request!
@Amdrel : everything is setup, the account is able to ssh to the encoder instances, I have setup a new environment in GitHub repository with the private key as a secret. I have restricted deployments to the main branch + a manual validation from either you or me to approve the deployment, so that we keep things under control.

Thank you! No worry, I hope you're all okay.

Account created, someone needs to validate its toolforge membership request I think: https://toolsadmin.wikimedia.org/tools/membership/status/2113

Sure! I look how to create it.

Nice, thanks to both of you :)

I see the instance is gone now. @bd808 : did someone fix the issue or did it solve by itself?

I agree of course, as I don't have any clue about doing it differently :)

Another one: https://commons.wikimedia.org/wiki/File:20260114_reliefEU_funding_2025_overview_DM(1).pdf

Another one: https://commons.wikimedia.org/wiki/File:ECDM_20260109_Sudan_Conflict_and_pop_displ.pdf

Right now my update worklow is:

• ssh to each encoder
• sudo celery -A video2commons.backend.worker --workdir=/srv/v2c multi stopwait --pidfile="/var/run/v2ccelery/%N.pid" 2
• wait sometimes several hours...
• sudo puppet apply backend-bookworm.pp --debug

Another one: https://commons.wikimedia.org/wiki/File:ECDM_20251215_Mozambique_Complex_Crisis.pdf

It seems quite stable now, thank you @Amdrel !

There's no way to limit the memory usage to a predictable amount when calling ffmpeg?

Yes, this is this key that currently allows you to see all the tasks, I added you directly in Redis. I still need to add you in toolsadmin as a maintainer so that you gain SSH access to the instances :)

One thing that should help a lot is a better dispatch of tasks across instances. It seems to me that the new tasks are often picked up by encoding01, even if one of its workers is currently trnscoding a video while other instances are chilling. I'm not 100% sure about that, but we need to check that the load is evenly distributed across the instances.

Right now the number of workers was above our needs I think. I doubled the number of instances when I performed the last Debian migration, and kept this number since then, as the platform was previously often overloaded. Let's see how it goes when reducing from 4 to 3 workers per instance: https://github.com/toolforge/video2commons/pull/267

Yes, it was more stable before, so it's probably linked to this change. I'll try to reduce the number of workers per instance.

And everything crashed again :( Looks like a memory issue:

Everything should be up and running again.

instance encoding04 didn't restart, I'm trying again. Other have restarted and I can see there is a lot of stale output files in /srv/v2c/output:

All instances crashed today, I'm restarting them manually in Horizon.

@Amdrel are you registered to https://toolsadmin.wikimedia.org ? I don't find you when I try to grant you maintainer role.

The system is still unstable, three out of six instances became unresponsive this week and I had to restart them manually. I have no idea what happened, the instances stopped reporting metrics to prometheus/grafana and I couldn't ssh to them, so I restarted them from Horizon.

Thanks a lot! I was able to upload all the videos, problem solved :)

Until a decision is made, could you please at least disable authentication for requests coming from Toolforge? Right now this authentication prevents video2commons to check for YouTube videos already imported.

Yes sure! Thank you :)

Since the creation of this ticket, I made several software updates on v2c backend so maybe it has been solved meanwhile. If you still have the transcoded videos on your laptop, would it be possible for you to upload them by overriding my files?

@Cparle I think we should arrange a presentation of everyone involved in v2c :)

Fix applied on all encoder instances, let's see in the next months if it works: https://github.com/toolforge/video2commons/pull/252

Nice, thank you!

Woops no, I didn't try as I genuinely thought I wasn't able to do it. Thank you Bryan!

@dcaro Is it possible to test this new heroku stack? I'm trying to migrate to openjdk 25, it's supposed to work with heroku-22 but it doesn't, so I'm hoping it would work with the new one.

Ah, now it's working: https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/jobs/621764

Thank you David! I've requested a quota increase in T403602. I'm surprised to not be able to see this information in Horizon, is it planned in a future version?

In T400119#11135537, @Vgutierrez wrote:

It looks like that test for some reason is using the default UA of the HttpClient library: Apache-HttpClient/5.5 (Java/24.0.2)

I still have an error in my unit tests from Gitlab CI when trying to access https://upload.wikimedia.org/wikipedia/commons/d/d2/Epichlorhydrin_vzorec.webp

A new one: https://commons.wikimedia.org/wiki/File:ECDM_20250826_SAIO_Preparedness_actions.pdf

Thank you!

It seems fixed now.

OK I no longer have the error, now it works as expected. I submitted two batches, it works very fine. Thank you for this very useful work, it will save me a lot of time in the future!

Changes deployed. I guess this is fixed now, please reopen if not.

I found a bug, I selected all files of a category that only contains 4 pictures, and I got this popup:

Wow this looks awesome, thank you! I'll try it as soon as I have the need to process an entire category and let you know.

Ah yes, ok.

I try the new version and have a weird bug with gadget being displayed twice:

Hi @GFontenelle_WMF, thank you very much! I agree with your decision, and I'm super happy to have a new monitoring tool at disposal :) I'll read/watch the resources and let you know if I have questions. Thank you again!

It works again! Thank you so much Andrew!

Confirmed also here: https://commons.wikimedia.org/wiki/Commons:Village_pump/Technical#User:FlickreviewR_2_appears_to_be_down

Flickr apparently now blocks all requests with a 403 error (API calls, web page retrievals, everything). Can you please check with Flickr if Wikimedia IP address has been blocked?

Yay, thank you! It works now:

@Raymond_Ndibe I think I have found the problem, can you please take a look at https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/70/diffs?

Hi @Raymond_Ndibe,
To reproduce you can

become spacemedia

and trigger a new build using

./build.sh

which performs the following:

cd /data/project/spacemedia && cd spacemedia && git pull && cd - && \
toolforge build clean -y && \
toolforge build start https://gitlab.wikimedia.org/toolforge-repos/spacemedia --ref main \
                -e MAVEN_JAVA_OPTS='--enable-native-access=ALL-UNNAMED' \
                -e MAVEN_CUSTOM_OPTS='-Pweb -Pversion -ntp -DskipTests -Dspring.profiles.active=toolforge'

Some info from the latest build:

$ toolforge build list
build_id                                 status    start_time            end_time              source_url                                               ref    envvars                                                                                destination_image
spacemedia-buildpacks-pipelinerun-xp7dl  ok        2025-04-13T12:24:33Z  2025-04-13T12:26:58Z  https://gitlab.wikimedia.org/toolforge-repos/spacemedia  main   MAVEN_CUSTOM_OPTS=-Pweb -Pversion -ntp -DskipTests -Dspring.profiles.active=toolforge  tools-harbor.wmcloud.org/tool-spacemedia/tool-spacemedia:latest
                                                                                                                                                               MAVEN_JAVA_OPTS=--enable-native-access=ALL-UNNAMED

$ toolforge build logs spacemedia-buildpacks-pipelinerun-xp7dl | grep -i "maven\|mvn\|env"
[step-prepare] 2025-04-13T12:24:48.869253465Z -> Parsing env variables...
[step-prepare] 2025-04-13T12:24:48.870162580Z > Processing any environment variables...
[step-prepare] 2025-04-13T12:24:48.870211288Z --> Creating 'env' directory: /platform/env
[step-prepare] 2025-04-13T12:24:48.872633629Z --> Writing /platform/env/MAVEN_JAVA_OPTS...
[step-prepare] 2025-04-13T12:24:48.873826517Z --> Writing /platform/env/MAVEN_CUSTOM_OPTS...
[step-inject-buildpacks] 2025-04-13T12:24:52.932925673Z    creating: locale-buildpack-move_to_api_0.10-d55ffc9f67d4305cde8cd95073c5019077316d30/target/fixtures/env/
[step-inject-buildpacks] 2025-04-13T12:24:52.932928088Z   inflating: locale-buildpack-move_to_api_0.10-d55ffc9f67d4305cde8cd95073c5019077316d30/target/fixtures/env/LANG
[step-detect] 2025-04-13T12:24:55.828015664Z heroku/maven    4.0.2
[step-build] 2025-04-13T12:25:00.020397146Z [Installing Maven]
[step-build] 2025-04-13T12:25:00.020410412Z Maven wrapper detected, skipping installation.
[step-build] 2025-04-13T12:25:00.020430465Z [Executing Maven]
[step-build] 2025-04-13T12:25:00.020441231Z $ ./mvnw -Pweb -Pversion -ntp -DskipTests -Dspring.profiles.active clean install
[step-build] 2025-04-13T12:25:00.740236219Z WARNING: java.lang.System::load has been called by org.fusesource.jansi.internal.JansiLoader in an unnamed module (file:/tekton/home/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1/lib/jansi-2.4.1.jar)
[step-build] 2025-04-13T12:25:01.159032647Z WARNING: sun.misc.Unsafe::objectFieldOffset has been called by com.google.common.util.concurrent.AbstractFuture$UnsafeAtomicHelper (file:/tekton/home/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1/lib/guava-33.2.1-jre.jar)
[step-build] 2025-04-13T12:25:46.324080593Z [INFO] --- enforcer:3.5.0:enforce (enforce-maven) @ spacemedia ---
[step-build] 2025-04-13T12:25:46.700182729Z [INFO] Rule 0: org.apache.maven.enforcer.rules.version.RequireMavenVersion passed
[step-build] 2025-04-13T12:25:47.180861421Z [INFO] argLine set to -javaagent:/layers/heroku_maven/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/workspace/target/jacoco.exec
[step-build] 2025-04-13T12:26:01.247204436Z [INFO] argLine set to -javaagent:/layers/heroku_maven/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/workspace/target/jacoco-it.exec
[step-build] 2025-04-13T12:26:01.387510033Z [INFO] Installing /workspace/pom.xml to /layers/heroku_maven/repository/org/wikimedia/commons/donvip/spacemedia/0.5.0-SNAPSHOT/spacemedia-0.5.0-SNAPSHOT.pom
[step-build] 2025-04-13T12:26:01.389395947Z [INFO] Installing /workspace/target/web/spacemedia-0.5.0-SNAPSHOT.jar to /layers/heroku_maven/repository/org/wikimedia/commons/donvip/spacemedia/0.5.0-SNAPSHOT/spacemedia-0.5.0-SNAPSHOT.jar
[step-build] 2025-04-13T12:26:02.066904397Z WARNING: java.lang.System::load has been called by org.fusesource.jansi.internal.JansiLoader in an unnamed module (file:/tekton/home/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1/lib/jansi-2.4.1.jar)
[step-build] 2025-04-13T12:26:02.420717834Z WARNING: sun.misc.Unsafe::objectFieldOffset has been called by com.google.common.util.concurrent.AbstractFuture$UnsafeAtomicHelper (file:/tekton/home/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1/lib/guava-33.2.1-jre.jar)
[step-fix-nested-procfile-launcher] 2025-04-13T12:26:09.813309345Z --> Writing /layers/heroku_procfile/nested_launcher_fix/env/CNB_PLATFORM_API= ...
[step-fix-nested-procfile-launcher] 2025-04-13T12:26:09.813772984Z --> Writing /layers/heroku_procfile/nested_launcher_fix/env/PATH.prepend...
[step-export] 2025-04-13T12:26:57.227896084Z Adding cache layer 'heroku/maven:repository'

@Yann I am not an admin. I remind that I am a volunteer and unwillingly maintain V2C because no one else stepped up when it was completely broken. I'm still hoping that the WMF proposes an official video upload feature someday so we don't rely on exhausted volunteers for this complex infrastructure.

Sorry it's me who deleted the files to ensure they were not filling the disks and making V2C completely unavailable like in May 2024. I was unaware they were related to this ticket. Large files are not supposed to remain this long. If no one actually hanlde these requests in a reasonible delay, I'll have to delete the functionality from V2C to avoid false hopes.

Another one: https://commons.wikimedia.org/wiki/File:ECDM_20250404_Ukraine_Moldova_ECHO(1).pdf

Thank you! It works, I was able to upload the entire collection.

In T384468#10552651, @Andrew wrote:

I would very much like to follow up with flickr support about this. Can I please get some feedback?