Before:

sudo radosgw-admin user info --uid 84baa6f9fe8d41afb4b7ca99891161f3\$84baa6f9fe8d41afb4b7ca99891161f3
{
    "user_id": "84baa6f9fe8d41afb4b7ca99891161f3$84baa6f9fe8d41afb4b7ca99891161f3",
    "display_name": "etherpads3",
    ...
    "max_buckets": 1000,
    ...
    "bucket_quota": {
        "enabled": false,
        ...
    },
    "user_quota": {
        "enabled": true,
        "check_on_raw": false,
        "max_size": 8589934592,
        "max_size_kb": 8388608,
        "max_objects": 4096
    },
    ...
}

Command:

sudo radosgw-admin quota set --quota-scope=user --uid 84baa6f9fe8d41afb4b7ca99891161f3\$84baa6f9fe8d41afb4b7ca99891161f3 --max-objects=65536

After:

sudo radosgw-admin user info --uid 84baa6f9fe8d41afb4b7ca99891161f3\$84baa6f9fe8d41afb4b7ca99891161f3
{
    "user_id": "84baa6f9fe8d41afb4b7ca99891161f3$84baa6f9fe8d41afb4b7ca99891161f3",
    "display_name": "etherpads3",
    ...
    "max_buckets": 1000,
    ...
    "bucket_quota": {
        "enabled": false,
        ...
    },
    "user_quota": {
        "enabled": true,
        "check_on_raw": false,
        "max_size": 8589934592,
        "max_size_kb": 8388608,
        "max_objects": 65536
    },
    ...
}

Oppose. Pushing a tag should be the action that triggers the release pipeline.

The tagging is still the action triggering the release.
This task is just about removing the need for a human to manually push the tag, which should make it more difficult to make mistakes.
If a human is not responsible for manually pushing the tag, then we don't have to worry about both pypi release or other contributors having stale tags locally.

Yes Danya, push-to-deploy doesn't support webservice yet. We are working on that, though it won't be out like say, tomorrow

In T422929#11808204, @taavi wrote:

This seems to be a reoccurance of T365048 (and thus a real bug). Something's caused the pod to exit and restart (but without recreating the Pod API object), and since the Secret references are injected at Pod creation time only it's blocking the pod from starting back up. I can think of a few different fixes:

• envvars-api could use a single Secret for all of the envvars
• envvars-api could restart all pods referencing a Secret that's being removed
• something could watch for pods that end up in this restarting problem state and manually delete them via the API.

did toolforge jobs load jobs.yaml

This will not restart jobs where the definition has not changed. toolforge jobs restart file-renaming will do that.

can you verify that providing an environment variable fixes this problem for you @MBH ?

It seems like your job is missing some toolforge environment variable. To see your available environment variable, run toolforge envvars list.

tools.mbh@tools-bastion-15:~$ toolforge jobs list
+---------------+---------------------+------------------------------------------+
|   Job name:   |      Job type:      |                 Status:                  |
+---------------+---------------------+------------------------------------------+
|   countries   |       one-off       |           Running for 1h4m20s            |
|     daily     | schedule: 3 0 * * * | Last schedule time: 2026-04-10T00:03:00Z |
|    monthly    | schedule: 5 0 2 * * | Last schedule time: 2026-04-01T00:05:00Z |
| file-renaming |     continuous      |               Not running                |
+---------------+---------------------+------------------------------------------+
tools.mbh@tools-bastion-15:~$ toolforge jobs show file-renaming
+---------------+------------------------------------------------------------------------+
| Job name:     | file-renaming                                                          |
+---------------+------------------------------------------------------------------------+
| Command:      | mono /data/project/mbh/bots/file-renaming.exe                          |
+---------------+------------------------------------------------------------------------+
| Job type:     | continuous                                                             |
+---------------+------------------------------------------------------------------------+
| Image:        | mono6.8                                                                |
+---------------+------------------------------------------------------------------------+
| Port:         | none                                                                   |
+---------------+------------------------------------------------------------------------+
| File log:     | yes                                                                    |
+---------------+------------------------------------------------------------------------+
| Output log:   | /data/project/mbh/file-renaming.out                                    |
+---------------+------------------------------------------------------------------------+
| Error log:    | /data/project/mbh/file-renaming.err                                    |
+---------------+------------------------------------------------------------------------+
| Emails:       | none                                                                   |
+---------------+------------------------------------------------------------------------+
| Resources:    | default                                                                |
+---------------+------------------------------------------------------------------------+
| Replicas:     | 1                                                                      |
+---------------+------------------------------------------------------------------------+
| Mounts:       | none                                                                   |
+---------------+------------------------------------------------------------------------+
| Retry:        | no                                                                     |
+---------------+------------------------------------------------------------------------+
| Timeout:      | no                                                                     |
+---------------+------------------------------------------------------------------------+
| Health check: | none                                                                   |
+---------------+------------------------------------------------------------------------+
| Status:       | Not running                                                            |
+---------------+------------------------------------------------------------------------+
| Hints:        | Last run at 2026-03-19T16:17:00Z. Pod in 'Running' phase. Pod has been |
|               | restarted 9 times. State 'waiting'. Reason                             |
|               | 'CreateContainerConfigError'. Additional message:'secret               |
|               | "toolforge.envvar.v1.conn-string" not found'.                          |
+---------------+------------------------------------------------------------------------+

Most promising paths:

1. Adding max_query_length: 336h to loki configuration. This will always raise the query time range exceeds the limit... when range exceeds 336h. works for 30d, 100d, 1000d, 10000d, etc.
2. Adding max_query_length: 336h and max_query_lookback: 336h to loki configuration. This will always ignore the the days that fall outside 336h and return everything else.

@Soda You can now see all your logs using --since to adjust how far in the past the logs should be gotten from.

Note to self:
for the link objects cleanup, maybe we can pull the image before the cleanup, and verify that we can pull the image after the cleanup. If we can't (meaning the cleanup broke the image), push it back to establish the link again. This can be done in addition to other ways to mitigate risks. Noting down here so we don't forget what we discussed during the session with David.

I added documentation for the s3 lifecycle configuration part here https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#Known_issues_with_Docker_registries.
I also configured that on harborstorage buckets for tools and toolsbeta and that got rid of the massive 20G difference between what is reported on horizon and what is reported on harbor:

Update After Further Research

TL;DR:

• maintain-harbor job to handle harbor vs s3 orphaned objects cleanup (is this safe? are their safer alternative? what happens if we just ignore this?)
• lifecycle rules to handle old multipart upload data and documentation of this on wikitech
• cookbook to handle ceph3 orphaned objects cleanup

I digged deeper into this. https://github.com/goharbor/harbor/issues/22111 is one of our problems, but is not the major one. Below are other related issues I researched:

• https://github.com/goharbor/harbor/issues/17651
• https://github.com/goharbor/harbor/issues/15641
• https://www.reddit.com/r/aws/comments/1oosbhh/s3_incomplete_multipart_uploads_are_dangerous_1tb/

not sure if we should do this. We had a decision request and the final decision was to manually handle version bumping https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T373072_to_strictly_enforce_semantic_versioning_rules_for_toolforge_services_APIs_or_not

described the issue and solution here https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/62#note_190853

In T417594#11632621, @MatthewVernon wrote:

@Raymond_Ndibe this is done - give it half an hour for puppet to run everywhere, and you should be good to go.

When you're ready to have the old software-only key removed, either open a new ticket, or reply to this one.

Done

In T417594#11626420, @MatthewVernon wrote:

@Raymond_Ndibe I sent you a slack message yesterday - can you either reply to that with your new ssh public key, or upload a patch set to gerrit with the new key in? If neither of those works for you we can find another way to confirm the new key, but those are usually the simplest methods.

In T412081#11533902, @NicoV wrote:

Hi @Raymond_Ndibe, a few answers:

• I'm using wpcleaner tool account
• The jobs are using WPCleaner tool
• The problem with toolforge jobs saying my jobs have no log seems systematic
• The problem with 2 jobs running at the same time didn't happen again, but I think you can easily test it (short schedule, job during more than the schedule)
• Since the problem with the job marked as running by toolforge, but waiting for resource by kubectl, I lowered my CPU requirements (from 3 to 1) : I think you can reproduce by starting a job with a high CPU requirement which will get queued by the cluster

Do you need more details ?

Here's an example with the logs:

tools.wpcleaner@tools-bastion-15:~$ toolforge jobs list
+------------------------+--------------------+------------------------------------------+
|       Job name:        |     Job type:      |                 Status:                  |
+------------------------+--------------------+------------------------------------------+
|  wpcleaner-cs-weekly   | schedule: @weekly  | Last schedule time: 2026-01-14T05:31:00Z |
|   wpcleaner-en-list    | schedule: @weekly  |           Running for 9h50m23s           |
|    wpcleaner-fr-dab    | schedule: @weekly  | Last schedule time: 2026-01-16T21:48:00Z |
|   wpcleaner-fr-daily   |  schedule: @daily  | Last schedule time: 2026-01-19T08:21:00Z |
|   wpcleaner-fr-list    | schedule: @weekly  | Last schedule time: 2026-01-18T00:46:00Z |
|  wpcleaner-fr-weekly   | schedule: @weekly  | Last schedule time: 2026-01-13T20:02:00Z |
|  wpcleaner-meta-list   | schedule: @monthly | Last schedule time: 2026-01-19T03:00:00Z |
| wpcleaner-meta-monthly | schedule: @monthly | Last schedule time: 2026-01-08T04:43:00Z |
+------------------------+--------------------+------------------------------------------+
tools.wpcleaner@tools-bastion-15:~$ toolforge jobs logs wpcleaner-en-list
ERROR: Job 'wpcleaner-en-list' does not have any logs available
tools.wpcleaner@tools-bastion-15:~$ kubectl get jobs
NAME                         STATUS    COMPLETIONS   DURATION   AGE
wpcleaner-en-list-29479915   Running   0/1           9h         9h
tools.wpcleaner@tools-bastion-15:~$ kubectl logs jobs/wpcleaner-en-list-29479915 | tail
03:59:36.272 [MW-1] INFO API - GET  https://en.wikipedia.org/w/api.php?curtimestamp=1&continue=&prop=revisions|info&inprop=protection&format=xml&rvslots=main&action=query&titles=The Rocky Horror Picture Show&rvprop=content|ids|timestamp
03:59:36.388 [MW-1] INFO DumpAnalysis - Detection confirmed for The Rocky Horror Picture Show: 064 - Link equal to linktext
03:59:36.582 [MW-3] INFO API - GET  https://en.wikipedia.org/w/api.php?curtimestamp=1&continue=&prop=revisions|info&inprop=protection&format=xml&rvslots=main&action=query&titles=Romanian language&rvprop=content|ids|timestamp
03:59:36.888 [MW-3] INFO DumpAnalysis - Detection confirmed for Romanian language: 048 - Title linked in text
03:59:36.979 [MW-1] INFO API - GET  https://en.wikipedia.org/w/api.php?curtimestamp=1&continue=&prop=revisions|info&inprop=protection&format=xml&rvslots=main&action=query&titles=R&rvprop=content|ids|timestamp
03:59:37.170 [MW-1] INFO DumpAnalysis - Detection confirmed for R: 085 - Tags without content
03:59:37.184 [MW-1] INFO API - GET  https://en.wikipedia.org/w/api.php?curtimestamp=1&continue=&prop=revisions|info&inprop=protection&format=xml&rvslots=main&action=query&titles=Richard Matheson&rvprop=content|ids|timestamp
03:59:37.280 [MW-4] INFO API - GET  https://en.wikipedia.org/w/api.php?curtimestamp=1&continue=&prop=revisions|info&inprop=protection&format=xml&rvslots=main&action=query&titles=Racism&rvprop=content|ids|timestamp
03:59:37.281 [MW-1] INFO DumpAnalysis - Detection confirmed for Richard Matheson: 048 - Title linked in text
03:59:37.471 [MW-4] INFO DumpAnalysis - Detection confirmed for Racism: 064 - Link equal to linktext

Here's my jobs definition:

---
- name: wpcleaner-fr-daily
  command: /data/project/wpcleaner/tools/scripts/fr_Daily.sh
  image: tf-jdk17
  schedule: "@daily"
  emails: onfailure
  no-filelog: true
- name: wpcleaner-fr-weekly
  command: /data/project/wpcleaner/tools/scripts/fr_Weekly.sh
  image: tf-jdk17
  schedule: "@weekly"
  emails: onfailure
  no-filelog: true
- name: wpcleaner-fr-dab
  command: /data/project/wpcleaner/tools/scripts/fr_UpdateDabWarnings.sh
  image: tf-jdk17
  schedule: "@weekly"
  emails: onfailure
  no-filelog: true
  mem: 3Gi
  cpu: "1"
- name: wpcleaner-fr-list
  command: /data/project/wpcleaner/tools/scripts/fr_ListCheckWiki_List.sh
  image: tf-jdk17
  schedule: "@weekly"
  emails: onfailure
  no-filelog: true
  mem: 3Gi
  cpu: "1"

- name: wpcleaner-en-list
  command: /data/project/wpcleaner/tools/scripts/en_ListCheckWiki_List.sh
  image: tf-jdk17
  schedule: "@weekly"
  emails: onfailure
  no-filelog: true
  mem: 3Gi
  cpu: "1"

- name: wpcleaner-cs-weekly
  command: /data/project/wpcleaner/tools/scripts/cs_Weekly.sh
  image: tf-jdk17
  schedule: "@weekly"
  emails: onfailure
  no-filelog: true

- name: wpcleaner-meta-monthly
  command: /data/project/wpcleaner/tools/scripts/meta_Monthly.sh
  image: tf-jdk17
  schedule: "@monthly"
  emails: onfailure
  no-filelog: true
- name: wpcleaner-meta-list
  command: /data/project/wpcleaner/tools/scripts/meta_ListCheckWiki.sh
  image: tf-jdk17
  schedule: "@monthly"
  emails: onfailure
  no-filelog: true
  mem: 3Gi

In T414978#11596509, @fnegri wrote:

I decided to assign the open MRs about this issue to the task https://phabricator.wikimedia.org/T415322, they are more or less the same thing, so I wanted to reduce clutter

Ok, makes sense! There's also https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 that is still pointing to this task.

Before:

raymond-ndibe@cloudcontrol1006:~$ sudo wmcs-openstack quota show catalyst-dev
+-----------------------+-------+
| Resource              | Limit |
+-----------------------+-------+
| cores                 |    40 |
...
| ram                   | 81920 |
...
| gigabytes             |   750 |
...
+-----------------------+-------+

In T417426#11619396, @jnuche wrote:

@Raymond_Ndibe We use catalyst-dev to test changes to our infrastructure before rolling those changes out to the production project catalyst, so we generally need the same resources there.

@Otcenas11 I looked at your tool. Prev commands show you are running toolforge webservice --backend=kubernetes --mount=all buildservice start,
which is failing with the error below, because you have a manually created deployment that has the same name as the name webservice is trying to use:

tools.changedetection-io@tools-bastion-15:~$ toolforge webservice --backend=kubernetes --mount=all buildservice start
Traceback (most recent call last):
  File "/usr/bin/toolforge-webservice", line 33, in <module>
    sys.exit(load_entry_point('toolforge-webservice==0.103.18', 'console_scripts', 'toolforge-webservice')())
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/toolsws/cli/webservice.py", line 561, in main
    start(job, "Starting webservice")
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/toolsws/cli/webservice.py", line 88, in start
    job.request_start()
    ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py", line 647, in request_start
    self.api.create_object(
    ~~~~~~~~~~~~~~~~~~~~~~^
        "deployments", self._get_deployment(started_at)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/toolforge_weld/kubernetes.py", line 249, in create_object
    return self.post(
           ~~~~~~~~~^
        kind,
        ^^^^^
    ...<2 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/toolforge_weld/api_client.py", line 190, in post
    response = self._make_request("POST", url, **kwargs).json()
               ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/toolforge_weld/api_client.py", line 157, in _make_request
    raise e
  File "/usr/lib/python3/dist-packages/toolforge_weld/api_client.py", line 136, in _make_request
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/apis/apps/v1/namespaces/tool-changedetection-io/deployments
tools.changedetection-io@tools-bastion-15:~$ toolforge jobs list

tools.changedetection-io@tools-bastion-15:~$ kubectl get deployments
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
changedetection-io   1/1     1            1           7h15m
tools.changedetection-io@tools-bastion-15:~$

It seems like when an image with digest (e.g. '192.168.5.15/tool-tf-test/cluebot3:latest@sha256:b43fe64ac24365bd7cf3731f010e08020b6ce7304dd7852cd689600318ff270d') is provided while attempting to create job, what ends up in k8s is something like 192.168.5.15/tool-tf-test/cluebot3:latest, the digest is being dropped.
This is happening because we are running Image.from_url_or_name again here https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/main/tjf/runtimes/k8s/runtime.py?ref_type=heads#L191, which drops the digest.
I'm not sure why we are doing that again in that line of code, since we already ran it in the api/models.py. Will just remove it and see what happens.

@dancy I don't know much about catalyst, It'd be interesting to know why we are requesting similar increases as catalyst, for catalyst-dev?
to be clear the question above won't prevent increase if approved by the team, forwarding to irc right now. It's just personal curiosity.

@Otcenas11 Can you let us know the name of the toolforge tool you tried deploying this on? and maybe give more details about how to reproduce the issue you are facing? I can try to check out what the problem is. We can proceed with this (pending approval from others) if it's confirmed that this can't be deployed on toolforge

In T414978#11579359, @fnegri wrote:

@taavi sorry I involuntarily resolved this by dragging it in the Toolforge board that has an action attached to columns. :/

My previous question remains: @Raymond_Ndibe why was this closed as declined? The patch https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1115 is still open.