Page MenuHomePhabricator

Migrate xiplus from Toolforge GridEngine to Toolforge Kubernetes
Closed, ResolvedPublic

Description

Moving tools running on Toolforge GridEngine to Kubernetes.
See: https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/
See: https://grid-deprecation.toolforge.org/t/xiplus

Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.

If you would rather shut down this tool, kindly do so and mark this as resolved.

Useful Resources:
Migrating Jobs from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engine_migration
Migrating Web Services from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move_a_grid_engine_webservice
Python
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Rebuild_virtualenv_for_python_users

Event Timeline

@Xiplus Hi! Have you had a look already to migrating this tool? Do you need any guidance?

Remember that we have a hard stop next week (14th of March).

Thanks!

I use cron to execute a logrotate job. The cron command is

13 5 * * * jsub /usr/sbin/logrotate -s /data/project/xiplus/logrotate-status /data/project/xiplus/logrotate.conf > /data/project/xiplus/cron.out

I tried rewriting it as a Kubernetes command:

toolforge jobs run test --command "/usr/sbin/logrotate -s /data/project/xiplus/logrotate-status /data/project/xiplus/logrotate.conf > /data/project/xiplus/cron.out" --image bullseye --wait

But i got an error:

/bin/sh: 1: /usr/sbin/logrotate: not found

Can you give me some guidance?

Change 1009264 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/docker-images/toollabs-images@master] bullseye-standalone: add logrotate

https://gerrit.wikimedia.org/r/1009264

I use cron to execute a logrotate job. The cron command is

13 5 * * * jsub /usr/sbin/logrotate -s /data/project/xiplus/logrotate-status /data/project/xiplus/logrotate.conf > /data/project/xiplus/cron.out

I tried rewriting it as a Kubernetes command:

toolforge jobs run test --command "/usr/sbin/logrotate -s /data/project/xiplus/logrotate-status /data/project/xiplus/logrotate.conf > /data/project/xiplus/cron.out" --image bullseye --wait

But i got an error:

/bin/sh: 1: /usr/sbin/logrotate: not found

Can you give me some guidance?

The issue is that the bullseye image does not come with logrotate installed, so it does not find it.

I think it would be ok to add it to the image (it's a very small binary). Let's see what the team thinks.

Actually (@taavi reminded me) we already have logrotate in one of our images, mariadb, so you cat try doing something like:

toolforge jobs run test --command '/usr/sbin/logrotate -s $TOOL_DATA_DIR/logrotate-status $TOOL_DATA_DIR/logrotate.conf > $TOOL_DATA_DIR/cron.out' --image mariadb --wait

Note that I changed the image, and used environment variables to get the tool data directory (for now is the same as before, but we might change it at some point, and this way you will not have to change anything if we do).

Can you try?

Change 1009264 abandoned by David Caro:

[operations/docker-images/toollabs-images@master] bullseye-standalone: add logrotate

Reason:

We already have logrotate in the mariadb image

https://gerrit.wikimedia.org/r/1009264

Xiplus claimed this task.

Thanks! I have migrated it.