Update

• Incorporate the code in ml-piplines
• Decouple model and data from the image
• Achieve it to run it locally using these changes:
  • Use base: python:3.11-slim
  • Use the actual model (locally)
  • Build the image
  • Mount the volumes from local machine
  • Run it successfully

Obstacles:

• Testing is getting more difficult when it comes to push it on repo and run ci/cd and kokkuri pipelines due to the absence of the volumes (data and base_model).
• Since the image will be slim and flat from data/models we need to decide how the gitlab-ci pipelines will run together attaching the needed volumes.
• I had issues on building the image using the amd-pytorch23 base:
  • This image is used the previous-example on the ml-pipelines successfully.
  • The previous example had the model and the data included in the container (using small model and toy data).
• Both fixes made for running it locally, they will not be part of the actual code:
  • Use python:3.11-slim base image
  • Install torch on the top

The current status of the code can be found in this brach.

Test it locally:

1. Clone branch:

git clone -b develop_retraining_notebook_tonecheck https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines.git && cd ml-pipelines

2. Build image:

docker build --target production -f .pipeline/training/tone_check/retrain/blubber.yaml -t retrain:slim .

3. Download data and model:
   • model_link
   • data_link

4. Locate the above into: ml-pipelines/training/tone_check/data and ml-pipelines/training/tone_check/base_model. Create an empty ml-pipelines/training/tone_check/output/ folder as well.

5. Make sure you are in the correct folder when you run the image:

$ pwd
# ml-pipelines/training/tone_check

$ ls -l base_model 
# Output
config.json
model.safetensors
optimizer.pt
rng_state.pth
scheduler.pt
special_tokens_map.json
tokenizer_config.json
tokenizer.json
trainer_state.json
training_args.bin
vocab.txt

$ ls -l data      
# Output
peacock_detection_dataset.csv

6. Run the container and mount the volumes:

docker run --rm \
  -v $(pwd)/data:/srv/edit_check/training/tone_check/data \
  -v $(pwd)/base_model:/srv/edit_check/training/tone_check/base_model \
  -v $(pwd)/output:/srv/edit_check/training/tone_check/output \
  retrain:slim

Output:

Update

Achieved to use base image: docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-3-20250511 and publish it on docker-registry.wikimedia.org/repos/machine-learning/ml-pipelines .

The problem I was facing on publishing the image before was probably that I was using a python:3.11-slim for base image.

I could not test this configuration locally using the docker-registry.wikimedia.org/amd-pytorch23:2.3.0rocm6.0-3-20250511 base image due to the following error during docker run:

I could build the image locally but not run it.

Nevertheless, both test and production variant ran and published successfully via the kokkuri pipeline.
You can find the results namely: kokkuri_test_pipeline and kokkuri_publish_pipeline.

Since this is done, we can move forward for applying the PVC volume when is available.

kevinbazira updated https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/44

tone-check: refactor retraining image and script for DAG integration

kevinbazira merged https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/44

tone-check: refactor retraining image and script for DAG integration