In T354659, we did exploratory work showing that it is feasible to build a single model that works for many (>50) or even all languages.
However, the current architecture of the model is not suitable to easily maintain such a model; for example, the training pipeline using a bashscript is not suitable to automate the creation/updating of multilingual models.

In this task, we want to improve the pipelines for training and inference.

• developing an end-to-end training pipeline for the multilingual model; ideally with airflow
• using the pipeline to train one or few (>=5) models for all languages (i.e. figure out the best way to group languages)
• revise the inference pipeline for the new model (keeping in mind the requirements to serve it via LiftWing later on)