Just a question: With "devise a deployment method", do we mean a proposal/documentation, or a working implementation that can be tested locally?

The latter, though it may still include some documentation. It doesn't need to work locally if that complicate things, but I'd say that this task is done once we've deployed at least once.

I tested the build service flow locally using pack, as suggested in the Toolforge docs for building container images (locally). This is only a local test, the real deployment would use toolforge build start etc.

Steps:

• The local test confirms that the Procfile is detected and that the backend starts.

web: daphne -b 0.0.0.0 lingualibre.asgi:application 
migrate: python manage.py migrate

From what I could understand is that daphne is the server used to run an asgi django application, and lingualibre.asgi:application points to the application object defined in lingualibre/asgi.py. This seemed like a reasonable starting point for testing without adding a new dependency such as Gunicorn.

• I also added a .python-version file containing: 3.11. This was needed because the repository's existing Dockerfile uses Python 3.11.
• The frontend starts using npm run build:toolforge (defined in front-end/package.json)

However, there appears to be an unresolved issue related the built frontend assets in front-end/dist. When accessing the locally running service, the logs show:

172.17.0.1:38732 - - [04/Jun/2026:13:02:43] "GET /" 200 434 
172.17.0.1:38732 - - [04/Jun/2026:13:02:43] "GET /assets/index-DTkXbudN.js" 200 434 
172.17.0.1:38742 - - [04/Jun/2026:13:02:43] "GET /assets/index-BUg98RLF.css" 200 434

and in the browser console:

Loading module from “http://127.0.0.1:8000/assets/index-DTkXbudN.js” was blocked because of a disallowed MIME type (“text/html”). 127.0.0.1:8000 The resource from “http://127.0.0.1:8000/assets/index-BUg98RLF.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).

This results in a blank page.

This suggests that the built frontend assets are not being served correctly in the local build test. I don't really know if there will be a difference between the local pack test setup and the actual Toolforge runtime.

I think the reason the frontend doesn't work locally may be related to the note in the Toolforge documentation about local testing:

Testing locally is not fully supported, Toolforge injects some buildpacks and runs some fixes at runtime that are not available in the builder image.

Since this repository has previously been deployed successfully on Toolforge, do you think it's worth trying the actual Toolforge Build Service flow before spending more time debugging the local pack setup? @Sebastian_Berlin-WMSE
I'm wondering if the asset issue might be specific to the local test environment rather than the Toolforge runtime itself.

In T427909#11987598, @Viktoria_Hillerud_WMSE wrote:

Since this repository has previously been deployed successfully on Toolforge, do you think it's worth trying the actual Toolforge Build Service flow before spending more time debugging the local pack setup? @Sebastian_Berlin-WMSE

Yes, if you suspect that it may run on Toolforge I'd say hold off until we've tried that. It's a bit sad if it doesn't work locally. I don't know if we're likely to build images when we develop, but it would be nice to be able to run the exact same thing™ as on Toolforge locally when troubleshooting.

Yes, I unfortunately suspect that. So we can keep this task open until we start to deploy our Toolforge tool: Wikispeech-sdc: T427879

viktoriahillerudwmse updated https://gitlab.wikimedia.org/sebastian-berlin-wmse/speech-data-collector/-/merge_requests/2

Deployment method

sebastian-berlin-wmse merged https://gitlab.wikimedia.org/sebastian-berlin-wmse/speech-data-collector/-/merge_requests/2

Deployment method

You could try running it on Toolforge now that you have access to the tool. I tried locally myself and couldn't get it to work either.

I'm a bit suspicious about front-end/dist being included in the repo. I think this is something that we'd like to create as part of the build process.

Yes, I also had my suspicions about the /dist folder, and whether the local build issues might be related to that.

Do you mean that I should already try deploying it through our Toolforge tool as part of this task?

Yes. Otherwise we don't know that it works.

Ok. I thought I could use the fork on GitLab, but I noticed that the Procfile-branch was reverted. Maybe it could be merged again so that I can test this on our tool in Toolforge?

You can build from a branch. I think the parameter is --ref or some such.

I tried building on Toolforge, but it fails because ofcourse config.ini is gitignored and the build process runs manage.py before any runtime config can be set up.

I see a few options:

1. Temporarily commit a config.ini to the deployment-method branch just to get the first build working.
2. Refactor settings.py to read from environment variables and use toolforge envvars instead.

What do you think makes most sense?

I think 2 is the way to go.

A variation could be to add an environment variable for the path of the config. The default would be the path that is hardcoded now. That way you don't need to make everything into environment variables and can keep the structure of the config. Just be sure to make the config file unreadable for anyone that doesn't have access to the tool.

Sounds like a good plan! I also realized the same issue exists for .env.toolforge in the frontend, it's also gitignored, so the build will fail there too.

Since .env.toolforge doesn't contain any secrets (only CLIENT_ID, not CLIENT_SECRET), would it make sense to just remove it from .gitignore?

I don't think we should add any actual config files to the repo. These are likely to look different for every setup, sometimes with secrets that really shouldn't go in the repo. That's why the repo only contains sample files and why the actual config files are in .gitignore.

I tried setting CONFIG_PATH via toolforge envvars, but the build fails with the same error. Toolforge envvars seems not available during the build step, only at runtime..?

So even if doing:

config_path = os.environ.get("CONFIG_PATH", os.path.dirname(__file__) + "/../config.ini")
config.read(config_path)

it reads nothing and then crashes with error because the config is empty:

[step-build] 2026-06-10T12:07:56.007154334Z [Generating Django static files]
[step-build] 2026-06-10T12:07:56.209836795Z 
[step-build] 2026-06-10T12:07:56.209915916Z [Error: Unable to inspect Django configuration]
[step-build] 2026-06-10T12:07:56.209930255Z The 'python manage.py help collectstatic' Django management command
[step-build] 2026-06-10T12:07:56.209940497Z (used to check whether Django's static files feature is enabled)
[step-build] 2026-06-10T12:07:56.209951495Z failed (exit status: 1).
[step-build] 2026-06-10T12:07:56.209962400Z 
[step-build] 2026-06-10T12:07:56.209973448Z Details:
[step-build] 2026-06-10T12:07:56.209984362Z 
[step-build] 2026-06-10T12:07:56.209995231Z Traceback (most recent call last):
[step-build] 2026-06-10T12:07:56.210005201Z   File "/workspace/manage.py", line 22, in <module>
[step-build] 2026-06-10T12:07:56.210015154Z     main()
[step-build] 2026-06-10T12:07:56.210025478Z   File "/workspace/manage.py", line 18, in main
[step-build] 2026-06-10T12:07:56.210035374Z     execute_from_command_line(sys.argv)
[step-build] 2026-06-10T12:07:56.210045793Z   File "/layers/heroku_python/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
[step-build] 2026-06-10T12:07:56.210056466Z     utility.execute()
[step-build] 2026-06-10T12:07:56.210067178Z   File "/layers/heroku_python/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 382, in execute
[step-build] 2026-06-10T12:07:56.210077062Z     settings.INSTALLED_APPS
[step-build] 2026-06-10T12:07:56.210088274Z   File "/layers/heroku_python/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 89, in __getattr__
[step-build] 2026-06-10T12:07:56.210097952Z     self._setup(name)
[step-build] 2026-06-10T12:07:56.210107712Z   File "/layers/heroku_python/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 76, in _setup
[step-build] 2026-06-10T12:07:56.210117520Z     self._wrapped = Settings(settings_module)
[step-build] 2026-06-10T12:07:56.210127466Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^
[step-build] 2026-06-10T12:07:56.210137485Z   File "/layers/heroku_python/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 190, in __init__
[step-build] 2026-06-10T12:07:56.210147278Z     mod = importlib.import_module(self.SETTINGS_MODULE)
[step-build] 2026-06-10T12:07:56.210156838Z           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[step-build] 2026-06-10T12:07:56.210226854Z   File "/layers/heroku_python/python/lib/python3.11/importlib/__init__.py", line 126, in import_module
[step-build] 2026-06-10T12:07:56.210237359Z     return _bootstrap._gcd_import(name[level:], package, level)
[step-build] 2026-06-10T12:07:56.210246926Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[step-build] 2026-06-10T12:07:56.210256581Z   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
[step-build] 2026-06-10T12:07:56.210267169Z   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
[step-build] 2026-06-10T12:07:56.210276884Z   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
[step-build] 2026-06-10T12:07:56.210286753Z   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
[step-build] 2026-06-10T12:07:56.210296573Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
[step-build] 2026-06-10T12:07:56.210316645Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
[step-build] 2026-06-10T12:07:56.210326967Z   File "/workspace/lingualibre/settings.py", line 23, in <module>
[step-build] 2026-06-10T12:07:56.210337657Z     is_production = config.getboolean("deployment", "production")
[step-build] 2026-06-10T12:07:56.210352113Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[step-build] 2026-06-10T12:07:56.210367663Z   File "/layers/heroku_python/python/lib/python3.11/configparser.py", line 844, in getboolean
[step-build] 2026-06-10T12:07:56.210381745Z     return self._get_conv(section, option, self._convert_to_boolean,
[step-build] 2026-06-10T12:07:56.210395405Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[step-build] 2026-06-10T12:07:56.210408839Z   File "/layers/heroku_python/python/lib/python3.11/configparser.py", line 824, in _get_conv
[step-build] 2026-06-10T12:07:56.210421895Z     return self._get(section, conv, option, raw=raw, vars=vars,
[step-build] 2026-06-10T12:07:56.210436296Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[step-build] 2026-06-10T12:07:56.210450300Z   File "/layers/heroku_python/python/lib/python3.11/configparser.py", line 819, in _get
[step-build] 2026-06-10T12:07:56.210465259Z     return conv(self.get(section, option, **kwargs))
[step-build] 2026-06-10T12:07:56.210479104Z                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[step-build] 2026-06-10T12:07:56.210492458Z   File "/layers/heroku_python/python/lib/python3.11/configparser.py", line 797, in get
[step-build] 2026-06-10T12:07:56.210506007Z     d = self._unify_values(section, vars)
[step-build] 2026-06-10T12:07:56.210521443Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[step-build] 2026-06-10T12:07:56.210535261Z   File "/layers/heroku_python/python/lib/python3.11/configparser.py", line 1170, in _unify_values
[step-build] 2026-06-10T12:07:56.210549497Z     raise NoSectionError(section) from None
[step-build] 2026-06-10T12:07:56.210564177Z configparser.NoSectionError: No section: 'deployment'
[step-build] 2026-06-10T12:07:56.210579364Z 
[step-build] 2026-06-10T12:07:56.210593944Z 
[step-build] 2026-06-10T12:07:56.210609398Z This indicates there is a problem with your application code or Django
[step-build] 2026-06-10T12:07:56.210623254Z configuration. Try running the 'manage.py' script locally to see if the
[step-build] 2026-06-10T12:07:56.210638401Z same error occurs.

I think I managed now: https://wikispeech-sdc.toolforge.org/

The only thing left is the frontend. I will also explain more how I solved this when the frontend is fixed.

Some progress to report:

Backend: The build was failing because config.ini is gitignored and not available during the build step, and toolforge envvars are only injected at runtime. The fix was to make settings.py flexible when no config is found, by falling back to placeholder values:

config_path = os.environ.get("CONFIG_PATH", os.path.dirname(__file__) + "/../config.ini")
config.read(config_path)
if not config.sections():
    config.read_dict({
        'deployment': {'production': 'no', 'localhost': 'yes'},
        'security': {'django_secret_key': 'build-time-placeholder'},
        'database': {'host': '', 'port': '3306', 'name': '', 'user': '', 'password': ''},
        'WIKIMEDIA_OAUTH': {'CLIENT_ID': '', 'CLIENT_SECRET': ''},
        'storage': {'commons_url': 'https://commons.wikimedia.beta.wmcloud.org'},
    })

At runtime, CONFIG_PATH (set via toolforge envvars) points to the real config.ini on the server.

Frontend: According to the Toolforge docs, Node.js is installed alongside another language as long as there is a package.json in the root of the repository. Adding a package.json in the project root with a build script that runs npm run build:toolforge got the build running as part of the Toolforge build process.

However, I'm still hitting the same MIME type issue as in the local test. The browser gets HTML instead of JS/CSS when requesting /assets/. My suspicion is that Djangos catch-all URL pattern in lingualibre/urls.py is stopping the asset requests before they can be served as files (?). Even though this seems to have worked before on Toolforge, but this is the only thing I can think of being the issue right now.

The resource from “https://wikispeech-sdc.toolforge.org/assets/index-BUg98RLF.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). wikispeech-sdc.toolforge.org

I think it would be better to figure out what config values are actually needed when building. By the looks of your hardcoced config a lot of the values aren't relevant. If you can instead find where those are accessed and see if you can excluded that when building, I think that's a better approach.

Does it really work to build the frontend outside its directory? Are you sure that it not uses the pre-built files included in the repo?

In T427909#12008506, @Sebastian_Berlin-WMSE wrote:

Does it really work to build the frontend outside its directory? Are you sure that it not uses the pre-built files included in the repo?

git ls-files front-end/dist/ returns nothing, so no pre-built files are committed. The build logs confirm Vite ran fresh from the root package.json: ✓ 213 modules transformed. ✓ built in 8.64s

Yes, that's just me being stupid. Not sure why I thought the dist directory was in the repo 😵‍💫

Looking at toolforge-deployment.md, it gave some hints on LinguaLibre's previous deployment method, which used uWSGI to serve the frontend assets:

uWSGI is a high-performance application server that runs Python web applications. It helps serve static files efficiently in a production environment and ensures Django applications function smoothly on Toolforge.

Create a `python/uwsgi.ini` defining the path to the `/static` folder.

ini
[uwsgi]
check-static = /data/project/lingua-libre/www/python/src/
static-map = /=/data/project/lingua-libre/www/python/src/front-end/dist/
static-index = index.html

The Toolforge docs on migrating an existing tool also state: "If you're migrating a Python tool that uses uWSGI, replace it with Gunicorn."

However, we are using Daphne (ASGI), which is already a dependency of this project in requirements.txt. Daphne does not serve static files by itself, so we need another solution. WhiteNoise appears to be the standard way to handle this for Django apps without a separate web server. See Serving static files with Django - WhiteNoise and How to serve static files in Django with WhiteNoise.

This could be the way to go.

Any reason not to use Gunicorn if that's what mentioned in the documentation? It's also in the Django documentation.

No strong reason, actually! Daphne ended up in the Procfile because it was already in requirements.txt, but looking at it now, switching to Gunicorn as Toolforge and Django recommend makes sense. We would still need WhiteNoise (or similar) for serving the frontend assets, since Gunicorn also doesn't serve static files by itself.

Now it works, at least the visual part. When trying to log in with oAuth, this is the screen that renders:

I don't really know why it wants to send us to their project, any ideas why? @Sebastian_Berlin-WMSE

Edit: I realized it must have something to do with the OAuth consumer, that wikispeech-sdc isn't registered in Beta Commons, but this is probably out of the scope of this task right?

At least, it seems like the "Whitenoise" thing was the thing that fixed it!

In T427909#12009720, @Viktoria_Hillerud_WMSE wrote:

I don't really know why it wants to send us to their project, any ideas why? @Sebastian_Berlin-WMSE

If you're running off a non-main branch you mauy not have the changes from Make Commons URL in the backend configurable. That would explain why you're sent to non-beta Commons for authentication. You can rebase your branch on "main" to include those changes.

In T427909#12009890, @Sebastian_Berlin-WMSE wrote:

In T427909#12009720, @Viktoria_Hillerud_WMSE wrote:

I don't really know why it wants to send us to their project, any ideas why? @Sebastian_Berlin-WMSE

If you're running off a non-main branch you mauy not have the changes from Make Commons URL in the backend configurable. That would explain why you're sent to non-beta Commons for authentication. You can rebase your branch on "main" to include those changes.

I did rebase it already

You need to set the envvars for the frontend (VITE_...) when you build. I guess there's a fallback to Commons somewhere. I build it now with VITE_AUTHORIZATION_ENDPOINT and VITE_OAUTH_ACCESS_TOKEN which looks like it may be enough.

However, it still won't work for the with the Ouath consumer we have now. Its callback URL is on localhost. We need to create a new one for Toolforge.

Aha! So if I understand correctly, there are two things I can fix:

1. Pass VITE_AUTHORIZATION_ENDPOINT and VITE_OAUTH_ACCESS_TOKEN as build-time envvars when running toolforge build start, so Vite picks up the correct beta Commons URL during the frontend build.
2. Create a new OAuth consumer on beta Commons with https://wikispeech-sdc.toolforge.org/oauth/verify/wikimedia/ as the callback URL:

{F88092974}

3. Update config.ini on the server with the new credentials.

Is that right?

Yes. There may be more envvars for the frontend that are required. There were more in the .env file.

I also think it would be a good idea to create a script for building.

In T427909#12013193, @Sebastian_Berlin-WMSE wrote:

Yes. There may be more envvars for the frontend that are required. There were more in the .env file.

I also think it would be a good idea to create a script for building.

Smart, you mean a script for building the envars?

Just a bash script with the toolforge build start ... command so you don't need to rely on bash's history.

Now a OAuth consumer for Speech Data Collector on Beta commons is requested. It could take a while before it will get approved.

I also created a script in the root of the Toolforge tool called build-toolforge.sh containing the build script along with envars.
In the root of the tool is also the config.ini, where the new credentials for the OAuth consumer will be set.

viktoriahillerudwmse opened https://gitlab.wikimedia.org/repos/wmse/wikispeech/speech-data-collector/-/merge_requests/5

Devise a deployment method

viktoriahillerudwmse closed https://gitlab.wikimedia.org/repos/wmse/wikispeech/speech-data-collector/-/merge_requests/5

Devise a deployment method

viktoriahillerudwmse opened https://gitlab.wikimedia.org/repos/wmse/wikispeech/speech-data-collector/-/merge_requests/6

New deployment methods

Something went wrong when I tried to revive the old deployment-method branch, probably something in GitLab that I did not fully understand, but since it was merged, then reverted, I got changes from another branch as well as a part of the branch. I created this new branch and MR, since there is a lot of changes to the deployment anyways, so I don't think its a problem to have created a new branch for this.

viktoriahillerudwmse merged https://gitlab.wikimedia.org/repos/wmse/wikispeech/speech-data-collector/-/merge_requests/6

New deployment methods

sebastian-berlin-wmse opened https://gitlab.wikimedia.org/repos/wmse/wikispeech/speech-data-collector/-/merge_requests/12

Add "https://" to URLs

sebastian-berlin-wmse merged https://gitlab.wikimedia.org/repos/wmse/wikispeech/speech-data-collector/-/merge_requests/12

Add "https://" to URLs