Per https://lists.wikimedia.org/pipermail/labs-l/2016-November/004777.html
A puppet role is needed for the TTS service (Speechoid or something), or the existing role::wikispeech should be enhanced to also include this, in order to set up TTS on wmflabs.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Add docker module | mediawiki/vagrant | master | +10 K -0 | |
| Create role for speechoid service | mediawiki/vagrant | master | +221 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T152430 Run Wikispeech offline | |||
| Resolved | HaraldBerthelsen | T143644 Multiple requests to TTS server should not cause delay | |||
| Resolved | Jopparn | T151786 Publically accessible demo (player) [Stage 1+2] | |||
| Declined | Lokal_Profil | T151877 Set up puppet role for TTS server | |||
| Resolved | Lokal_Profil | T161094 Set up non-puppet TTS server on wmflabs | |||
| Declined | None | T207293 Make import_all work in docker based vagrant role |
Created subtask for non-puppet implementation and decreased story points due to partial overlap.
So the TTS server can now be deployed via docker-compose which should simplify thinga but also means my previous work is largely deprecated.
But so far my attempts at having the docker-compose done via vagrant/puppet is foundering on vagrant not being able to provision docker-compose correctly. [Error: Could not find a suitable provider for docker_compose]
Part of the issue seems to be that existing docker (puppet) modules cannot be used out of the box since mediawiki-vagrant deploys its own version of e.g. apt.
I've now at least bumped stdlib to a compatible version in gerrit:427860 . This also allows for a change in the misc module preventing it from clashing with docker-compose on declaring the curl package gerrit:428316.
A bit more searching and I found how to enable debug on Puppet (via vagrant).
Debug: Puppet::Type::Docker_compose::ProviderRuby: file docker does not exist ... Error: Could not find a suitable provider for docker_compose
Some more digging seems to put the blame on apt not finding docker-ce in the repo. Despite it being listed in stretch-wikimedia/thirdparty/ci.
==> default: Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install docker-ce' returned 100: Reading package lists... ==> default: Building dependency tree... ==> default: Reading state information... ==> default: Package docker-ce is not available, but is referred to by another package. ==> default: This may mean that the package is missing, has been obsoleted, or ==> default: is only available from another source ==> default: ==> default: E: Package 'docker-ce' has no installation candidate
Replacing docker::repos by one purpose built for apt.wikimedia seems to do the trick.
The WIP gerrit:428320 gets docker (and docker-compose) provisioned correctly finally allowing for debugging of the implementation of the STTS compose files
Change 428320 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[mediawiki/vagrant@master] [WIP] Add docker module
End of day update:
Despite creating the .env file from a template successfully it seems as though docker-compose fails to read it
Error: Execution of '/usr/local/bin/docker-compose -f /vagrant/srv/speechoid/docker/wikispeech.yml up -d --abort-on-container-exit --remove-orphans' returned 1: The RELEASE variable is not set. Defaulting to a blank string. The LEXDATA_DIR variable is not set. Defaulting to a blank string.
There is a second error triggered by this:
--abort-on-container-exit and -d cannot be combined.
--abort-on-container-exit comes from STTS instructions, -d is hard coded into the docker_composer provider. Sadly there is no detach => false for docker::composer.
Temporarily commenting out -- takes us a step further resulting in the following error:
no such image: sttsse/pronlex:: invalid reference format
Here composer is relying on docker pull returning sttsse/pronlex which doesn't seem to be happening.
Per the docs this is likely due to docker-compose being run from another directory. Sadly there seems to be no way around that hard-coding yet.
Pre lunchtime update
I worked around the .env issue by templating wikispeech.yml directly providing it with the variables.
After commenting out --abort-on-container-exit and doing the above vagrant provision finally runs to completion! However none of the services seem to be running.
ssh:ing into the machine allows me to curl the ports opened by docker giving me a No route to host error. Running docker info shows three stopped containers
Need to investigate if the containers were stopped after successful creation or if they crashed/stoped even earlier.
re provisioning (only difference being setting socker log level to debug) and now magically sttsse/wikispeech and sttsse/marytts are running and only sttsse/pronlex is stopped (it's running when I ssh in but is up for less than a minute).
Digging into the logs gives :
2018/04/24 10:23:17 lexserver: parsed version info [Application name: pronlex Build timestamp: 2018-01-09 13:08:04 UTC Built by: docker Git release: v0.3.0beta2 Started: 2018-04-24 10:23:17 UTC] 2018/04/24 10:23:17 lexserver: started 2018/04/24 10:23:17 demo_setup: creating demo database ... 2018/04/24 10:23:17 demo_setup: deleting demo db: /wikispeech/appdir/db_files/lexserver_testdb.db COULDN'T INITIALISE DEMO DB : failed to define db: sql error : database is locked
@HannaLindgren I think I've now reached a point where it's something inside the pronlex docker image which is breaking. I've tracked it back to setupDemoDB
Change 428652 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[mediawiki/vagrant@master] [WIP] Create role for speechoid service
I just run our docker setup tests (from the wikispeech_compose repository), and they seem to work properly without errors.
I made some small changes in the pronlex source code just now, to clarify the error messages, but I'm fairly sure where the error occurs. I don't know why it happens, though. It could be caused by a previous error which hasn't been handeled correctly.
Is it possible to re-run the build with the master branch (not the release), and send me the full log file?
If not, do you have the complete log from your most recent build?
I'm running the docker-compose script with RELEASE=latest. The full log from the pronlex docker container (docker logs docker_pronlex_1) is:
I can't find any other log files.
Logging into the container I see that lexserver_testdb.db is a 0 byte file in case that is of any use.
OK. Is there any way we could run the same complete setup/commands here, to try to reproduce the error?
If you want the identical setup then mediawiki-Vagrant with https://gerrit.wikimedia.org/r/428652 checked out and the Wikispeech and speechoid roles activated is what I'm running.
Note that the os used in MediaWiki-Vagrant is Debian 9 (Stretch) which I spotted is not listed under tested OSes.
What the vagrant role does is in short:
* both these are run with docker-compose -f <yml_name> up -d --remove-orphans when digging in to how the docker module works.
A few questions and comments:
Another thing you could do if possible, start the pronlex docker image (docker exec or similar, I don't remember the details), and start lexserver without the test switch. It should look something like this (but chances are you need to adapt it to match your environment)
/wikispeech/bin/setup /wikispeech/appdir GOPATH=$(go env GOPATH) lexserver -ss_files /wikispeech/appdir/symbol_sets -db_files /wikispeech/appdir/db_files -static $GOPATH/src/github.com/stts-se/pronlex/lexserver/static 8787)
I just published a new Docker for sttsse/pronlex:latest. The functionality should be the same, but we improved some of the log/error messages, so if you run into this problem again with the new version, there may be more information in the log.
Make sure you get the latest version by running this command (or similar):
$ docker-compose --file wikispeech.yml pull
You can also verify the build timestamp by execing into the docker file and looking at the build info file:
$ docker run -ti sttsse/pronlex:latest bash
# cat /wikispeech/.pronlex_build_info.txt
Application name: pronlex
Build timestamp: 2018-04-27 09:20:49 UTC
Built by: docker
Git release: v0.3.0beta4-9-g89a40a5
@HannaLindgren Thanks for the feedback earlier and appologies for not following up before.
Sadly the docker_compose puppet class does not support the pull_on_start argument meaning old images need to be nuked for the new ones to be pulled.
For future debuging (ssh into vagrant machine, stop all docker containers that show up at docker ps, remove the containers: docker container prune, list all images docker image ls, remove each docker image rm <image name>. Reprovision vagrant.
Now running
and getting the following log
Did you check out the open patch set for implementing wikispeech? If not you need to do git fetch https://gerrit.wikimedia.org/r/mediawiki/vagrant refs/changes/52/428652/5 && git checkout FETCH_HEAD.
Then run vagrant roles enable speechoid wikispeech before running vagrant provision.
I believe I tested using those instructions, including that patch, yes.
I realise I had a draft comment here that apparently hasn't been published yet. I meant to publish this in late May, I think:
We did an update in the lexserver (lexserver.go and setupDemoDB.go), regarding db closing. It's possible that this could have to do with your locking problem. (But I'm not sure, since we haven't seen the same error in our setups.)
It's in the pronlex master branch 97221a5.
Would it help if I published a new docker image for the pronlex:latest Docker tag?
If you think branch 97221a5 might help then lets give a try with a new docker image.
I might also ask @Sebastian_Berlin-WMSE to try installing the Vagrant role to see if its somehow my Vagrant that is acting up.
Would it help if I published a new docker image for the pronlex:latest Docker tag?
If you think branch 97221a5 might help then lets give a try with a new docker image.
I have updated the docker images tagged latest. I'm not sure it will help but as I said, we updated some locking code a while back, so I think it's worth a try.
I checked out the commit and enabled the roles, but got an error (below) from vagrant provision. I'm running vagrant version 1.9.0.
That should have been fixed in https://gerrit.wikimedia.org/r/#/c/mediawiki/vagrant/+/428316/ which is part of the dependency train you should get when checking out https://gerrit.wikimedia.org/r/#/c/mediawiki/vagrant/+/428652/
Thanks. Got the new image up and running after updating the yml files. Pronlex is still crashing though with the following log.
Pronlex build_info (/wikispeech/pronlex/build_info.txt)
docker_pronlex_1 Application name: pronlex Build timestamp: 2018-09-07 08:07:36 UTC Built by: docker Release: master
OK, sorry to hear that. A few thoughts:
Clean database environment?
Maybe I asked this before, but did you run the lexicon server on a clean environment (from scratch)? The log indicates that there is already a database present ("No setup needed"). Is there any way you could ls the files in the database folder before you run the pronlex command?
What pronlex command are you running? Is that the full log output?
Reproducing the problem
As we haven't been able to set up the same environment here to reproduce this issue, maybe you could show us this live on-site, and/or is there any way we could log in to your environment online to be able to debug this more efficiently?
Separate task?
Just a suggestion: Is it a good idea to set up a separate task for this issue?
For archive happiness. I had an offline sit down with Nikolaj to brainstorm a bit about what was occurring.
One suggestion was upgrading go-sqlite3 from 1.2 to 1.9 in case our issue is somehow related to either of #274 or #607.
I've also rebased my patch train onto the latest MediaWiki-vagrant and updated the speechoid patch to include the yml updates.
I created a new Docker image for sttsse/pronlex:latest (built today 2018-10-01).
It has go-sqlite3 v1.9 and we also did some small changes to database locking.
Thanks. This changed the log to
whoami root [setup] Docker folder /wikispeech/pronlex/appdir is already configured. No setup needed. 2018/10/01 15:17:38 lexserver: parsed version info [Application name: pronlex Build timestamp: 2018-10-01 13:59:57 UTC Built by: docker Release: undefined Started: 2018-10-01 15:17:38 UTC] 2018/10/01 15:17:38 lexserver: started 2018/10/01 15:17:38 demo_setup: creating demo database ... 2018/10/01 15:17:38 demo_setup: deleting db file: /wikispeech/pronlex/appdir/db_files/lexserver_testdb.db 2018/10/01 15:23:44 COULDN'T INITIALISE DEMO DB : failed to define db /wikispeech/pronlex/appdir/db_files/lexserver_testdb.db | lexserver_testdb:sv : DBManager.DefineSqliteDB: failed to open db : DBManager.OpenDB: failed to set foreign keys : database is locked
Hm, OK. I think it would still be good if we could reproduce this in the same kind of environment that you are running, or that we come by your office to review this together.
A few thoughts:
Clean database environment?
Did you run the lexicon server on a clean environment (from scratch)? The log indicates that there is already a database present ("No setup needed"). Is there any way you could ls the files in the database folder before you start the docker-compose server?
You could also check for open connections before you run the failing command:
$ lsof <APPDIR>/db_files/lexserver_testdb.db
Run the lexserver as a standalone docker environment (without mounting external disk)
You could try running the lexserver as a standalone image (i.e., without mounting external disk):
docker run -ti sttsse/pronlex:latest
If there is some kind of issue with externally mounting files, this command could work better than the full docker-compose (which I assume you are running when you get the error?)
Binary lexserver (without docker)
Instead of running the lexserver inside docker, you can create your own binaries to include in your puppet:
If you want us to provide the binaries, let me know and we'll do that.
I completely deleted the mounted directory (/vagrant/srv/speechoid when ssh:ed into vagrant machine) and the containers. Pronlex still fails to start but the log message is changed to:
whoami root
[setup] Setting up basic files in docker folder /wikispeech/pronlex/appdir...
2018/10/10 08:50:24 database is locked
[setup] couldn't create empty db in docker: /wikispeech/pronlex/appdir/db_files/wikispeech_testdb.db
Note that lexserver_testdb.db is never created as the process crashes on wikispeech_testdb.db
You could also check for open connections before you run the failing command:
$ lsof <APPDIR>/db_files/lexserver_testdb.db
I cannot run this before the file exits. If I try to run it on (the empty) wikispeech_testdb.db i get an assortment of warnings about stat() not being able to deal with various file systems but no actual output.
Run the lexserver as a standalone docker environment (without mounting external disk)
You could try running the lexserver as a standalone image (i.e., without mounting external disk):
docker run -ti sttsse/pronlex:latest
If there is some kind of issue with externally mounting files, this command could work better than the full docker-compose (which I assume you are running when you get the error?)
Currently I'm running the docker-compose (through vagrant). If I just run the standalone image (ssh:ed into the vagrant machine) it spins up ok. Problem is of course that the wikispeech service can't detect it.
I changed the command to docker run -ti -p 8787:8787/tcp --network docker_wikispeech sttsse/pronlex:latest and that makes it works!
So it seems as though to problems are stemming from the mounting.
Binary lexserver (without docker)
Instead of running the lexserver inside docker, you can create your own binaries to include in your puppet:
I've started investigating this. If we abandon the docker path then it's actually the marytts service which I'm more worried about from looking at the Dockerfile.
Did you try to run the pronlex image standalone like you did above, but also mounting the file system? (so that any changes can be saved to disk between sessions)
Maybe it could also provide some clues about the mounting/file system issues.
docker run -ti -p 8787:8787/tcp -v /tmp/wikispeech/appdir:/wikispeech/appdir --network docker_wikispeech sttsse/pronlex:latest
sudo docker run -ti -p 8787:8787/tcp -v /vagrant/srv/speechoid/docker:/wikispeech/pronlex/appdir --network docker_wikispeech sttsse/pronlex:latest
whoami root
[setup] Setting up basic files in docker folder /wikispeech/pronlex/appdir...
2018/10/10 15:23:40 database is locked
[setup] couldn't create empty db in docker: /wikispeech/pronlex/appdir/db_files/wikispeech_testdb.db
as before there is an empty wikispeech_testdb.db left behind after running this.
Just a note that with pronlex running unmounted I have now successfully managed to run the entire Wikispeech package in Vagrant with the Wikispeech extension communicating successfully with the TTS server. I had to make a small patch to the repo to get the last step working.
By dropping the volume mounting on the pronlex server I've managed to get a docker based Vagrant role which successfully spins up and retains lexicon changes across vagrant halt/up. There is a known issue with the docker containers not restarting automatically after a vagrant halt. This can be solved by either running vagrant reprovision (deletes any additions to the lexicon) or by running vagrant up followed by docker restart docker_marytts_1 docker_pronlex_1 docker_wikispeech_1 in the vagrant ssh shell.
@Sebastian_Berlin-WMSE will test installing this on his machine to make sure it doesn't just work for me. If it works for him as well I'd consider this task resolved.
Since this implementation relies on also installing a docker provider I do not expect that it will be merged into the MediaWiki-vagrant repo (correct me if I'm wrong @bd808). I'll work on that in T207023.
@HannaLindgren Is Arabic not included in the Docker images? http://localhost:10000/languages gives me ["sv", "nb", "en"]. Diving in to it a bit further I see that "ar" is missing in http://localhost:10000/textprocessing/languages.
I do not have a reason to permanently block the use of Docker inside MediaWiki-Vagrant, so if this role ends up working for multiple people it can be merged. I would encourage continued work towards a MediaWiki-Vagrant integration that removes the need for Docker and Docker Compose to make development, review, and live debugging easier.
@Lokal_Profil Arabic is only enabled if the lexicons are imported properly. I will make an update so that Arabic is also enabled with a demo lexicon (as with Swedish, Norwegian and US English). It is pushed to master, and it will be included in the next release.
Hum. I thought I'd set it up so that it would also run the lexicon importer. Will have to take a look at why that didn't run.
OK, maybe I was to fast to assume it had to do with the import (since that's how I first was able to reproduce this). Let me know if this was run with or without the import, and if you still have no Arabic after the import, I will look into this again!
I managed to get it up and running.
There were initially some problem with Pronlex not starting properly, but that seemed to have solved itself; I've managed to restart Vagrant and get the server up and running using the method from T151877#4666290.
Looks like import_all was failing for other reasons. I'm tracking that in T207293. for the sake of the current task it's fine with an arabic demo lexicon.
Let me know when there is a new release. At that point I'll update the patch to make use of that release (instead of "latest") and if everything runs fine then we can consider this task solved.
Thanks for the clarification. We might then use this as a staging point and remove docker once T207023: Set up Docker-less puppet role for TTS server is done .
Many thanks. With this I now get Arabic up and running when I run without import_all
While there is a patch set functioning puppet role this relies on also installing a role for docker (note that the patch set will likely need a re-base by now). While having the docker dependency is not a strict blocker (T151877#4666916) there is definitely a push to make a new version of the puppet role which does not rely on docker. The work on this T207023: Set up Docker-less puppet role for TTS server/gerrit:469883 is at an extremely early stage.
For closing/resolving this task it would make sense to have the docker-based patch set merged.
We are now using Blubber internally. Unclear if a way of running the service from Vagrant is needed, if so it should probably start from the Blubber files.
Change 428652 abandoned by Lokal Profil:
Create role for speechoid service
Reason:
The service(es) can now be created via Blubber which seems more sane.
Change 428320 abandoned by Lokal Profil:
Add docker module
Reason:
Abandoned as the patch requiring this (428652) has been abandoned
Will no longer use puppet for the server. If a need for this arises again in the future then it should start from the Blubber results