Page MenuHomePhabricator

Make import_all work in docker based vagrant role
Closed, DeclinedPublic

Description

T151877#4671118 indicated that the import_all docker_compose might not have been executed sucessfully.

Investigate why and try to get it up and running.

Event Timeline

Lokal_Profil created this task.
Lokal_Profil removed Lokal_Profil as the assignee of this task.EditedOct 17 2018, 3:37 PM

@HannaLindgren Is Arabic not included in the Docker images? http://localhost:10000/languages gives me ["sv", "nb", "en"]. Diving in to it a bit further I see that "ar" is missing in http://localhost:10000/textprocessing/languages.

@Lokal_Profil Arabic is only enabled if the lexicons are imported properly. I will make an update so that Arabic is also enabled with a demo lexicon (as with Swedish, Norwegian and US English). It is pushed to master, and it will be included in the next release.

Hum. I thought I'd set it up so that it would also run the lexicon importer. Will have to take a look at why that didn't run.

OK, maybe I was to fast to assume it had to do with the import (since that's how I first was able to reproduce this). Let me know if this was run with or without the import, and if you still have no Arabic after the import, I will look into this again!

Looks like import_all fails silently. I executed import_all manually from within the container and spotted that for RELEASE=latest (as passed in pronlex-import-all.yml) I of course get an error saying Could not find remote branch latest to clone.. So the environment file will likely need two separate RELEASE variables or some logic is needed to convert "latest". (e.g. by changing https://github.com/stts-se/pronlex/blob/7e137f746e38d467fad313e9059685ad58d54dc4/docker/import_all#L70 to append /releases/latest to the url and dropping the --branch flag).

If I instead use "master" I get the following error/log

[import_all] APPDIR=/wikispeech/pronlex/appdir
[import_all] LOCALDIR=/vagrant/srv/speechoid/docker
[import_all] RELEASETAG=master
whoami root
[setup] Docker folder /wikispeech/pronlex/appdir is already configured. No setup needed.
[import_all] Setup completed
Cloning into '/wikispeech/pronlex/appdir/lexdata.git'...
remote: Enumerating objects: 102, done.
remote: Counting objects: 100% (102/102), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 584 (delta 93), reused 96 (delta 93), pack-reused 482
Receiving objects: 100% (584/584), 271.03 MiB | 3.46 MiB/s, done.
Resolving deltas: 100% (274/274), done.

IMPORT: sv_se_nst_lex.db
2018/10/17 15:24:07 Input file: /wikispeech/pronlex/appdir/lexdata.git/sv-se/nst/swe030224NST.pron-ws.utf8.sql.gz
2018/10/17 15:24:07 Output db: /wikispeech/pronlex/appdir/db_files/sv_se_nst_lex.db
Error: near line 3709835: database or disk is full
Error: near line 3709836: no such table: main.Entry
Error: near line 3709837: no such table: main.Entry
Error: near line 3709838: no such table: main.EntryTag
Error: near line 3709839: no such table: main.EntryTag
Error: near line 3709840: no such table: main.entryTag
Error: near line 3709844: no such table: main.entryTag
Error: near line 3709848: no such table: main.EntryValidation
Error: near line 3709849: no such table: main.EntryValidation
Error: near line 3709850: no such table: main.EntryValidation
Error: near line 3709851: no such table: main.EntryValidation
Error: near line 3709852: no such table: main.EntryStatus
Error: near line 3709853: no such table: main.EntryStatus
Error: near line 3709854: no such table: main.EntryStatus
Error: near line 3709855: no such table: main.EntryStatus
Error: near line 3709856: no such table: main.EntryStatus
Error: near line 3709857: no such table: main.EntryStatus
Error: near line 3709858: no such table: main.EntryStatus
Error: near line 3709859: no such table: main.EntryStatus
Error: near line 3709860: no such table: main.Transcription
Error: near line 3709861: no such table: main.Transcription
Error: near line 3709862: no such table: main.Lemma2Entry
Error: near line 3709863: no such table: main.Lemma2Entry
Error: near line 3709864: no such table: main.Lemma2Entry
Error: near line 3709865: no such table: main.ENTRY
Error: near line 3709869: no such table: main.ENTRY
Error: near line 3709873: no such table: main.ENTRYSTATUS
Error: near line 3709877: no such table: main.ENTRYSTATUS
Error: near line 3709881: cannot commit - no transaction is active
2018/10/17 15:25:47 Couldn't load sql dump /wikispeech/pronlex/appdir/lexdata.git/sv-se/nst/swe030224NST.pron-ws.utf8.sql.gz into db /wikispeech/pronlex/appdir/db_files/sv_se_nst_lex.db : exit status 1
sv_se_nst_lex.db FAILED

I don't understand how you get the error log above with RELEASE set to master (you should get an error message saying there is no image sttsse/pronlex:master). But that's not really the point here...

I have updated the yml files for wikispeech, so that you set the lexdata release separately (LEXDATA_RELEASE) from the main wikispeech release specified as RELEASE . Please note that the lexdata release tag is now required if you want to run the full import.

To test the updated latestrelease:

  1. git pull the latest version of wikispeech_compose:master
  2. Add LEXDATA_RELEASE=master to your .env file

I don't understand how you get the error log above with RELEASE set to master (you should get an error message saying there is no image sttsse/pronlex:master).

Sorry for being unclear. I used master from within the docker container which had alread been set up using the latest tag.
Thanks for adding a separate lexdata tag. I'll give it a try when I spin up the new image and give an update here.

I can't get this to install the extra lexicon on the Vagrant machine but I believe this is due to us no longer mounting the pronlex volume and the imported lexicon then either get stored in the wrong place, or they get overwritten when the demo lexicon are created.

I believe we can leave this as is for now the current docker based role is good enough (has functioning languages). The way the docker module is set up in Vagrant does not allow us to run docker-compose --file wikispeech.yml down so it will always be wildly different from how the docker images were meant to be run.

The fix with the two Release tags is valid independently of the Vagrant implementation.