Creating an instance

Follow instructions on https://wikitech.wikimedia.org/wiki/Help:Instances#Creating_an_instance

1. Go to horizon.wikimedia.org and go to the wikispeech project.
2. Compute → Overview and check that there are instances available.
3. Create security groups:
   • Compute → Access & security → Create security group, name TTS-consumer
   • Compute → Access & security → Create security group, name TTS-provider
     • Add rule: port = 10000, Remote = Security group, Security Group = TTS-consumer
4. In launch instance dialogue:
   1. Details: only set name (wikispeech-tts)
   2. Source: Select boot source = image
   3. Under available choose image (ubuntu)
   4. Flavor: Choose flavor (m1.medium, marytts uses ~ 1.1 GB on local machine)
   5. Security groups: choose security group (default, tts-provider)
5. Compute → Instances → Dropdown for wikispeech-wiki → Edit Security Groups: add tts-consumer

For ssh'ing: https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Project_Instances

Install TTS server

Install MaryTTS

1. Log into wikispeech-tts
2. Install java: sudo apt install openjdk-7-jdk
3. Create a user to run the server: sudo useradd -m tts-agent -p <password>.
4. Become this user using sudo su - tts-agent
5. Clone marytts-installer repo from Home
6. Follow instructions to install English voices
7. Download needed STTS-voices from https://github.com/HaraldBerthelsen/marytts/tree/master/stts_voices into /installed/ (e.g. voice-stts_sv_nst-hsmm-5.2-SNAPSHOT.jar)
8. Install marrytts-lang-sv:
   1. Clone the forked maryTTS repo into stts_marytts
   2. Run ./gradlew build
   3. Copy Swedish language into marytts (cp build/install/marytts/lib/marytts-lang-sv-6.0-SNAPSHOT.jar ../marytts-installer/installed/
   4. Delete stts_marytts
9. To start marytts run: ./marytts

Install pronlex

Following instruction at https://github.com/stts-se/lexdata/wiki/Create-lexicon-database

1. Install go:
   1. Get teh compatible version (1.7.4)
   2. Install using tar -C /usr/local -xzf go1.7.4.linux-amd64.tar.gz
   3. Add export PATH=$PATH:/usr/local/go/bin to .bashrc
2. Install Sqlite3: per instructions
3. "Clone the source code": Follow instructions but use https://github.com/stts-se/pronlex.git
4. "Clone the lexdata repository": Follow instructions but use https://github.com/stts-se/lexdata.git
5. "Prepare symbol set files": Standing in Pronlex repo (not gitrepos). Follow instructions
6. "Create an empty database":
   1. Install gcc: sudo apt-get install gcc
   2. Install build-essential: `sudo apt-get install build-essential
   3. Install go-sqlite3: go get github.com/mattn/go-sqlite3
   4. Install regexp2: go get github.com/dlclark/regexp2
   5. Follow instructions.
7. Start server: per instructions
8. "Import lexicon files": Instruction require GUI so instead
   1. ~/go/src/github.com/stts-se/pronlex$ go run lexio/import/import.go lexserver/pronlex.db sv-se.nst ~/gitrepos/lexdata/sv-se/nst/swe030224NST.pron-ws.utf8.gz sv-se_ws-sampa

Install wikispeech-mockup

Following instruction at https://github.com/stts-se/wikispeech_mockup

1. Install the Prerequisites but skip the remaining steps
2. Install opusenc: sudo apt install opus-tools.
3. Clone https://github.com/stts-se/wikispeech_mockup.git from Home
4. Create tmp directory in wikispeech_mockup: mkdir tmp
5. Add host=0.0.0.0 to app.run() in wikispeech.py (see: http://stackoverflow.com/questions/30554702/cant-connect-to-flask-web-service-connection-refused).
6. Start wikispeech server: python3 wikispeech.py

wikispeech_mockup also requires opusenc: sudo apt install opus-tools.

In T161094#3125929, @Sebastian_Berlin-WMSE wrote:

wikispeech_mockup also requires opusenc: sudo apt install opus-tools.

Added above

In T161094#3125200, @Sebastian_Berlin-WMSE wrote:

Creating an instance

• Compute → Access & scurity → Create security group, name TTS-provider
  • Add rule: port = 10000, Remote = Security group, Security Group = TTS-consumer

While this is the way we would probably want to do it it does not work since the request is client side. So instead it needs to be opened up for everyone:

• Add rule: port = 10000, Remote = CIDR, CIDR = 0.0.0.0/0
• DNS → Web proxies → Create proxy
  • Hostname: wikispeech-tts, Backend instance wikispeech-tts, Backend port: 10000

To make audio files accessible:

1. Install apache: sudo apt install apache2
2. Link the audio file directory in /var/www/html: sudo ln -s ~/wikispeech_mockup/tmp/ audio
3. Add proxy: Hostname: wikispeech-tts-audio, Backend instance wikispeech-tts, Backend port: 80

Audio files generated by the TTS should now be accessible through: http://wikispeech-tts-audio.wmflabs.org/audio/.

• Disallow access to non-opus files by adding a .htaccess file to /audio/ with the following content

<FilesMatch "\.*$">
  Deny from all
</FilesMatch>
<FilesMatch "\.opus$">
  Order deny,allow
  Allow from all
</FilesMatch>

• Remove directory listing by adding a index.html file to /audio/ with something like the following content:

This is a service to serve Wikispeech audio files. There is really never any reason to see this page.

• To also display this page in the root directory: sudo rm index.html;sudo ln -s audio/index.html

Note that adding Options -Indexes to the top of .htaccess did not prevent the directory listing.

Update the server with the new path:

1. In wikispeech_mockup/wikispeech.py, change:
   • In synthesise()

-    audio_url = "%s/wikispeech_mockup/%s" % (hostname,opus_audio)
+    audio_url = "//wikispeech-tts-audio.wmflabs.org/audio/%s" % (opus_audio)

• In saveAndConvertAudio()

-    opus_url_suffix = re.sub("^.*/%s/" % tmpdir, "%s/" % tmpdir, tmpopus)
+    opus_url_suffix = re.sub("^.*/%s/" % tmpdir, "", tmpopus)

@Sebastian_Berlin-WMSE Add the following to "To make audio files accessible:"

• Disallow access to non-opus files by adding a .htaccess file to /audio/ with the following content

<FilesMatch "\.*$">
  Deny from all
</FilesMatch>
<FilesMatch "\.opus$">
  Order deny,allow
  Allow from all
</FilesMatch>

• Remove directory listing by adding a index.html file to /audio/ with something like the following content:

This is a service to serve Wikispeech audio files. There is really never any reason to see this page.

• To also display this page in the root directory: sudo rm index.html;sudo ln -s audio/index.html

Note that adding Options -Indexes to the top of .htaccess did not prevent the directory listing.

Start processes in screen

As user tts-agent:

1. script /dev/null, needed to get screen running (http://serverfault.com/a/116830)
2. Start MaryTTS
   1. $ cd ~/marytts-installer/
   2. $ ./marytts
3. Create new window: ⌃+a, w
4. Start pronlex
   1. $ cd ~/go/src/github.com/stts-se/pronlex/lexserver/
   2. $ go run *.go
5. Create new window
6. Start wikispeech.py
   1. $ cd ~/wikispeech_mockup/
   2. $ python3 wikispeech.py

You can now detach the screen ⌃+a, d and resume with $ screen -r.

@Lokal_Profil: I've compiled the documentation from the comments in the description. Please have a read through and fix anything that's unclear or if we ended up doing things some other way.

Note for the future. While the audio needs to be publicly accessible it could be possible to make the main tts server only speak to the wiki by having client requests go to the action api (of the MediaWiki extension) which in turn forwards these to he TTS service and handles the reply. If so the Security group settings are those in T161094#3125200.

The question which such an approach is the trade-of between any performance losses (due to the api go-between) and potential security gains. This is worth revisiting after T153841 which will likely introduce/require some of the action api functionality anyway.

For "Install pronlex", I have updated the instructions on https://github.com/stts-se/lexdata/wiki/Create-lexicon-database, so all the special exceptions/instructions should be deleted I think. Please have a look, and see if you agree with me.

Your more detailed instructions for installing go and sqlite3 should probably not be deleted though.

Please note that we support go 1.8. I will update the wiki.

As for the pronlex prerequisites (gcc and build-essentials), we didn't notice these dependencies, probably since we already had these packages installed.
Should we add this also to the instructions on our Wiki?

Thanks for the update. Everything seems to be working.

I think it's best to mention that gcc and build-essentials are required. Even if these may often be installed, it's good to have for when they're not.

In T161094#3136863, @Sebastian_Berlin-WMSE wrote:

Thanks for the update. Everything seems to be working.

I think it's best to mention that gcc and build-essentials are required. Even if these may often be installed, it's good to have for when they're not.

So maybe add these to our instructions? Better than to have it in separate places?
But apt install is not a universal solution either though.... This is more what works on your server I guess.

For Arabic, a few extra tricks are needed at the moment:

1. Make sure Java 8 is installed
   1. For Ubuntu 14.04, follow http://askubuntu.com/a/464894 (under "Original Message")

Install Arabic resources for MaryTTS

1. cd marytts-installer
2. cd installed
3. wget https://github.com/HaraldBerthelsen/marytts-lang-ar/releases/download/v0.1-alpha/marytts-lang-ar-6.0-SNAPSHOT.jar
4. wget https://github.com/HaraldBerthelsen/voice-ar-nah-hsmm/releases/download/v5.2/voice-ar-nah-hsmm-5.2.jar
5. cd ..
6. ./marytts

Install Msihkal

git clone https://github.com/linuxscout/mishkal.git
cd mishkal
EDIT mishkal/tashkeel/tashkeel.py
CHANGE line 385
from
vocalized_text = u" ".join([vocalized_text, self.display(word, format_display)])
to
vocalized_text = u" ".join([vocalized_text, self.display(voc_word, format_display)])

(The variable "word" is wrong, should be "voc_word")

Run webserver with:
python interfaces/web/mishkal-webserver.py

AND:

In marytts-installer-5.2 (or wherever your marytts-installer is):

cd installed
wget http://repo1.maven.org/maven2/org/glassfish/javax.json/1.0.4/javax.json-1.0.4.jar
cd ..
./marytts