Workaround: use lighttpd with fcgi.

pip install flipflop

create ~/www/python/src/app.fcgi:

#!/data/project/fc-importer/www/python/venv/bin/python
from flipflop import WSGIServer
from app import app

from flask import request

import time
import logging
from logging import FileHandler
logger = FileHandler('error.log')
app.logger.setLevel(logging.DEBUG)
app.logger.addHandler(logger)
app.logger.debug(u"Flask server started " + time.asctime())


@app.after_request
def write_access_log(response):
    app.logger.debug(u"%s %s -> %s" % (time.asctime(), request.path, response.status_code))
    return response

if __name__ == '__main__':
    WSGIServer(app).run()

(change fc-importer into your tools name)

create ~/.lighttpd.conf:

fastcgi.server += ( "/fc-importer" =>
    ((
        "socket" => "/tmp/fc-importer-fcgi.sock",
        "bin-path" => "/data/project/fc-importer/www/python/src/app.fcgi",
        "check-local" => "disable",
        "max-procs" => 1,
    ))
)

url.redirect = ( "^/fc-importer$" => "/fc-importer/" )

debug.log-request-handling = "enable"

(change fc-importer into your tools name)

webservice lighttpd start

I've followed the instructions above, and I'm reasonably confident I've done it as described. But when I try to start the webservice, I get this in the logs:

2015-07-28 17:22:23: (log.c.166) server started
2015-07-28 17:22:23: (mod_fastcgi.c.1103) the fastcgi-backend /data/project/movestats/www/python/src/app.fcgi failed to start:
2015-07-28 17:22:23: (mod_fastcgi.c.1107) child exited with status 13 /data/project/movestats/www/python/src/app.fcgi
2015-07-28 17:22:23: (mod_fastcgi.c.1110) If you're trying to run your app as a FastCGI backend, make sure you're using the FastCGI-enabled version.
If this is PHP on Gentoo, add 'fastcgi' to the USE flags.
2015-07-28 17:22:23: (mod_fastcgi.c.1398) [ERROR]: spawning fcgi failed.
2015-07-28 17:22:23: (server.c.1021) Configuration of plugins failed. Going down.

My app.py runs okay standalone.

Have I done something wrong?

I'm at a loss why this would happen. I've changed the app.fcgi to be completely wrapped by a try-except block, and even that fails. That's consistent with the output: 'Flask server started <time>' is not sent to the error log, so it's completely not running the fcgi file. I'm not sure /why/, though :/

@yuvipanda is working on getting the uwsgi option to work, so that would also solve the issue (although I think this points to some underlying issue we don't understand)

Change 227503 had a related patch set uploaded (by Yuvipanda):
toollabs: Add support for uwsgi-plain webservice type

https://gerrit.wikimedia.org/r/227503

Change 227503 merged by Yuvipanda:
toollabs: Add support for uwsgi-plain webservice type

https://gerrit.wikimedia.org/r/227503

ok! so if your webservice is of type uwsgi-plain, you can put whatever you want in your uwsgi.ini - that should be able to get you working with python3 / uwsgi with just following uwsgi tutorials someplace. Once we have an incantation that works generally for people I can codify that into uwsgi-python3

Mmmmkay. I've put this in ~/uwsgi.ini:

[uwsgi]
socket = /tmp/movestats.sock
chdir = /data/project/movestats/www/python/src
pythonpath = /data/project/movestats/www/python/venv
module = app

Then:

$ webservice2 uwsgi-plain start

and I get a bunch of these in error.log:

/usr/bin/uwsgi: unrecognized option '--venv'
getopt_long() error

Sorry if this is really obvious, but I can't immediately spot the connection between the error in the log file and what I've put in uwsgi.ini.

On a little more investigation, it seems that --venv does work if --plugin python3 (or --plugin python) is also specified. Putting plugin = python3 in the ini file doesn't change anything. Perhaps adding this to modules/toollabs/files/tool-uwsgi-plain would be a workaround, though not very satisfactory:

if os.path.exists(os.path.expanduser('~/www/python/venv/bin/python2')):
    args += ['--plugin', 'python']

if os.path.exists(os.path.expanduser('~/www/python/venv/bin/python3')):
    args += ['--plugin', 'python']

I guess another option would be this:

from itertools import chain
if os.path.exists(os.path.expanduser('~/uwsgi-plugins')):
    with open(os.path.expanduser('~/uwsgi-plugins')) as f:
        plugins = list(chain.from_iterable([('--plugin', p) for p in f.readlines()]))
        args += plugins

A bit more generic but a bit more black-box as well.

Or, in fact, even just reversing the order in which the --venv and --ini arguments are added to the args list would be good enough. If --ini comes first, then you can specify the plugin in the ini file, and then --venv will work.

I think --venv shouldn't be passed at all by default, and instead should be provided in the .ini file.

Yes, I guess that'd also work!

Change 227690 had a related patch set uploaded (by Merlijn van Deen):
toollabs: uwsgi-plain: remove python specifics

https://gerrit.wikimedia.org/r/227690

Change 227690 merged by Yuvipanda:
toollabs: uwsgi-plain: remove python specifics

https://gerrit.wikimedia.org/r/227690

I've updated the wiki with some guidelines on how to get python3 working using this. The biggest problem seems to be that uWSGI mountpoints don't work with Python 3, but that's a uWSGI bug.

Thanks to @bd808 and unbit it works!

virtualenv --python=python3 ~/www/python/venv
source ~/www/python/venv/bin/activate
pip install flask
webservice2 uwsgi-plain start

~/uwsgi.ini

[uwsgi]
plugin = python3
socket = /data/project/jshint/jshint.sock
chdir = /data/project/jshint/www/python/src
venv = /data/project/jshint/www/python/venv
module = app
callable = app
manage-script-name = true
mount = /jshint=app:app

~/www/python/src/app.py

import sys

from flask import Flask, jsonify

app = Flask(__name__)


@app.route('/')
def index():
    return jsonify({'I am': sys.version, 'route': '/'})


@app.route('/test')
def test():
    return jsonify({'I am': sys.version, 'route': '/test'})


if __name__ == '__main__':
    app.run()

Also routes and static_folder work correctly as shown at https://tools.wmflabs.org/jshint/test and https://tools.wmflabs.org/jshint/static/xyz.css

See notes at rOPUPe38255caa48e393cad47092f815f11b3345b98e4#1392953 for a possible fix for tool-uwsgi-python that matches the config used in T104374#1911373.

Maybe @yuvipanda is able to unstall this based on T98440: Deploy new unified webservice code

Hello! I've more direct and better python3 support in the kubernetes backend - I've written up some docs on https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes#python_.28uwsgi_.2B_python3.4.29 :)

In T104374#2494348, @yuvipanda wrote:

Once we migrate all the uwsgi-plain webservice to kubernetes, I think we can call this done *and* remove the uwsgi-plain webservice type.

Do you advise tool maintainers to migrate all of them to kubernetes right now?

@Ricordisamoa yes, I do! Can I co-ordinate with you on IRC or email one of these days to move all your uwsgi-plain tools? :) https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes#python_.28uwsgi_.2B_python3.4.29 has more info...

In T104374#2718393, @yuvipanda wrote:

@Ricordisamoa yes, I do! Can I co-ordinate with you on IRC or email one of these days to move all your uwsgi-plain tools? :) https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes#python_.28uwsgi_.2B_python3.4.29 has more info...

Some of my app.py files live under ~/www/python/src/src instead of ~/www/python/src and I'd love to keep the ability of customizing that with Kubernetes.