After following the instructions on the wiki

Could you please link to the instructions on the wiki?

Potentially related:
T159892: Make tools-webservice use the official kubernetes python client rather than pykube, T174769: Make it less cumbersome to bootstrap and update python webservices

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Using_virtualenv_with_webservice_shell

Given that the current grid hosts have an old version of python 2.7.6

This seems like an XY Problem. What exactly makes you require a newer version than 2.7.6?

the only way to use a newer version is the k8s

Another method is for you to compile your own python. See T182820#3835754

The k8s no longer have some of the standard packages that the old grid hosts have. (specifically mwparserfromhell) I asked about getting it installed and was told to use virtualenv because it would bloat the pods.

Packages written entirely in python will not be installed on k8s containers even if k8s supports CGI. See T140110, and "it would bloat the pods" is the reason.

Im not using a uwsgi app, but rather just the basic cgi. Can we please get the k8s to support cgi and virtualenv ?

• How would k8s run CGI?
  • CGI is currently run on grid with lighttpd, and this is also how php5.6 container in k8s runs PHP. You may be able to use that to start lighttpd
• Can k8s run containers run generic CGI, supporting lots of languages?
  • No, this was tested by yuvi in T152089: Build toollabs trusty 'catch all' container, and rumors say it failed badly.
• Can CGIs execute python in virtual environments?
  • Possibly. Virtual environments require a virtualenv-specific python interpretor, in /path/to/your/virtualenv/bin/python. If script remain unchanged, they will be executed by the site-wide python /usr/bin/python. Some workarounds I thought of:
    • The straightfoward way: find out how lighttpd determine the path of the interpretor. This could likely be either:
      • via the shebang line. Then you can just change the shebang of each of your CGI scripts to point to the python interpretor of your virtualenv
      • via lighttpd configuration file. Then you can just override it in your ~/.lighttpd.conf
    • The hacky way: make a wrapper for each CGI script, sets up the environment (eg. $PATH), then os.exec* into the python interpretor of your virtualenv.
  • What about performance?
    • Virtualenvs loads all its installed packages (and may even include python built-in libraries) from NFS. This means, slow startup time due to imports. (Although T136712 is fixed, it will still be slower than the site-wide python on the local storage of each instance.) CGI scripts starts a process for each request, and that process has to bootstrap itself from NFS with imports. That means: slowwww. Uwsgi uses a persistent process, and imports usuallyonly happen during webservice startup or first request (depending on how the code is written), and therefore is not really affected by this slowness.

TL;DR: virtualenvs + CGI on k8s is probably neither what you want nor what would be supported.

there is an issue with sslv3 in older versions of python which has been fixed in .8 or .9 which is why I was investigating k8s

In T183090#3843358, @Betacommand wrote:

there is an issue with sslv3 in older versions of python which has been fixed in .8 or .9

You mean PEP 0466?

In T183090#3843370, @zhuyifei1999 wrote:

In T183090#3843358, @Betacommand wrote:

there is an issue with sslv3 in older versions of python which has been fixed in .8 or .9

You mean PEP 0466?

I am not able to directly correlate the two, but I think it is the same issue. The specific error that I am running into is:

<urlopen error [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>
args = (SSLError(1, '_ssl.c:510: error:14077410:SSL rout..._GET_SERVER_HELLO:sslv3 alert handshake failure'),)
errno = None
filename = None
message = ''
reason = SSLError(1, '_ssl.c:510: error:14077410:SSL rout..._GET_SERVER_HELLO:sslv3 alert handshake failure')
strerror = None

@Betacommand wrote:

Can we please get the k8s to support cgi and virtualenv ?

The php5.6 containers come configured to process .py and .pyc files as CGI scripts. This container type also comes with Python 2.7.9 installed. The default config however uses /usr/bin/python as the Python CGI runner which would not use a local virtualenv.

One way to work around this would be to pick a new file extension for execution from a virtualenv. I tested this out by creating a virtualenv using the instructions on setting up a venv for a uwsgi webservice and then adding some customization to $HOME/.lighttpd.conf:

$ webservice --backend=kubernetes python2 shell
If you don't see a command prompt, try pressing enter.
$ mkdir -p ~/www/python
$ python2 -m virtualenv ~/www/python/venv
Already using interpreter /usr/bin/python2
New python executable in /data/project/bd808-test//www/python/venv/bin/python2
Also creating executable in /data/project/bd808-test//www/python/venv/bin/python
Installing setuptools, pip...done.
$ source ~/www/python/venv/bin/activate
$ pip install --upgrade pip
Downloading/unpacking pip from https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.6
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...
$ pip install mwparserfromhell
Collecting mwparserfromhell
  Downloading mwparserfromhell-0.5.1.tar.gz (132kB)
    100% |████████████████████████████████| 133kB 610kB/s
Installing collected packages: mwparserfromhell
  Running setup.py install for mwparserfromhell ... done
Successfully installed mwparserfromhell-0.5.1
$ logout
$ vim ~/.lighttpd.conf

# Run python CGI from venv
cgi.assign += (
  ".py2" => "/data/project/bd808-test/www/python/venv/bin/python2"
)

See the result at https://tools.wmflabs.org/bd808-test/venv.py2

Adding a rewrite rule wold let you keep the same URLs and only need to change the file names on disk:

# Run python CGI from venv
cgi.assign += (
  ".py2" => "/data/project/bd808-test/www/python/venv/bin/python2"
)
url.rewrite-once += (
  "^(.*)\.py$" => "$1.py2"
)

Example: https://tools.wmflabs.org/bd808-test/venv.py