Page MenuHomePhabricator

Create demonstration/user acceptance testing environment for Toolhub
Closed, ResolvedPublic

Description

We need a better system than screen sharing for demonstrating Toolhub to the advisory board and other interested community members.

The two options that come to mind are hosting on Toolforge and a dedicated Cloud VPS project. Currently Toolforge hosting would require special exemptions from Toolforge administrators to publish custom Docker images into the Toolforge Docker registry or creating a Toolforge compatible deployment using the existing Toolforge Python 3.7 images.

Event Timeline

It would be neat to host the demo version of Toolhub in Toolforge, but I think it would be simpler for now to request and use a dedicated Cloud VPS project. This will still take a bit of work to setup and deploy, but it will not require negotiating special hosting support with the Toolforge admins or inventing new deployment tooling to fit into supported Toolforge containers and their workflows.

bd808 triaged this task as Medium priority.Nov 30 2020, 5:44 PM

I ran head first into T263597: Allow use of copied artifacts in subsequent build step while trying to make a Blubber config that would let me:

  1. Run npm run build:vue
  2. Copy the generated vue/dist output into a python container
  3. Run poetry run ./manage.py collectstatic including the vue/dist output

I have an idea for a really ugly hack to work around this that involves adding nodejs and npm to a python base container, but there really needs to be a better solution for this in Blubber.

I have an idea for a really ugly hack to work around this that involves adding nodejs and npm to a python base container, but there really needs to be a better solution for this in Blubber.

My ugly hack idea ran into a different wall (needing buster-backports to get a working npm version), so I'm going to try to make T263597: Allow use of copied artifacts in subsequent build step happen.

Change 648382 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] vue: Move webpack output location

https://gerrit.wikimedia.org/r/648382

Change 648382 merged by jenkins-bot:
[wikimedia/toolhub@main] vue: Move webpack output location

https://gerrit.wikimedia.org/r/648382

Change 648395 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] pipeline: Build demo server using Blubber

https://gerrit.wikimedia.org/r/648395

Out of laziness, and for speed of prototyping, I'm going to manually setup the initial toolhub-beta01.toolhub.eqiad1.wikimedia.cloud instance.

Install Docker CE from docker.com
$ sudo apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common
$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
$ sudo apt update
$ sudo apt install docker-ce

Once docker was installed on toolhub-beta01.toolhub.eqiad1.wikimedia.cloud, I was able to start up the demo server with this script:

run_demo_server.sh
#!/bin/bash

# FIXME: use an env file instead of cli args
exec sudo docker run --name toolhub \
    --rm -it \
    -p 0.0.0.0:8000:8000/tcp \
    -e LOGGING_LEVEL=DEBUG \
    -e DJANGO_SECRET_KEY=REDACTED_SECRET_KEY \
    -e WIKIMEDIA_OAUTH2_KEY=7b3bd7ccc00873b35dc25d7e81fe44f8 \
    -e WIKIMEDIA_OAUTH2_SECRET=REDACTED_OAUTH_SECRET \
    bd808/toolhub-beta:latest

To make a nice demo, after starting the demo container do these things:

Probably not the prettiest systemd unit file ever written, but this seems to be working on toolhub-beta01.toolhub.eqiad1.wikimedia.cloud:

docker.toolhub.service
[Unit]
Description=Toolhub demo server
After=docker.service
Wants=network-online.target docker.socket
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
WorkingDirectory=/srv/toolhub
ExecStartPre=-/usr/bin/docker stop toolhub-demo
ExecStartPre=-/usr/bin/docker rm toolhub-demo
ExecStartPre=/usr/bin/docker pull bd808/toolhub-beta:latest
ExecStartPre=/usr/bin/docker run --name toolhub-demo \
  --rm -d \
  -p 0.0.0.0:8000:8000/tcp \
  --env-file /srv/toolhub/demo.env \
  -v /dev/log:/dev/log \
  --log-driver=journald \
  bd808/toolhub-beta:latest
ExecStart=/usr/bin/docker start -a toolhub-demo
ExecStop=/usr/bin/docker stop toolhub-demo

[Install]
WantedBy=local.target

The server still needs some manual help after starting up to be in a good state for demonstrations at this point. Ideally I will add more to the container startup to make this as automatic as possible.

Change 648395 merged by jenkins-bot:
[wikimedia/toolhub@main] pipeline: Build demo server using Blubber

https://gerrit.wikimedia.org/r/648395

The server still needs some manual help after starting up to be in a good state for demonstrations at this point. Ideally I will add more to the container startup to make this as automatic as possible.

{{Done}} in https://gerrit.wikimedia.org/r/648395

The container now loads an initial dataset and performs a crawler run at startup.