Update demo environment to support search
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	Mar 3 2021, 5:12 PM

Description

The demo server that we are currently running is a single docker container with an sqlite database. This was a nice way to get started, but we need to make things a bit more complex as we add features to the system.

I see three equally possible ways to build things out:

Deploy elasticsearch (and mysql) as a Puppet managed bare metal service
Deploy elasticsearch (and mysql) as a Docker container using a systemd unit similar to the one created for Toolhub
Deploy Toolhub and elasticsearch using docker-compose (similar to the dev environment)

Currently I'm leaning towards the docker-compose solution, mostly because of its similarity to the dev environment setup. I'm not sure if we can refactor things so that a single docker-compose config file is used in both environments, but it should be reasonably simple to keep the two configurations in sync.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	demo: add demo server config	wikimedia/toolhub	main	+215 -0

Customize query in gerrit

Related Objects

Mentioned In: rWTHU14b337e0442b: demo: add demo server config

Event Timeline

bd808 created this task.Mar 3 2021, 5:12 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2021, 5:12 PM

My first attempt at this is just bringing up an elasticsearch container using a systemd unit in the same way that I have been running the toolhub container.

docker.elasticsearch.service

[Unit]
Description=Toolhub elasticsearch server
After=docker.service
Wants=network-online.target docker.socket
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
WorkingDirectory=/srv/toolhub
ExecStartPre=-/usr/bin/docker stop toolhub-es
ExecStartPre=-/usr/bin/docker rm toolhub-es
ExecStartPre=/usr/bin/docker pull docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.14
ExecStartPre=/usr/bin/docker run --name toolhub-es \
  --rm -d \
  -p 0.0.0.0:9200:9200/tcp \
  -e "node.name=es01" \
  -e "cluster.name=es-toolhub" \
  -e "discovery.type=single-node" \
  -e "bootstrap.memory_lock=true" \
  -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
  --ulimit memlock=-1:-1 \
  --ulimit nofile=65536:65536 \
  -v /dev/log:/dev/log \
  --log-driver=journald \
  docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.14
ExecStart=/usr/bin/docker start -a toolhub-es
ExecStop=/usr/bin/docker stop toolhub-es

[Install]
WantedBy=local.target

I'm going to try docker-compose next.

Restricted Application added a project: User-bd808. · View Herald TranscriptMar 5 2021, 1:10 AM

bd808 triaged this task as High priority.Mar 5 2021, 1:10 AM

I have a new and improved demo stack running on toolhub-demo01.toolhub.eqiad1.wikimedia.cloud and connected to the https://toolhub-demo.wmcloud.org/ proxy. The new system is using docker-compose to manage containers for the Toolhub runtime, MariaDB 10.4, and Elasticsearch 6.8.14. A custom systemd unit is used to start/stop the docker-composed managed collection of Docker containers. Additionally a Makefile provides convenient commands for manually triggering a restart, updating the Toolhub container image, and watching logs.

The code/configuration for this all lives in a local git repo at toolhub-demo01:/srv/toolhub/demo currently. I would like to move this into a more durable git tree to make recreating the demo server easier. To that end I am leaving this task open, but lowering its priority as the most urgent work has been completed.

Change 672832 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] demo: add demo server config

https://gerrit.wikimedia.org/r/672832

gerritbot added a project: Patch-For-Review.Mar 16 2021, 10:08 PM

bd808 moved this task from In Progress to Review on the Toolhub board.Mar 16 2021, 10:10 PM

Change 672832 merged by jenkins-bot:
[wikimedia/toolhub@main] demo: add demo server config

https://gerrit.wikimedia.org/r/672832

bd808 mentioned this in rWTHU14b337e0442b: demo: add demo server config.Mar 18 2021, 9:40 PM

Maintenance_bot removed a project: Patch-For-Review.Mar 18 2021, 10:11 PM

bd808 closed this task as Resolved.Mar 19 2021, 5:19 PM