Page MenuHomePhabricator

Introduce wikidata termbox SSR to kubernetes
Closed, ResolvedPublic

Description

Subtask of the TEC3:O3:O3.1:Q4 Goal to introduce wikidata termbox SSR to use the deployment pipeline

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
akosiaris triaged this task as Medium priority.Apr 8 2019, 2:43 PM

Change 505755 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[integration/config@master] Create/Publish OCI images for termbox

https://gerrit.wikimedia.org/r/505755

Change 505755 merged by jenkins-bot:
[integration/config@master] Create/Publish OCI images for termbox

https://gerrit.wikimedia.org/r/505755

Change 505788 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] First draft of a wikibase-termbox chart

https://gerrit.wikimedia.org/r/505788

@Tarrow, @WMDE-leszek. I 've been working on the termbox helm chart and while the service seems to be up and running, I see no /_info endpoint nor a swagger/openapi[1] spec published under /?spec. Both are crucial for deploying, as the former is used as a kubernetes readiness probe (aka if an instance of the app can't serve that for any reason it will temporarily not see new traffic) and the latter is used by our monitoring, so we can't proceed without those. Could you please have a look at it and add them? Thanks!

[1] https://swagger.io/docs/specification/about/

@Tarrow, @WMDE-leszek. I 've been working on the termbox helm chart and while the service seems to be up and running, I see no /_info endpoint nor a swagger/openapi[1] spec published under /?spec. Both are crucial for deploying, as the former is used as a kubernetes readiness probe (aka if an instance of the app can't serve that for any reason it will temporarily not see new traffic) and the latter is used by our monitoring, so we can't proceed without those. Could you please have a look at it and add them? Thanks!

[1] https://swagger.io/docs/specification/about/

Hmmm, is the service using service-runner? We've kind of standardised on that and it would good to follow that practice here as well.

We are indeed using service-runner; I don't think this provides /_info or /?spec though? Or are we missing something?

We are indeed using service-runner; I don't think this provides /_info or /?spec though? Or are we missing something?

Ah, indeed, sorry, it does not :) I meant service template. If you base the service on that, you get the standard testing end points.

@mobrovac During T221755 & T221754 we tended to [[ https://ssr-termbox.wmflabs.org/?spec | /?spec ]] and [[ https://ssr-termbox.wmflabs.org/_info | /_info ]] - could you please take a look?

Yup, lookin' good! Now service-checker can get the spec. The next step is to define (a) whether a specific end point should be tested; and (b) if so, what are the request/response pairs that it ought to send/expect.

For the former, each end point + method combo should have an x-monitor stanza that is set to true or false (whether to test the end point + method combo or not). If this stanza is not present, service-checker assumes its value to be true. For the latter, if x-monitor: true, then you need an additional stanza - x-amples. It is an array of objects, consisting of request and response properties that define the request to be sent and the response that is expected back. Both properties are objects themselves that can have the uri, headers and body properties, depending on the request.

Here are some examples of its usage to get you an idea of how this works in practice: RESTBase, Mathoid, MCS, Citoid.

@mobrovac Thanks for the feedback. If it is possible at all we would really appreciate if you could link us to the check list you seem to be following while reviewing this, so we can try for a more holistic approach in bringing this to completion. This would allow for us to better estimate how many steps (and time effort) are still to be expected, and for you not having to regularly context switch and come back to our ever so little progress in that regard.

With respect to the end point checks it would be great to hear what we are trying to achieve with them. Our service depends on the availability of another service. If the examples are to act as smoke tests then their reliability depends on the upstream service; a dependency which would need to be configured (are we going to point it against prod for this?) & modeled (how to express service inter-dependency in the config?) in order to be able to make sense of the information down the line (i.e. "no need to be alarmed that this service reported 500 while the mw api was down").

Cheers

@akosiaris and @WMDE-leszek I think the needed changes to proceed with the helm chart are now done. Feel free to poke me if there is something we missed though: we're keen not to be a blocker for getting this deployed :)

Hey @akosiaris and @mobrovac we've been wondering if you had a chance to look into our service again.
As reported above, to our best knowledge, we've made changes that have been stopping you from moving on with the helm chart.

As also mentioned in one of above comments, we'd be happy to perform any further changes required, if there was a list of (next) steps on the road for introducing the service to kubernetes infra. Hopefully this way the process would go even smoother for both sides.

As we assume it might be of relevance, I could also report from our side, that we believe to have made a significant progress in terms of service production readiness from the security perspective (T216419).

@WMDE-leszek. Yes I did.

Using https://locust.io/, wrote P8511 and benchmarked the service locally on my minikube instance. A rough howto (minus the locust part) is at https://wikitech.wikimedia.org/wiki/User:Alexandros_Kosiaris/Benchmarking_kubernetes_apps.

Below are graphs with the results from the benchmarks

CPU

termbox_cpu_usage.png (1×1 px, 133 KB)

Memory

termbox_mem_usage.png (1×1 px, 128 KB)

Requests:

termbox_locust_stats.png (1×1 px, 118 KB)

So a single worker termbox installation uses some 1cpu max, ~235MB memory and is able to serve ~16 req/s. When idling we are at ~140MB and 0 cpu usage. Those numbers should be used as a driver to gauge how many instances of the app we will need. I 've updated the chart with those numbers and will proceed with merging it.

Change 509391 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[wikibase/termbox@master] Add x-amples to termbox openapi.json

https://gerrit.wikimedia.org/r/509391

@Tarrow , @WMDE-leszek I 've noticed 3 things while working on the above

  • The service seems to be configurable to reach directly out to the wikidata endpoint. e.g. WIKIBASE_REPO: '{env(WIKIBASE_REPO,https://www.wikidata.org/w)}'. Since, in the general case, this goes via the edge caches (possibly polluting them with artificial requests, and limits operational flexibility, e.g. in the case of a datacenter switchover we can't really switch the app without switching the edge layer, we 've been defaulting to talking to internal endpoints and sending a Host: HTTP for the identification of the exact project. Would it be possible to add that functionality? (or guide me on how this is achieved if it is already implemented).

What exactly you want to do for that is up to you, but up to now services have been using https://github.com/wikimedia/service-template-node/blob/master/lib/util.js#L107 to wrap the route handlers which provides them out of the box with requests rates per endpoints, latencies per endpoint, errors and so on. I highly suggest that.

For service specific stats, e.g. how many time language=de has been passed as a request, options.metrics.increment is what has been used up to now as they have invariably been of a counter nature.

Overall I think we are ready to deploy to staging, production will have to wait a bit until we solve at least the first 2 items. The 3rd is not a hard blocker but I am willing to bet you want stats :-)

I also think we should schedule a training session for how deployers deploy code in the pipeline/kubernetes environment.

Hi @akosiaris - thanks for getting back to us.

sending a Host: HTTP for the identification of the exact project. Would it be possible to add that functionality?

It certainly is possible and depending on operational needs we certainly can make this happen. We quickly discussed this in the team and would like to first truly understand the goal to make sure we don't mix up the different layers of our proverbial sausage pizza without a valid reason. The service would be run in a container inside a k8s pod, controlling its DNS - why not use this option to make sure requests reach the intended endpoint? The correct host would then come "for free" per the host part configured in WIKIBASE_REPO.

FTR: One internal talking point here was that intentionally circumventing edge side caching might put additional strain on the infrastructure, increase response times - but we assume you have this covered.

x-amples stanza to /termbox endpoint

I looked at the change in gerrit, please find my comment there.

app does not emit statsd metrics for requests

I will create a story for that => T223202

service specific stats

Will be added as we go per the requirements from the technical product owner(s)

Thanks

With respect to the end point checks it would be great to hear what we are trying to achieve with them. Our service depends on the availability of another service. If the examples are to act as smoke tests then their reliability depends on the upstream service; a dependency which would need to be configured (are we going to point it against prod for this?) & modeled (how to express service inter-dependency in the config?) in order to be able to make sense of the information down the line (i.e. "no need to be alarmed that this service reported 500 while the mw api was down").

As per https://gerrit.wikimedia.org/r/#/c/wikibase/termbox/+/509391/,

The x-amples spec defines checks that will be exercised run using the service-checker[1] software in wikimedia infrastructure. Those run on a cron, contacting the service every 1 minute. The dependency part on mediawiki is something we can work on the alerting system level, but I have to say we don't currently do that for any service (and multiple ones rely on mediawiki) for mostly technical reasons related to our current monitoring infrastructure. That will not always be the case though.Also note that we know and accept that those monitoring checks create inorganic traffic and increase system load. It's a price that we have accepted to pay.

Hi @akosiaris - thanks for getting back to us.

sending a Host: HTTP for the identification of the exact project. Would it be possible to add that functionality?

It certainly is possible and depending on operational needs we certainly can make this happen. We quickly discussed this in the team and would like to first truly understand the goal to make sure we don't mix up the different layers of our proverbial sausage pizza without a valid reason. The service would be run in a container inside a k8s pod, controlling its DNS - why not use this option to make sure requests reach the intended endpoint? The correct host would then come "for free" per the host part configured in WIKIBASE_REPO.

The intended hostname is going to be (as is for all services) appservers.discovery.wmnet. That is an internal hostname that point to all the wiki projects and allows us to do a set of operations easily like switching traffic from one datacenter to another. That being said, as you can see, nothing specific about a project is coded in that hostname. That's on purpose, as we don't have a need - and don't really want - to treat operationally e.g. enwiki differently from dewiki. Technically it's just a set of apache webservers with many many virtualhosts for all of the hundreds of project we have. Hence the need for the Host: wikidata.org HTTP header. k8s can't help with that.

FTR: One internal talking point here was that intentionally circumventing edge side caching might put additional strain on the infrastructure, increase response times - but we assume you have this covered.

x-amples stanza to /termbox endpoint

I looked at the change in gerrit, please find my comment there.

Answered already, thanks for looking into it.

app does not emit statsd metrics for requests

I will create a story for that => T223202

service specific stats

Will be added as we go per the requirements from the technical product owner(s)

Thanks

Thanks!

Change 509391 merged by jenkins-bot:
[wikibase/termbox@master] Add x-amples to termbox openapi.json

https://gerrit.wikimedia.org/r/509391

Hi @akosiaris,

thanks for taking the time to explain the way the Host header is intended to be used.
If I understand correctly the goal is to ensure that requests originating from our service bear a header Host: (www.)wikidata.org and reach which ever IP(s) appservers.discovery.wmnet resolves to on the system running it. This sounds like a name resolution challenge and a case for HostAliases or, more traditionally, a CNAME record.
Again, if we are missing something, implementing this can be arranged, but we really rather not as it sounds like there are more appropriate tools for the job.

Thanks

Hi @akosiaris,

thanks for taking the time to explain the way the Host header is intended to be used.
If I understand correctly the goal is to ensure that requests originating from our service bear a header Host: (www.)wikidata.org and reach which ever IP(s) appservers.discovery.wmnet resolves to on the system running it. This sounds like a name resolution challenge and a case for HostAliases or, more traditionally, a CNAME record.

Unfortunately no. It is not a name resolution issue and hence HostAliases and/or CNAME records will not help. Note that we want to do here IS NOT direct the traffic to the correct endpoint, which is indeed a name resolution issue, but rather inform the endpoint which of the hundreds of virtualhosts the request is for, something that needs to be done at the HTTP level.

Again, if we are missing something, implementing this can be arranged, but we really rather not as it sounds like there are more appropriate tools for the job.

If there were, we would not be asking it :-)

May 14 08:24:34 <_joe_> and thanks for helping with that. In an ideal world, the teams developing new services would install them on beta
May 14 08:24:56 <_joe_> it might work out most of the times, but then we have people with no beta access, like wmde folks for termbox
May 14 08:31:15 <Krenair> Well, that's trivial to fix.

Looking at the WMDE people subscribed to this task...

Already have access:
Addshore
WMDE-leszek

Missing:
Matthias_Geisler_WMDE
Pablo-WMDE
Tarrow
Lea_WMDE
Jakob_WMDE

Are all the people on the 'Missing' list software engineers on the termbox project?

May 14 08:24:34 <_joe_> and thanks for helping with that. In an ideal world, the teams developing new services would install them on beta
May 14 08:24:56 <_joe_> it might work out most of the times, but then we have people with no beta access, like wmde folks for termbox
May 14 08:31:15 <Krenair> Well, that's trivial to fix.

Looking at the WMDE people subscribed to this task...

Already have access:
Addshore
WMDE-leszek

Missing:
Matthias_Geisler_WMDE
Pablo-WMDE
Tarrow
Lea_WMDE
Jakob_WMDE

Are all the people on the 'Missing' list software engineers on the termbox project?

All of the ones you've listed above are, except for Lea_WMDE who is not a software engineer.

Change 510470 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[wikibase/termbox@master] openapi.json: Fix x-amples syntax

https://gerrit.wikimedia.org/r/510470

Change 505788 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Introduce the wikibase-termbox chart

https://gerrit.wikimedia.org/r/505788

Hi @akosiaris,

could you please provide a (curl) example of how the request is supposed to look like? Seems we will need this either way - as specification for what we are supposed to implement - because of lack of understanding on our part.

What I understood we are supposed to implement (the resolve part under ops control on a container level; specific IP just as an example)...

$ curl -v --resolve appservers.discovery.wmnet:80:217.160.0.248 -H "Host: www.wikidata.org" http://appservers.discovery.wmnet/w/api.php
* Added appservers.discovery.wmnet:80:217.160.0.248 to DNS cache
* Hostname appservers.discovery.wmnet was found in DNS cache
*   Trying 217.160.0.248...
* TCP_NODELAY set
* Connected to appservers.discovery.wmnet (217.160.0.248) port 80 (#0)
 > GET /w/api.php HTTP/1.1
 > Host: www.wikidata.org
...

The following does the same thing...

$ curl -v --resolve www.wikidata.org:80:217.160.0.248 http://www.wikidata.org/w/api.php
* Added www.wikidata.org:80:217.160.0.248 to DNS cache
* Hostname www.wikidata.org was found in DNS cache
*   Trying 217.160.0.248...
* TCP_NODELAY set
* Connected to www.wikidata.org (217.160.0.248) port 80 (#0)
 > GET /w/api.php HTTP/1.1
 > Host: www.wikidata.org
...

What are we missing?
Thanks

@Pablo-WMDE all you have to do is have the request be of this form:

URI: http://appservers.discovery.wmnet/{+your_path_here}
headers:
  Host: www.wikidata.org

The curl you are trying does not work because you are executing it outside the production environment and then Varnish strips out the Host header. All of the services work this way in prod, cf. e.g. the MCS config for making requests to MediaWiki.

@mobrovac and @akosiaris Sorry to drop in late to the conversation but I've been on holiday and then at the hackathon. I wanted to check a couple of things:

@Pablo-WMDE all you have to do is have the request be of this form:

URI: http://appservers.discovery.wmnet/{+your_path_here}
headers:
  Host: www.wikidata.org

The curl you are trying does not work because you are executing it outside the production environment and then Varnish strips out the Host header. All of the services work this way in prod, cf. e.g. the MCS config for making requests to MediaWiki.

Should we be using http rather than https internally?

I also wanted to check if we should be using appservers.discovery.wmnet as the host; I'm not sure if I read things right but I didn't think this host is defined? Or should we aiming for appservers-ro.discovery.wmnet (since we are not writing)? I then saw https://phabricator.wikimedia.org/T199219#4452041 and was wondering if that is indeed the right place or if it is oddly slow/not cached.

In general is there a place on wikitech (or similar) we should be looking to make sure we're getting things right?

Should we be using http rather than https internally?

Yes, indeed, sorry that was an omission on my part. Since setting up TLS connections takes time, though, you also want to keep these connections alive. You can do so by explicitly adding this agent option to the request template:

uri: https://appservers-ro.discovery.wmnet/{+path}
headers:
  Host: '{req.params.domain}'
agentOptions:
  keepAlive: true

I also wanted to check if we should be using appservers.discovery.wmnet as the host; I'm not sure if I read things right but I didn't think this host is defined? Or should we aiming for appservers-ro.discovery.wmnet (since we are not writing)? I then saw https://phabricator.wikimedia.org/T199219#4452041 and was wondering if that is indeed the right place or if it is oddly slow/not cached.

Yup, +1, use appservers-ro here, in that case. The end point itself is synonymous with appservers-rw for now, but once we start splitting MW traffic based on reads/writes, this will mean that your requests will always land in the local DC, both giving it a speed-up and decreasing the load from the master appservers, resulting in a net win for everybody :)

In general is there a place on wikitech (or similar) we should be looking to make sure we're getting things right?

Not that I'm aware of :/

@mobrovac Thanks! I think we've now taken most of this onboard and merged it.

@akosiaris could you take a look at out new x-amples block and Host header? Hopefully this meets our needs :).

You'll see there are now two new ENV variables: HEALTHCHECK_QUERY and WIKIBASE_REPO_HOSTNAME_ALIAS respectively. I think this should provide all the flexibility we need

Mentioned in SAL (#wikimedia-operations) [2019-05-27T15:40:07Z] <akosiaris> initialize termbox namespace on eqiad/codfw/staging kubernetes clusters T220402

Hello @akosiaris, this is a friendly ping from your favourite WMDE customers :)
Any news on the deployment front? Is there anything you would like us to do next, or we could be of help with in any way?

Change 510470 abandoned by Alexandros Kosiaris:
openapi.json: Fix x-amples syntax

Reason:
Already added in a better way in https://gerrit.wikimedia.org/r/#/c/wikibase/termbox/ /511892/

https://gerrit.wikimedia.org/r/510470

Change 515028 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] termbox: Use newer ENV variables

https://gerrit.wikimedia.org/r/515028

@Tarrow, @WMDE-leszek
Hi, sorry for taking so long to answer to this, it's been really busy.

@mobrovac Thanks! I think we've now taken most of this onboard and merged it.

@akosiaris could you take a look at out new x-amples block and Host header? Hopefully this meets our needs :).

I did have a look. Thanks for catering to the Host header support!

I 've updated the helm chart to honor the above and went through a testing phase. I've also added you both as maintainers of the helm chart. https://gerrit.wikimedia.org/r/515028

Almost everything is fine, except some issue in the generator for the x-amples in the HEALTHCHECK_URL. The generated /openapi.json stanza in question is

"x-amples": [
          {
            "title": "get rendered termbox",
            "request": {
              "query": {
                "language": "de",
                "entity": "Q1",
                "revision": 103,
                "editLink": "/edit/Q1347",
                "preferredLanguages": [
                  "de",
                  "en"
                ]
              },
              "response": {
                "status": 200,
                "headers": {
                  "content-type": "text/html"
                }
              }

Note how preferred languages is an array indeed but is not pipeDelimited as the openapi spec requires, but is rather given in an array format and not as a pipedelimited string. That in turn causes service-checker to error out with the following

DEBUG:urllib3.connectionpool:http://192.168.99.100:18788 "GET /termbox?revision=103&editLink=%2Fedit%2FQ1347&entity=Q1&preferredLanguages=%5B%27de%27%2C+%27en%27%5D&language=de HTTP/1.1" 400 169
/termbox (get rendered termbox) is CRITICAL: Test get rendered termbox returned the unexpected status 400 (expecting: 200)

as it tries to convert [] and ] into url encoded strings, thus creating a parameter than violates the spec and making the health check fail. I had a quick look at the code but found no quick way to fix it, could you please have a look?

Indeed this was fixed. However another regression has crept up it's head

Doing a curl 'http://192.168.99.100:18788/termbox?editLink=%2Fedit%2FQ1347&preferredLanguages=de&language=de&entity=Q1&revision=103' returns a 500 with an error in the logs

{
  "name": "termbox",
  "hostname": "termbox-termbox-64d46c6dcb-8jl86",
  "pid": 16,
  "level": 50,
  "err": {
    "message": "",
    "name": "wikibase-termbox",
    "stack": "Error: wikibase-shortcopyrightwarning-heading is not a valid message-key.\n    at wikibase.termbox.main.js:18168:17\n    at Array.forEach (<anonymous>)\n    at AxiosWikibaseMessagesRepo.transformMessages (src/server/data-access/AxiosWikibaseMessagesRepo.ts:77:11)\n    at AxiosWikibaseMessagesRepo.getMessageTranslationCollection (src/server/data-access/AxiosWikibaseMessagesRepo.ts:71:24)\n    at wikibase.termbox.main.js:18147:27\n    at process._tickCallback (internal/process/next_tick.js:68:7)",
    "levelPath": "error/service"
  },
  "msg": "wikibase-shortcopyrightwarning-heading is not a valid message-key.",
  "time": "2019-06-07T12:04:33.596Z",
  "v": 0
}

Note that the previous image, 2019-06-03-162801-production (aka git sha1 7c5347f48b4225a48e94bb846c7e4d8820cde347) did not exhibit this. So this is a regression introduced since then.

Change 515067 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] Assign termbox.svc.{eqiad,codfw}.wmnet LVS IPs

https://gerrit.wikimedia.org/r/515067

Change 515069 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] Enable discovery for termbox

https://gerrit.wikimedia.org/r/515069

Change 515074 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Introduce termbox LVS configuration

https://gerrit.wikimedia.org/r/515074

Indeed this was fixed. However another regression has crept up it's head

This should now be fixed. Sadly this was due to a mismatch between the code in wikibase master and that deployed on Wikidata.org

This should now be fixed. Sadly this was due to a mismatch between the code in wikibase master and that deployed on Wikidata.org

And yes, we are finally in greener pastures.

helm test termbox
RUNNING: termbox-termbox-service-checker
PASSED: termbox-termbox-service-checker

Thanks for taking care of it so quickly!

Change 515028 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] termbox: Use newer ENV variables

https://gerrit.wikimedia.org/r/515028

Change 515177 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] termbox: Add kubernetes stanzas

https://gerrit.wikimedia.org/r/515177

Just as an FYI, everything looks ok on this end, but there's a train freeze this week, so we have to wait before deploying this. Patches are up and waiting to be merged on Monday the 17th

@akosiaris Great! Thanks for that. We look forward to seeing how it all goes forward post-offsite :)

Change 515177 merged by Alexandros Kosiaris:
[operations/puppet@production] termbox: Add kubernetes stanzas

https://gerrit.wikimedia.org/r/515177

Change 515067 merged by Alexandros Kosiaris:
[operations/dns@master] Assign termbox.svc.{eqiad,codfw}.wmnet LVS IPs

https://gerrit.wikimedia.org/r/515067

Change 515074 merged by Alexandros Kosiaris:
[operations/puppet@production] Introduce termbox LVS configuration

https://gerrit.wikimedia.org/r/515074

Change 517628 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] termbox: Use HTTP for WMF service checks

https://gerrit.wikimedia.org/r/517628

Change 517628 merged by Alexandros Kosiaris:
[operations/puppet@production] termbox: Use HTTP for WMF service checks

https://gerrit.wikimedia.org/r/517628

Change 515069 merged by Alexandros Kosiaris:
[operations/dns@master] Enable discovery for termbox

https://gerrit.wikimedia.org/r/515069

akosiaris claimed this task.
curl -s -I -X GET 'http://termbox.discovery.wmnet:3030/termbox?editLink=%2Fedit%2FQ1347&preferredLanguages=de&language=de&entity=Q1&revision=103'
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 1568
ETag: W/"620-MeNQXY3hfRVxLzBPruUZ418lGUc"
Vary: Accept-Encoding
Date: Tue, 18 Jun 2019 14:30:54 GMT
Connection: keep-alive

This is deployed finally. The canonical endpoint to talk to the service in production is termbox.discovery.wmnet as per the command above. @Tarrow , @WMDE-leszek, I 'll resolve this, feel free to reopen