Page MenuHomePhabricator
Create Task
Maniphest T365839

Deploy an instance of GrowthBook to Kubernetes
Closed, ResolvedPublic
Actions

Description

As part of SDS2.1 we want to validate that Growthbook can be deployed on our infrastructure. This does not include any integration with other systems. We will be using the OpenSource version of Growth book.

Steps

  • Minimal research and design is done to have a coherent plan of how to execute that deployment (this will create additional steps in this list)
  • (optional) deploy Growthbook to WMCS or another simpler environment

These are the detailed steps that have been identified so far:

  • Build a container image for Growthbook (open core version)
  • Build a container image for FerretDB mongoDB
  • Create a PostgreSQL database on an-db100[1-2] for Growthbook
  • Create a namespace for Growthbook on the dse-k8s cluster
  • Create a helm chart for Growthbook with FerretDB mongoDB
  • Define a helmfile deployment on the dse-k8s cluster
  • Deploy Growthbook and FerretDB mongoDB
  • Document the tunneling/ingress requirements of the Growthbook UI and API

AC

  • A GrowthBook instance is deployed to DSE Kubernetes
  • The GrowthBook UI and API endpoints are not externally accessible but are accessible via an SSH tunnel

Details

SubjectRepoBranchLines +/-
growthbook: small fixes to the valuesoperations/deployment-chartsmaster+7 -4
Growthbook: disable aythmechanism PLAINoperations/deployment-chartsmaster+3 -3
Update the hostname for mongodb accessoperations/deployment-chartsmaster+1 -1
Configure growthbook/mongodb deployment with a recreate strategyoperations/deployment-chartsmaster+4 -2
growthbook: deploy mongodb with auth enabledoperations/deployment-chartsmaster+10 -3
Use a 10 GB persistent volume for mongodboperations/deployment-chartsmaster+20 -4
growthbook: fix mongo connection string and update imageoperations/deployment-chartsmaster+2 -2
growthbook: replace ferretdb by mongo itselfoperations/deployment-chartsmaster+157 -61
growthbook: bump chart version to allow subchart upgradeoperations/deployment-chartsmaster+1 -1
growthbook: fix typo in tls hostnameoperations/deployment-chartsmaster+1 -1
growthbook: fix volume/configmap name problemoperations/deployment-chartsmaster+4 -4
growthbook: define one TLS hostname per subchartoperations/deployment-chartsmaster+4 -1
growthbook: split chart into 2 charts (frontend/backend)operations/deployment-chartsmaster+695 -641
growthbook: provision DNS records for each backend/frontend serviceoperations/dnsmaster+2 -0
growthbook: add mesh service, configuration, container and networkpoliciesoperations/deployment-chartsmaster+8 -2
growthbook: bind ferretdb service to 0.0.0.0 instead of localhostoperations/deployment-chartsmaster+2 -1
growthbook: small fixes to the chartoperations/deployment-chartsmaster+14 -5
growthbook: define helmfile and production valuesoperations/deployment-chartsmaster+77 -0
Create a new chart for growbook using scaffolding.operations/deployment-chartsmaster+2 K -0
growthbook: add svc.eqiad and discovery recordsoperations/dnsmaster+2 -0
Add a growthbook namespace to the dse-k8s clusteroperations/deployment-chartsmaster+2 -0
Add kubeconfig files for growthbook on dse-k8soperations/puppetproduction+4 -0
Add a PostgreSQL database for Growthbookoperations/puppetproduction+1 -0
Correct the signing key for the yarn apt repooperations/puppetproduction+1 -1
Fix the thirdparty/yarn repooperations/puppetproduction+1 -1
Add the thirdparty/yarn deb repository to repreprooperations/puppetproduction+230 -0
Show related patches Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Minimize the amount of layers needed to download mongodbrepos/data-engineering/growthbook!15brouberolshrink-image-sizemain
Install mongosh, the mongoDB shell, in the imagerepos/data-engineering/growthbook!14brouberolT365839-install-mongo-shellmain
Install mongodb in the growthbook imagerepos/data-engineering/growthbook!12brouberoladd_growthbook_servicemain
Attempt to set a proxy for undici on trusted runnersrepos/data-engineering/growthbook!7btullisconfigure_undici_proxymain
Disable the use of SWC, as it fails to download on trusted runnersrepos/data-engineering/growthbook!6btullisdisable_swcmain
Set npm proxy instead of yarn https-proxyrepos/data-engineering/growthbook!5btullisadd_npm_proxymain
Set the https-proxy as well as the plaintext version.repos/data-engineering/growthbook!4btullisfix_https_proxymain
Configure the yarn proxy supportrepos/data-engineering/growthbook!3btullisfix_yarn_proxymain
Add ferretdb and growthbook data-engineering projects to trusted runnersrepos/releng/gitlab-trusted-runner!92btullisadd_growthbook_ferretdbmain
Add the build pipeline for ferretdbrepos/data-engineering/ferretdb!1btullisadd_initial_pipelinemain
Draft: Modify the build process to exclude enterprise modulesrepos/data-engineering/growthbook!2btullisexclude_enterprise_modulesadd_initial_build_pipeline
Add the initial build pipeline for Growthbookrepos/data-engineering/growthbook!1btullisadd_initial_build_pipelinemain
Show related patches Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
BTullis added a comment.Jul 15 2024, 4:40 PM

I have created an initial blubber/kokkuri pipeline for building our own version of Growthbook, based on the upstream Dockerfile.

It works, in as much as I have tested it with a local docker compose up command, which pulls in a regular mongodb container for the metadata DB. I can then browse the app on http://localhost:3000

image.png (999×1 px, 185 KB)

Unfortunately, the code for Growbook is released under somewhat unfavourable licensing terms: LICENSE

  • All content that resides under the "packages/enterprise/" directory of this repository is licensed under the license defined in "packages/enterprise/LICENSE".

That licence is named The GrowthBook Enterprise License and it is proprietary in its nature.

I have attempted to build Growthbook without including this directory (as this diff shows), but unfortunately both of the front-end and back-end package directories refer to this module in many places.
Therefore, I believe that it would require considerable work to remove it altogether.

I don't think that it will be possible to continue to build and publish this container image without seeking approval from senior management to make an exception for Growthbook.
cc @Gehel and @odimitrijevic who I believe have already been discussing this possibility.

odimitrijevic added a comment.Jul 16 2024, 7:25 PM

@BTullis Please go ahead with the deployment. The goal with the POC to understand if this is a viable solution. The decision around non-OSI licensing will follow once a recommendation is made.

Ottomata added a comment.EditedJul 16 2024, 8:29 PM

FWIW, I think the License Ben is mentioning is different than GrowthBook's 'non-OSI but still open-source' license.

Everything else in GrowthBook is MIT-expat, but stuff in the packages/enterprise/ directory is 'The GrowthBook Enterprise License (the “Enterprise License”)' with the following text:

The GrowthBook Enterprise License (the “Enterprise License”)
Copyright (c) 2023 GrowthBook Inc.

With regard to the GrowthBook Software:

This software and associated documentation files (the "Software") may only be used in production, if you (and any entity that you represent) have > agreed to, and are in compliance with, the GrowthBook Subscription Terms of Service, available at https://www.growthbook.io/terms (the “Terms of > Service”), or other agreement governing the use of the Software, as agreed by you and GrowthBook, and otherwise have a valid GrowthBook Enterprise > or Pro license for the correct number of user seats. Subject to the foregoing sentence, you are free to modify this Software and publish patches to > the Software. You agree that GrowthBook and/or its licensors (as applicable) retain all right, title and interest in and
to all such modifications and/or patches, and all such modifications and/or patches may only be used, copied, modified, displayed, distributed, or > otherwise exploited with a valid GrowthBook Enterprise license for the correct number of user seats. Notwithstanding the foregoing, you may copy > and modify the Software for development and testing purposes, without requiring a
subscription. You agree that GrowthBook and/or its licensors (as applicable) retain all right, title and interest in and to all such modifications. > You are not granted any other rights beyond what is expressly stated herein. Subject to the foregoing, it is forbidden to copy, merge, publish, > distribute, sublicense, and/or sell the Software.

The full text of this Enterprise License shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, > FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER > LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER > DEALINGS IN THE SOFTWARE.

For all third party components incorporated into the GrowthBook Software, those components are licensed under the original license provided by the > owner of the applicable component.

I'd assume that GrowthBook intends their non-enterprise licensed stuff to be usable without the enterprise licensed stuff, so this seems like an oversight on their part. It might be worth asking them about this.

CodeReviewBot added a comment.Jul 18 2024, 1:53 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/ferretdb/-/merge_requests/1

Add the build pipeline for ferretdb

BTullis added a comment.Jul 18 2024, 2:03 PM

I've now got the ferretdb image ready as well. I will add submit requests to add both projects to the trusted runners, then start working on the helm chart for it.

CodeReviewBot added a comment.Jul 18 2024, 3:33 PM

btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/92

Add ferretdb and growthbook data-engineering projects to trusted runners

CodeReviewBot added a comment.Jul 18 2024, 3:44 PM

btullis closed https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/2

Draft: Modify the build process to exclude enterprise modules

CodeReviewBot added a comment.Jul 18 2024, 3:57 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/ferretdb/-/merge_requests/1

Add the build pipeline for ferretdb

CodeReviewBot added a comment.Jul 18 2024, 4:02 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/92

Add ferretdb and growthbook data-engineering projects to trusted runners

BTullis updated the task description. (Show Details)Jul 18 2024, 5:06 PM
gerritbot added a comment.Jul 19 2024, 8:40 AM

Change #1055385 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a PostgreSQL database for Growthbook

https://gerrit.wikimedia.org/r/1055385

gerritbot added a comment.Jul 19 2024, 9:03 AM

Change #1055385 merged by Btullis:

[operations/puppet@production] Add a PostgreSQL database for Growthbook

https://gerrit.wikimedia.org/r/1055385

BTullis updated the task description. (Show Details)Jul 19 2024, 9:18 AM
CodeReviewBot added a comment.Jul 19 2024, 9:20 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/1

Add the initial build pipeline for Growthbook

gerritbot added a comment.Jul 19 2024, 9:28 AM

Change #1055390 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add kubeconfig files for growthbook on dse-k8s

https://gerrit.wikimedia.org/r/1055390

gerritbot added a comment.Jul 19 2024, 9:36 AM

Change #1055391 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Add a growthbook namespace to the dse-k8s cluster

https://gerrit.wikimedia.org/r/1055391

CodeReviewBot added a comment.Jul 19 2024, 10:39 AM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/3

Configure the yarn proxy support

gerritbot added a comment.Jul 19 2024, 10:41 AM

Change #1055390 merged by Btullis:

[operations/puppet@production] Add kubeconfig files for growthbook on dse-k8s

https://gerrit.wikimedia.org/r/1055390

gerritbot added a comment.Jul 19 2024, 11:02 AM

Change #1055391 merged by jenkins-bot:

[operations/deployment-charts@master] Add a growthbook namespace to the dse-k8s cluster

https://gerrit.wikimedia.org/r/1055391

BTullis updated the task description. (Show Details)Jul 19 2024, 11:06 AM
gerritbot added a comment.Jul 19 2024, 11:19 AM

Change #1055417 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] [WIP] Create a new chart for growbook using scaffolding.

https://gerrit.wikimedia.org/r/1055417

CodeReviewBot added a comment.Jul 19 2024, 11:22 AM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/3

Configure the yarn proxy support

CodeReviewBot added a comment.Jul 19 2024, 1:07 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/4

Set the https-proxy as well as the plaintext version.

CodeReviewBot added a comment.Jul 19 2024, 1:15 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/4

Set the https-proxy as well as the plaintext version.

CodeReviewBot added a comment.Jul 19 2024, 2:38 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/5

Set npm proxy instead of yarn https-proxy

CodeReviewBot added a comment.Jul 19 2024, 2:46 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/5

Set npm proxy instead of yarn https-proxy

BTullis added a comment.Jul 19 2024, 3:15 PM

There is a slight problem with building the Growthbook image.

I had to set the proxy server for yarn to use on the trusted runners.
https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/3

That worked fine, but there is another step which is proving resistant to configuration.
This is because Next.js used in the front-end uses a node-fetch, which in turn uses a library called undici to retrieve something.

#52 61.15  | Attention: Next.js now collects completely anonymous telemetry regarding usage.
#52 61.15  | This information is used to shape Next.js' roadmap and prioritize features.
#52 61.15  | You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
#52 61.15  | https://nextjs.org/telemetry
#52 61.15  | 
#52 61.27  |    ▲ Next.js 14.1.0
#52 61.27  | 
#52 61.27  |    Skipping validation of types
#52 61.27  |    Skipping linting
#52 61.71  |    Creating an optimized production build ...
#52 62.33  |    Downloading swc package @next/swc-linux-x64-gnu...
#52 65.71  | TypeError: fetch failed
#52 65.71  |     at Object.fetch (node:internal/deps/undici/undici:11576:11)
#52 65.71  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
#52 65.71  |     at async extractBinary (/usr/src/growthbook/node_modules/next/dist/lib/download-swc.js:90:9)
#52 65.71  |     at async downloadNativeNextSwc (/usr/src/growthbook/node_modules/next/dist/lib/download-swc.js:151:9)
#52 65.71  |     at async tryLoadNativeWithFallback (/usr/src/growthbook/node_modules/next/dist/build/swc/index.js:286:5)
#52 65.71  |     at async /usr/src/growthbook/node_modules/next/dist/build/swc/index.js:270:40 {
#52 65.71  |   cause: AggregateError
#52 65.71  |       at internalConnectMultiple (node:net:1114:18)
#52 65.71  |       at internalConnectMultiple (node:net:1177:5)
#52 65.71  |       at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
#52 65.71  |       at listOnTimeout (node:internal/timers:575:11)
#52 65.71  |       at process.processTimers (node:internal/timers:514:7) {
#52 65.71  |     code: 'ETIMEDOUT',

Undici doesn't support automatic use of the http_proxy enviroinment variable (and friends) until version 6.14.0 - https://github.com/nodejs/undici/pull/2994
We are currently bound to major version 5 of undici by the yarn.lock file.

I tried setting proxy options via a couple of other methods, but nothing has worked yet.
I will look and see what other options we have of working around this.

phuedx added a comment.Jul 19 2024, 3:47 PM

A search for Next.js download SWC (from download-swc.js in that stacktrace you provided yielded: https://nextjs.org/docs/messages/failed-loading-swc

Quoting that page:

If SWC continues to fail to load you can opt-out by disabling swcMinify in your next.config.js or by adding a .babelrc to your project with the following content:

.babelrc
{
  "presets": ["next/babel"]
}
BTullis added a comment.Jul 19 2024, 3:55 PM
In T365839#9998908, @phuedx wrote:

A search for Next.js download SWC (from download-swc.js in that stacktrace you provided yielded: https://nextjs.org/docs/messages/failed-loading-swc

Quoting that page:

If SWC continues to fail to load you can opt-out by disabling swcMinify in your next.config.js or by adding a .babelrc to your project with the following content:

.babelrc
{
  "presets": ["next/babel"]
}

Great! Thanks @phuedx , I will try that now.

CodeReviewBot added a comment.Jul 19 2024, 4:12 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/6

Disable the use of SWC, as it fails to download on trusted runners

BTullis added a comment.Jul 19 2024, 4:17 PM

I'm not sure if it will work, because the message still says Downloading swc... but then it says SWC is disabled.

image.png (226×1 px, 61 KB)

I won't know until I merge to main, anyway.

CodeReviewBot added a comment.Jul 19 2024, 4:20 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/6

Disable the use of SWC, as it fails to download on trusted runners

BTullis added a comment.Jul 19 2024, 4:28 PM

Same issue, unfortunately. I also found this conversation: https://github.com/vercel/next.js/discussions/44959 which mentions being unable to prevent the downloading.

I'll keep looking for other options. Some people have had success by enabling a global proxy agent with setGlobalDispatcher e.g. https://github.com/vercel/next.js/issues/47039#issuecomment-2188970821

Maybe that's worth a go.

CodeReviewBot added a comment.Jul 19 2024, 5:00 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/7

Remove custom babel configuration

CodeReviewBot added a comment.Jul 19 2024, 5:09 PM

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/7

Attempt to set a proxy for undici on trusted runners

BTullis added a comment.Jul 19 2024, 5:18 PM

Hmm. I tried patching the growthbook front-end code with this gist: https://gist.github.com/zicklag/1bb50db6c5138de347c224fda14286da
...with the intention of using setGlobalDispatcher if the http_proxy environment variable is set.

Unfortunately, it didn't work. I suspect that I probably put it in the wrong place, so that it was ineffective. I'll check again.

brouberol subscribed.Jul 22 2024, 1:06 PM

I've taken an alternate approach: I've defined a small bash script that downloads the swc packages, extract them at the right location under node_modules, which causes the official download-swc.js script to skip the packages download. It's not beautiful, but it works.

gerritbot added a comment.Jul 22 2024, 1:50 PM

Change #1055943 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/dns@master] growthbook: add svc.eqiad and discovery records

https://gerrit.wikimedia.org/r/1055943

BTullis reassigned this task from BTullis to brouberol.Jul 22 2024, 1:52 PM
BTullis updated the task description. (Show Details)
BTullis subscribed.
gerritbot added a comment.Jul 22 2024, 1:52 PM

Change #1055943 merged by Brouberol:

[operations/dns@master] growthbook: add svc.eqiad and discovery records

https://gerrit.wikimedia.org/r/1055943

gerritbot added a comment.Jul 23 2024, 7:30 AM

Change #1055417 merged by Brouberol:

[operations/deployment-charts@master] Create a new chart for growbook using scaffolding.

https://gerrit.wikimedia.org/r/1055417

gerritbot added a comment.Jul 23 2024, 7:32 AM

Change #1056078 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: define helmfile and production values

https://gerrit.wikimedia.org/r/1056078

gerritbot added a comment.Jul 23 2024, 8:14 AM

Change #1056078 merged by Brouberol:

[operations/deployment-charts@master] growthbook: define helmfile and production values

https://gerrit.wikimedia.org/r/1056078

gerritbot added a comment.Jul 23 2024, 8:41 AM

Change #1056118 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: small fixes to the chart

https://gerrit.wikimedia.org/r/1056118

gerritbot added a comment.Jul 23 2024, 9:02 AM

Change #1056118 merged by Brouberol:

[operations/deployment-charts@master] growthbook: small fixes to the chart

https://gerrit.wikimedia.org/r/1056118

gerritbot added a comment.Jul 23 2024, 9:23 AM

Change #1056123 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: bind ferretdb service to 0.0.0.0 instead of localhost

https://gerrit.wikimedia.org/r/1056123

gerritbot added a comment.Jul 23 2024, 9:39 AM

Change #1056123 merged by Brouberol:

[operations/deployment-charts@master] growthbook: bind ferretdb service to 0.0.0.0 instead of localhost

https://gerrit.wikimedia.org/r/1056123

gerritbot added a comment.Jul 23 2024, 9:52 AM

Change #1056126 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: add mesh service, configuration, container and networkpolicies

https://gerrit.wikimedia.org/r/1056126

gerritbot added a comment.Jul 23 2024, 10:00 AM

Change #1056126 merged by Brouberol:

[operations/deployment-charts@master] growthbook: add mesh service, configuration, container and networkpolicies

https://gerrit.wikimedia.org/r/1056126

gerritbot added a comment.Jul 23 2024, 11:26 AM

Change #1056137 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: split chart into 2 charts (frontend/backend)

https://gerrit.wikimedia.org/r/1056137

gerritbot added a comment.Jul 23 2024, 12:27 PM

Change #1056147 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/dns@master] growthbook: provision DNS records for each backend/frontend service

https://gerrit.wikimedia.org/r/1056147

gerritbot added a comment.Jul 23 2024, 1:29 PM

Change #1056147 merged by Brouberol:

[operations/dns@master] growthbook: provision DNS records for each backend/frontend service

https://gerrit.wikimedia.org/r/1056147

gerritbot added a comment.Jul 23 2024, 1:38 PM

Change #1056137 merged by Brouberol:

[operations/deployment-charts@master] growthbook: split chart into 2 charts (frontend/backend)

https://gerrit.wikimedia.org/r/1056137

gerritbot added a comment.Jul 23 2024, 2:00 PM

Change #1056164 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: define one TLS hostname per subchart

https://gerrit.wikimedia.org/r/1056164

gerritbot added a comment.Jul 23 2024, 2:00 PM

Change #1056165 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: fix volume/configmap name problem

https://gerrit.wikimedia.org/r/1056165

gerritbot added a comment.Jul 23 2024, 3:32 PM

Change #1056164 merged by Brouberol:

[operations/deployment-charts@master] growthbook: define one TLS hostname per subchart

https://gerrit.wikimedia.org/r/1056164

gerritbot added a comment.Jul 23 2024, 3:35 PM

Change #1056165 merged by Brouberol:

[operations/deployment-charts@master] growthbook: fix volume/configmap name problem

https://gerrit.wikimedia.org/r/1056165

gerritbot added a comment.Jul 23 2024, 3:36 PM

Change #1056191 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: fix typo in tls hostname

https://gerrit.wikimedia.org/r/1056191

gerritbot added a comment.Jul 23 2024, 3:46 PM

Change #1056191 merged by Brouberol:

[operations/deployment-charts@master] growthbook: fix typo in tls hostname

https://gerrit.wikimedia.org/r/1056191

gerritbot added a comment.Jul 23 2024, 4:03 PM

Change #1056196 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: bump chart version to allow subchart upgrade

https://gerrit.wikimedia.org/r/1056196

gerritbot added a comment.Jul 23 2024, 4:05 PM

Change #1056196 merged by Brouberol:

[operations/deployment-charts@master] growthbook: bump chart version to allow subchart upgrade

https://gerrit.wikimedia.org/r/1056196

CodeReviewBot added a comment.Jul 24 2024, 9:35 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/12

Install mongodb in the growthbook image

CodeReviewBot added a comment.Jul 24 2024, 11:13 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/12

Install mongodb in the growthbook image

gerritbot added a comment.Jul 24 2024, 12:16 PM

Change #1056485 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: replace ferretdb by mongo itself

https://gerrit.wikimedia.org/r/1056485

gerritbot added a comment.Jul 25 2024, 7:26 AM

Change #1056485 merged by Brouberol:

[operations/deployment-charts@master] growthbook: replace ferretdb by mongo itself

https://gerrit.wikimedia.org/r/1056485

CodeReviewBot added a comment.Jul 25 2024, 8:14 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/14

Install mongosh, the mongoDB shell, in the image

CodeReviewBot added a comment.Jul 25 2024, 8:37 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/14

Install mongosh, the mongoDB shell, in the image

gerritbot added a comment.Jul 25 2024, 9:03 AM

Change #1056879 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: fix mongo connection string and update image

https://gerrit.wikimedia.org/r/1056879

gerritbot added a comment.Jul 25 2024, 9:23 AM

Change #1056879 merged by Brouberol:

[operations/deployment-charts@master] growthbook: fix mongo connection string and update image

https://gerrit.wikimedia.org/r/1056879

gerritbot added a comment.Jul 25 2024, 9:34 AM

Change #1056887 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: deploy mongodb with auth enabled

https://gerrit.wikimedia.org/r/1056887

gerritbot added a comment.Jul 25 2024, 11:09 AM

Change #1056910 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Use a 10 GB persistent volume for mongodb

https://gerrit.wikimedia.org/r/1056910

gerritbot added a comment.Jul 25 2024, 11:18 AM

Change #1056910 merged by jenkins-bot:

[operations/deployment-charts@master] Use a 10 GB persistent volume for mongodb

https://gerrit.wikimedia.org/r/1056910

gerritbot added a comment.Jul 25 2024, 11:41 AM

Change #1056887 merged by jenkins-bot:

[operations/deployment-charts@master] growthbook: deploy mongodb with auth enabled

https://gerrit.wikimedia.org/r/1056887

gerritbot added a comment.Jul 25 2024, 12:33 PM

Change #1056936 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Configure growthbook/mongodb deployment with a recreate strategy

https://gerrit.wikimedia.org/r/1056936

gerritbot added a comment.Jul 25 2024, 12:39 PM

Change #1056936 merged by jenkins-bot:

[operations/deployment-charts@master] Configure growthbook/mongodb deployment with a recreate strategy

https://gerrit.wikimedia.org/r/1056936

gerritbot added a comment.Jul 25 2024, 4:11 PM

Change #1056975 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the hostname for mongodb access

https://gerrit.wikimedia.org/r/1056975

gerritbot added a comment.Jul 25 2024, 4:15 PM

Change #1056975 merged by jenkins-bot:

[operations/deployment-charts@master] Update the hostname for mongodb access

https://gerrit.wikimedia.org/r/1056975

gerritbot added a comment.Jul 25 2024, 4:22 PM

Change #1056977 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Growthbook: disable aythmechanism PLAIN

https://gerrit.wikimedia.org/r/1056977

gerritbot added a comment.Jul 25 2024, 4:24 PM

Change #1056977 merged by jenkins-bot:

[operations/deployment-charts@master] Growthbook: disable aythmechanism PLAIN

https://gerrit.wikimedia.org/r/1056977

gerritbot added a comment.Jul 26 2024, 9:15 AM

Change #1057183 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] growthbook: small fixes to the values

https://gerrit.wikimedia.org/r/1057183

gerritbot added a comment.Jul 26 2024, 9:18 AM

Change #1057183 merged by Brouberol:

[operations/deployment-charts@master] growthbook: small fixes to the values

https://gerrit.wikimedia.org/r/1057183

brouberol added a comment.Jul 26 2024, 10:10 AM

We have been able to deploy an instance of Growthbook to the dse-k8s-eqiad Kubernetes cluster. To that effect, we have written a growthbook chart, which deploys the following components:

  • the growthbook frontend
  • the growthbook backend
  • a mongoDB database, persisting its data within our Ceph cluster, using a Persistent Volume

The application itself is not exposed to the outside world, through a wikimedia.org subdomain. If you want to have access to it locally, you need to perform several operations:

  1. Map growthbook.discovery.wmnet and growthbook-backend.discovery.wnnet to localhost in your /etc/hosts file. For example, mine looks like this
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost growthbook.discovery.wmnet growthbook-backend.discovery.wmnet
255.255.255.255	broadcasthost
::1             localhost
  1. Open 2 ssh tunnels to the Growthbook backend and frontend, by running
ssh -N -L 3000:growthbook.discovery.wmnet:30443 -L 3100:growthbook-backend.discovery.wmnet:30443 bast1003.wikimedia.org

You should then be able to access the application by connecting to https://growthbook.discovery.wmnet:3000, after having accepted the security risks (due to the fact that the application x509 certificate wasn't signed by a Certificate Authority (our own) trusted by your browser).

For now, the application isn't connected to any datasources.

image.png (2×6 px, 937 KB)

brouberol updated the task description. (Show Details)Jul 26 2024, 3:41 PM
brouberol added a subscriber: Milimetric.EditedJul 26 2024, 3:53 PM

One thing to consider is how Growthbook will be able to access Presto data via Kerberos in the future.

First, let's have a look at how things are configured for Superset (which also access Presto via Kerberos. Superset has a keytab allowing it to authenticate against Kerberos. Once it gets a TGT, it can then use it to connect to Presto. The data Presto uses is exposed via Hive, and stored in HDFS. When Superset performs a query, it does so by impersonating the UNIX user associated to the OpenID identity of the logged in Superset user. This means that a user with belonging to restricted groups will not be able to query certain tables via Superset.

Now, to Growthbook. Growthbook does not have a notion of identity impersonation. It gets access to Presto through its own UID/kerberos token, and that's it. This means either of 2 things:

  • Presto must be given analytics-privatedata-users access to be able to access all our data, so we're able to run experiments on everything
  • Presto must be given access to a specific HDFS directory, and data must be mirrored to this directory via regular jobs (Airflow, timers, etc).

After having talked with @Milimetric, it appears that the 2nd option is much more agreeable, as it respects our security policy of only having access to the data we need, for the extra price of having to maintain additional data jobs.

Assuming we go this way, I identified the following steps to setup Growthbook -> Presto communication and querying:

  • create a growthbook UNIX user on the analytics machines
  • create a specific UNIX group on analytics machines (say, analytics-poc)
  • add the growthbook user to the analytics-poc group
  • create hdfs:///wmf/data/proof_of_concepts, and chown it to growthbook:analytics-poc
  • change the uid of the growthbook containers runuser to the UID of the growthbook user
  • create a keytab for Growthbook, allowing it to connect to Presto via Kerberos
  • render the keytab into a kerberos-kinit sidecar for the growthbook-backend pod, using the usual dance
  • enable growthbook-->kerberos and growthbook-->presto traffic via external_services entries
brouberol added a comment.EditedJul 26 2024, 4:14 PM

This message contains the mongo commands we had to run to create the admin user/password, as well as the growthbook user/password.

First, we deploy mongo with Values.mongodb.auth.enabled: false. We then exec into the mongodb container and run

runuser@growthbook-backend-production-mongodb-7888bd9bd4-89mz5:/opt/growthbook$ /opt/mongodb/bin/mongosh
Current Mongosh Log ID:	66a21a900f1cfd4a9b482f8a
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.2.12
Using MongoDB:		7.0.12
Using Mongosh:		2.2.12

For mongosh info see: https://docs.mongodb.com/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.
test> use admin
admin> db.createUser(
... {
...   user: "myUserAdmin",
...   pwd: passwordPrompt(),
...   roles: [
...     { role: "userAdminAnyDatabase", db: "admin" },
...     { role: "readWriteAnyDatabase", db: "admin" }
...   ]
... }
... )
Enter password
**********************{ ok: 1 }
admin>

We use the mongodb_admin_password password defined in the private puppet repo.

Then, we redeploy the growthbook chart, this time with Values.mongodb.auth.enabled, which causes mongo to start with the --auth flag.

We re-exec into the mongo container, and run

runuser@growthbook-backend-production-mongodb-7888bd9bd4-89mz5:/opt/growthbook$ /opt/mongodb/bin/mongosh --host growthbook-backend-production-mongodb --port 27017 --username myUserAdmin
...
test> use admin
admin> db.createUser( { user: "growthbook", pwd: "redacted", roles: [ { role: "readWrite", db: "growthbook" }] } )
{ ok: 1 }

Here, we use the password defined in the chart secret, under MONGODB_PASSWORD.

NOTE: we had to use a fully alphanumeric password. Experience has shown that any non-alphanumeric character needs to be URI encoded, but seems to be causing issues at the mongooselevel (the nodejs mongo client library). By sticking to fully alphanumeric characters, we were able to bypass authentication issues. Go figure.

At that point, the backend should then be fully operational.

NOTE: if I were to do it all over again, I'd name the admin user admin and not myAdminUser, which was a copy/paste from the mongo docs.
brouberol updated the task description. (Show Details)Jul 26 2024, 4:17 PM
brouberol updated the task description. (Show Details)
brouberol added a comment.Jul 26 2024, 4:25 PM

For now, we have packaged the mongo binaries within the growthbook docker image itself, in order to minimize the amount of new things we ship to our registries. We install mongo using a tar.gz download and unpacking, which also avoids publishing a non-OSI deb package to our apt repos.

CodeReviewBot added a comment.Jul 26 2024, 4:52 PM

brouberol updated https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/15

Minimize the amount of layers needed to download mongodb

CodeReviewBot added a comment.Jul 29 2024, 7:22 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/growthbook/-/merge_requests/15

Minimize the amount of layers needed to download mongodb

brouberol closed this task as Resolved.Jul 29 2024, 7:24 AM
brouberol moved this task from In Progress to Done on the Data-Platform-SRE (2024.07.08 - 2024.07.28) board.
brouberol closed subtask T372620: Report on the initial growthbook installation PoC as Resolved.Aug 16 2024, 10:09 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits