Page MenuHomePhabricator

Allow for user implementation of extensions
Closed, ResolvedPublicFeature

Description

Feature summary (what you would like to be able to do and where):

It would be nice for third-party users to be able to install their own extensions, regardless of if its through code changes, through Composer inside the Docker container, through a .env file, or YML, or something else.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

I've tried two solutions to try to solve this myself (specifically for WikibaseLexeme), but no dice so far. I should note that I'm running this on a Windows 11 laptop and I'm running commands in Windows Powershell (I can add other version information if deemed necessary).

The first solution involved attempting to install in the Docker container itself using Composer. I was able to install Composer fine using the instructions on the Composer website, but running composer update results in an error along the lines of "does not comply with psr-4 autoloading standard".

The second potential solution was to add the the Dockerfile in ./build/Wikibase/, specifically:

(1) Adding "WikibaseLexeme" to the end of the ALL_EXTENSIONS argument on line 117.
(2) Adding "ARG WIKIBASELEXEME_COMMIT" to line 134.
(3) Running .\build.sh in the wikibase-release-pipeline directory.
(4) Running docker compose up --wait.

Unfortunately when this is run I get dependency failed to start: container wbs-deploy-wikibase-1 is unhealthy. While .\build.sh doesn't "fail" per se, it does open another shell that outputs something like:

2024/08/15 15:59:58 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
2024/08/15 15:59:58 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
2024/08/15 15:59:59 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
2024/08/15 15:59:59 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
2024/08/15 15:59:59 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
2024/08/15 15:59:59 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
2024/08/15 15:59:59 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
2024/08/15 15:59:59 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
sha256:[alphanumeric sequence]
sha256:[alphanumeric sequence]
sha256:[alphanumeric sequence]
sha256:[alphanumeric sequence]
sha256:[alphanumeric sequence]

Basically the same thing prints when removing the WikibaseLexeme content and re-running though.

Benefits (why should this be implemented?):

There are a lot of beneficial MediaWiki extensions that users have interest in using. Not all of them should be included in a general pipeline release-- but there should be an option to install them if people want to use them.

Event Timeline

As a simple test I:

(1) Installed a clean install of Docker.
(2) Downloaded a clean version of the wikibase-release-pipeline.
(3) Added "WikibaseQualityConstraints" to the end of the ALL_EXTENSIONS argument on line 117 so it reads:

ARG ALL_EXTENSIONS="Babel,cldr,CirrusSearch,ConfirmEdit,Elastica,EntitySchema,Nuke,OAuth,Scribunto,SyntaxHighlight_GeSHi,UniversalLanguageSelector,VisualEditor,WikibaseCirrusSearch,WikibaseManifest,WikibaseQualityConstraints"

(4) Added "ARG WIKIBASEQUALITYCONSTRAINTS_COMMIT" to line 132 (after WIKIBASEMANIFEST_COMMIT, to match its position in the list on line 117), so the list of extensions looks like this:

ARG BABEL_COMMIT
ARG CLDR_COMMIT
ARG CIRRUSSEARCH_COMMIT
ARG CONFIRMEDIT_COMMIT
ARG ELASTICA_COMMIT
ARG ENTITYSCHEMA_COMMIT
ARG NUKE_COMMIT
ARG OAUTH_COMMIT
ARG SCRIBUNTO_COMMIT
ARG SYNTAXHIGHLIGHT_GESHI_COMMIT
ARG UNIVERSALLANGUAGESELECTOR_COMMIT
ARG VISUALEDITOR_COMMIT
ARG WIKIBASECIRRUSSEARCH_COMMIT
ARG WIKIBASEMANIFEST_COMMIT
ARG WIKIBASEQUALITYCONSTRAINTS_COMMIT
ARG WIKIBASEEDTF_COMMIT
ARG WIKIBASELOCALMEDIA_COMMIT

I was worried that opening the Dockerfile and editing it in Visual Studio Code may have been causing the build.sh errors, so instead of adding these changes manually, I added them programmatically using a small script and verified the changes in a copy of the file.

(5) Ran build.sh wikibase in the wikibase-release-pipeline directory. This ran apparently successfully, as nothing was printed in console and it closed without apparent error of any kind.
(6) Ran docker compose up --wait in the deploy directory, which resulted in the following:

[+] Running 17/17
 ✔ Network wbs-deploy_default                    Created                                                               0.0s
 ✔ Volume "wbs-deploy_elasticsearch-data"        Created                                                               0.0s
 ✔ Volume "wbs-deploy_traefik-letsencrypt-data"  Created                                                               0.0s
 ✔ Volume "wbs-deploy_mysql-data"                Created                                                               0.0s
 ✔ Volume "wbs-deploy_wikibase-image-data"       Created                                                               0.0s
 ✔ Volume "wbs-deploy_quickstatements-data"      Created                                                               0.0s
 ✔ Volume "wbs-deploy_wdqs-data"                 Created                                                               0.0s
 ✔ Container wbs-deploy-traefik-1                Started                                                               1.1s
 ✔ Container wbs-deploy-elasticsearch-1          Started                                                               1.0s
 ✔ Container wbs-deploy-mysql-1                  Healthy                                                              11.5s
 ✘ Container wbs-deploy-wikibase-1               Error                                                                13.2s
 ✔ Container wbs-deploy-quickstatements-1        Created                                                               0.1s
 ✔ Container wbs-deploy-wikibase-jobrunner-1     Created                                                               0.1s
 ✔ Container wbs-deploy-wdqs-1                   Created                                                               0.1s
 ✔ Container wbs-deploy-wdqs-proxy-1             Created                                                               0.1s
 ✔ Container wbs-deploy-wdqs-updater-1           Created                                                               0.1s
 ✔ Container wbs-deploy-wdqs-frontend-1          Created                                                               0.1s

Is there anything else I could potentially try? Maybe I'm missing a line or not building correctly? Is there a different kind of output I should be seeing?

I will note this: after making any changes to the Dockerfile, even reverting them causes:

[+] Running 17/17
 ✔ Network wbs-deploy_default                    Created                                                               0.0s
 ✔ Volume "wbs-deploy_mysql-data"                Created                                                               0.0s
 ✔ Volume "wbs-deploy_traefik-letsencrypt-data"  Created                                                               0.0s
 ✔ Volume "wbs-deploy_wdqs-data"                 Created                                                               0.0s
 ✔ Volume "wbs-deploy_wikibase-image-data"       Created                                                               0.0s
 ✔ Volume "wbs-deploy_elasticsearch-data"        Created                                                               0.0s
 ✔ Volume "wbs-deploy_quickstatements-data"      Created                                                               0.0s
 ✔ Container wbs-deploy-traefik-1                Started                                                               0.9s
 ✔ Container wbs-deploy-mysql-1                  Healthy                                                              11.3s
 ✔ Container wbs-deploy-elasticsearch-1          Started                                                               0.9s
 ✘ Container wbs-deploy-wikibase-1               Error                                                                13.1s
 ✔ Container wbs-deploy-wdqs-1                   Created                                                               0.1s
 ✔ Container wbs-deploy-wikibase-jobrunner-1     Created                                                               0.1s
 ✔ Container wbs-deploy-quickstatements-1        Created                                                               0.1s
 ✔ Container wbs-deploy-wdqs-proxy-1             Created                                                               0.1s
 ✔ Container wbs-deploy-wdqs-updater-1           Created                                                               0.1s
 ✔ Container wbs-deploy-wdqs-frontend-1          Created                                                               0.1s
dependency failed to start: container wbs-deploy-wikibase-1 is unhealthy

The only solution I have found to this so far is to completely uninstall and reinstall Docker (see: https://phabricator.wikimedia.org/T372600). I am trying to see if there is some work-around to avoid this but I haven't found anything yet.

The complete reinstallation is not working anymore; documented this here: https://phabricator.wikimedia.org/T373138. The whole pipeline now seems to be broken and I can't get it up at all.

Got it back up and running (I think this was a certificates error, but it failed silently, not sure what's wrong), see: https://github.com/wmde/wikibase-release-pipeline/blob/main/deploy/README.md (under the section "Removing Wikibase Suite Completely with all its Data")

• roti_WMDE subscribed.

Hi @Superraptor123,

Thanks a lot for reporting this!

Custom Extensions

Allowing users to add extensions is definitely on our roadmap. Unfortunately, we are not there yet.

As an interim solution, I just tried it manually by rebuilding a customized wikibase/wikibase and succeeded to install WikibaseLexeme! Here is how I did it:

I did it on today's HEAD of the main branch. Note that this changed quite a bit from the latest stable release of deploy-3 due to a recent refactoring of our build and release system. The idea how to do that remains the same though.

  1. Add the commit SHA of the extension you want to add in build/wikibase/build.env. (In deploy-3, this would need to be added to variables.env). Is this the step you have skipped in your approach!?
diff --git a/build/wikibase/build.env b/build/wikibase/build.env
index 6050942..f20b359 100644
--- a/build/wikibase/build.env
+++ b/build/wikibase/build.env
@@ -61,6 +61,8 @@ UNIVERSALLANGUAGESELECTOR_COMMIT=752ea5965b7b93f4e14fa861d587b0966b15413d
 WIKIBASECIRRUSSEARCH_COMMIT=0d8f5907ea9f4274e28ea2707440b176b2d8c071
 # https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikibaseManifest/+/refs/heads/REL1_42
 WIKIBASEMANIFEST_COMMIT=5413c72af830a031fbf485b9c6b9e49057ac88c3
+# https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikibaseLexeme/+/refs/heads/REL1_42
+WIKIBASELEXEME_COMMIT=996bb21eff34c1e05148f36dbdb38c7934bae4e5
 
 # ##############################################################################
 # Community maintained extensions
  1. Add the extension in build/wikibase/Dockerfile:
diff --git a/build/wikibase/Dockerfile b/build/wikibase/Dockerfile
index c0a4ff3..9a8dec2 100644
--- a/build/wikibase/Dockerfile
+++ b/build/wikibase/Dockerfile
@@ -130,7 +130,7 @@ RUN set -x; \
     rm -f /var/www/html/extensions/Wikibase/.travis.yml && \
     find /var/www/html/extensions/Wikibase -name ".git*" -exec rm -rf {} +
 
-ARG ALL_EXTENSIONS="Babel,cldr,CirrusSearch,Elastica,EntitySchema,OAuth,UniversalLanguageSelector,WikibaseCirrusSearch,WikibaseManifest"
+ARG ALL_EXTENSIONS="Babel,cldr,CirrusSearch,Elastica,EntitySchema,OAuth,UniversalLanguageSelector,WikibaseCirrusSearch,WikibaseManifest,WikibaseLexeme"
 ARG BABEL_COMMIT
 ARG CLDR_COMMIT
 ARG CIRRUSSEARCH_COMMIT
@@ -142,6 +142,7 @@ ARG WIKIBASECIRRUSSEARCH_COMMIT
 ARG WIKIBASEMANIFEST_COMMIT
 ARG WIKIBASEEDTF_COMMIT
 ARG WIKIBASELOCALMEDIA_COMMIT
+ARG WIKIBASELEXEME_COMMIT
 
 SHELL ["/bin/bash", "-euo", "pipefail", "-c"]
 RUN set -x; \

(In deploy-3 you also need to add WIKIBASELEXEME_COMMIT to the docker build command in ./build.sh for wikibase in the same way the other build args are handled. On current main this is not required anymore, the whole build.env file will be available to the Dockerfile automatically.)

  1. Add a LocalSettings.php fragment in order to load the new extension:
diff --git a/build/wikibase/LocalSettings.d/50_WikibaseLexeme.php b/build/wikibase/LocalSettings.d/50_WikibaseLexeme.php
new file mode 100644
index 0000000..1fa695c
--- /dev/null
+++ b/build/wikibase/LocalSettings.d/50_WikibaseLexeme.php
@@ -0,0 +1,4 @@
+<?php
+
+# Wikibase Lexeme
+wfLoadExtension( 'WikibaseLexeme' );
  1. Rebuild wikibase/wikibase with ./nx build wikibase. (In deploy-3 this would be ./build.sh wikibase)
  2. Adjust deploy/docker-compose.yml in order to use your newly built wikibase/wikibase image.
diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml
index 336ffb2..d6e49d7 100644
--- a/deploy/docker-compose.yml
+++ b/deploy/docker-compose.yml
@@ -6,7 +6,7 @@ services:
   # --------------------------------------------------
 
   wikibase:
-    image: wikibase/wikibase:3
+    image: wikibase/wikibase:latest
     depends_on:
       mysql:
         condition: service_healthy
  1. Start your instance with docker compose up.

This made the extension show up on Special:Version:

image.png (1,209×293 px, 142 KB)

Logs

Please note that I am starting my stack with docker compose up omitting the --wait. This will print informative logs on the terminal and will probably ease the identification of problems.

Certificates

Certificate generation rate-limiting from LetsEncrypt should not happen when you stick to *example.com domains for your local setups. Traefik does give an error, but will continue with self-signed certificates automatically. I do not expect any issues here.

traefik-1             | 2024-09-06T07:42:43Z ERR Unable to obtain ACME certificate for domains error="cannot get ACME client acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-acct :: urn:ietf:params:acme:error:invalidContact :: Error creating new account :: invalid contact domain. Contact emails @example.com are forbidden" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["wikibase.example.com"] providerName=letsencrypt.acme routerName=wikibase@docker rule=Host(`wikibase.example.com`)

I hope that helps,
Best,
Robert

EDIT: Add step 3 that was missing before. Sorry for that!

So exciting! I just finished testing your solution re: https://phabricator.wikimedia.org/T372177, so I'm super excited to get this up and running! I'm beginning testing on my machine now! You're the best!

Okay so I'm testing the deploy-3 instructions now.

I've done the following, running on my Windows machine (using Powershell for CMD and Visual Studio Code as IDE):

  1. Adding to variables.env (yes, this is a step I missed-- it didn't come up during our meeting and had no idea that there was anything here to add re: extensions, but looking at it now, it should've been obvious!):
# https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikibaseLexeme/+/refs/heads/REL1_42
WIKIBASELEXEME_COMMIT=996bb21eff34c1e05148f36dbdb38c7934bae4e5
  1. Adding to build/wikibase/Dockerfile:
ARG ALL_EXTENSIONS="Babel,cldr,CirrusSearch,ConfirmEdit,Elastica,EntitySchema,Nuke,OAuth,Scribunto,SyntaxHighlight_GeSHi,UniversalLanguageSelector,VisualEditor,WikibaseCirrusSearch,WikibaseManifest,WikibaseLexeme"
ARG BABEL_COMMIT
ARG CLDR_COMMIT
ARG CIRRUSSEARCH_COMMIT
ARG CONFIRMEDIT_COMMIT
ARG ELASTICA_COMMIT
ARG ENTITYSCHEMA_COMMIT
ARG NUKE_COMMIT
ARG OAUTH_COMMIT
ARG SCRIBUNTO_COMMIT
ARG SYNTAXHIGHLIGHT_GESHI_COMMIT
ARG UNIVERSALLANGUAGESELECTOR_COMMIT
ARG VISUALEDITOR_COMMIT
ARG WIKIBASECIRRUSSEARCH_COMMIT
ARG WIKIBASEMANIFEST_COMMIT
ARG WIKIBASEEDTF_COMMIT
ARG WIKIBASELOCALMEDIA_COMMIT
ARG WIKIBASELEXEME_COMMIT
  1. Adding to ./build.sh:
function build_wikibase {
    docker build \
        --quiet \
        $DOCKER_BUILD_CACHE_OPT \
        --build-arg PHP_IMAGE_URL="$PHP_IMAGE_URL" \
        --build-arg COMPOSER_IMAGE_URL="$COMPOSER_IMAGE_URL" \
        --build-arg MEDIAWIKI_VERSION="$MEDIAWIKI_VERSION" \
        \
        --build-arg WIKIBASE_COMMIT="$WIKIBASE_COMMIT" \
        --build-arg BABEL_COMMIT="$BABEL_COMMIT" \
        --build-arg CLDR_COMMIT="$CLDR_COMMIT" \
        --build-arg BABEL_COMMIT="$BABEL_COMMIT" \
        --build-arg CLDR_COMMIT="$CLDR_COMMIT" \
        --build-arg CIRRUSSEARCH_COMMIT="$CIRRUSSEARCH_COMMIT" \
        --build-arg CONFIRMEDIT_COMMIT="$CONFIRMEDIT_COMMIT" \
        --build-arg ELASTICA_COMMIT="$ELASTICA_COMMIT" \
        --build-arg ENTITYSCHEMA_COMMIT="$ENTITYSCHEMA_COMMIT" \
        --build-arg NUKE_COMMIT="$NUKE_COMMIT" \
        --build-arg OAUTH_COMMIT="$OAUTH_COMMIT" \
        --build-arg SCRIBUNTO_COMMIT="$SCRIBUNTO_COMMIT" \
        --build-arg SYNTAXHIGHLIGHT_GESHI_COMMIT="$SYNTAXHIGHLIGHT_GESHI_COMMIT" \
        --build-arg UNIVERSALLANGUAGESELECTOR_COMMIT="$UNIVERSALLANGUAGESELECTOR_COMMIT" \
        --build-arg VISUALEDITOR_COMMIT="$VISUALEDITOR_COMMIT" \
        --build-arg WIKIBASECIRRUSSEARCH_COMMIT="$WIKIBASECIRRUSSEARCH_COMMIT" \
        --build-arg WIKIBASEMANIFEST_COMMIT="$WIKIBASEMANIFEST_COMMIT" \
        --build-arg WIKIBASEEDTF_COMMIT="$WIKIBASEEDTF_COMMIT" \
        --build-arg WIKIBASELOCALMEDIA_COMMIT="$WIKIBASELOCALMEDIA_COMMIT" \
        --build-arg WIKIBASELEXEME_COMMIT="$WIKIBASELEXEME_COMMIT" \
        \
        -t "$WIKIBASE_SUITE_WIKIBASE_IMAGE_URL" \
        \
        build/Wikibase

    mapfile -t tags < <(version_tags "wikibase")
    for tag in "${tags[@]}"; do
        docker tag "$WIKIBASE_SUITE_WIKIBASE_IMAGE_URL" "$WIKIBASE_SUITE_WIKIBASE_IMAGE_URL:$tag"
    done

}
  1. Creating a file called 50_WikibaseLexeme.php in the directory ./build/Wikibase/LocalSettings.d with the following contents:
<?php

# Wikibase Lexeme
wfLoadExtension( 'WikibaseLexeme' );
  1. Running ./build.sh wikibase. In Powershell, this opens up another CMD window (in my case, an instance of Git bash). This hangs for about a minute before printing out an alphanumeric string and then closing. Unfortunately it closes a bit too quick for me to see the contents of the string, but I assume that means it worked.
  1. Changing deploy/docker-compose.yml to latest:
services:
  # --------------------------------------------------
  # A. CORE WIKIBASE SUITE SERVICES
  # --------------------------------------------------

  wikibase:
    image: wikibase/wikibase:latest
  1. Starting with docker compose up in ./deploy fails with the following (this is the clipped tail of the log):
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T14:23:43,805Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "b828da98372f", "message": "[controller/237] [Main.cc@114] controller (64 bit): Version 7.10.2 (Build 40a3af639d4698) Copyright (c) 2021 Elasticsearch BV" }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T14:23:44,091Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "b828da98372f", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T14:23:44,602Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "docker-cluster", "node.name": "b828da98372f", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]" }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T14:23:44,643Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "b828da98372f", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
wikibase-1            | /entrypoint.sh: line 2: $'\r': command not found
wikibase-1            | /entrypoint.sh: line 4: $'\r': command not found
wikibase-1            | /entrypoint.sh: line 46: syntax error near unexpected token `$'do\r''
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T14:23:44,845Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "docker-cluster", "node.name": "b828da98372f", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
wikibase-1            | /entrypoint.sh: line 46: `    for var in "${req'ired_vars[@]}"; do
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T14:23:45,020Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "b828da98372f", "message": "initialized" }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T14:23:45,020Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "b828da98372f", "message": "starting ..." }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T14:23:45,166Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "b828da98372f", "message": "publish_address {172.18.0.2:9300}, bound_addresses {[::]:9300}" }
wikibase-1 exited with code 0
wikibase-1            | /entrypoint.sh: line 2: $'\r': command not found
wikibase-1            | /entrypoint.sh: line 4: $'\r': command not found
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 7/0
 - Container wbs-deploy-traefik-1             Stopping            0.1s
 ✔ Container wbs-deploy-wikibase-jobrunner-1  Stopped             0.0s
 ✔ Container wbs-deploy-quickstatements-1     Stopped             0.0s
 ✔ Container wbs-deploy-wdqs-frontend-1       Stopped             0.0s
 - Container wbs-deploy-elasticsearch-1       Stopping            0.1s
 ✔ Container wbs-deploy-wdqs-updater-1        Stopped             0.0s
 ✔ Container wbs-deploy-wdqs-proxy-1          Stopped             0.0s
 ✔ Container wbs-deploy-wdqs-1                Stopped             0.0s
 ✔ Container wbs-deploy-wikibase-1            Stopped             0.0s
 - Container wbs-deploy-mysql-1               Stopping            0.0s
dependency failed to start: container wbs-deploy-wikibase-1 is unhealthy

I think maybe this may have happened because I'm editing the files in Visual Studio Code and it might be adding Windows line endings. Will continue to investigate, but please advise if you think there's something I've missed or should change.

Thanks again!

Checked all files for line endings and there is not \r anywhere in any of the files I've edited.

Worse still the lines being pointed at in ./build/Wikibase/entrypoint.sh are:

  1. Line 2: Empty.
  2. Line 4: Empty.
  3. Line 46: for var in "${required_vars[@]}"; do. Seems fine, no apparent syntax error?

I'm going to try re-running the build.sh command (step 5 in my response comment) in Git Bash instead of Powershell and see if there is a difference. Will report back shortly.

Running ./build.sh wikibase in Git Bash prints out a SHA code and then exits basically immediately. Running docker compose up in Git Bash results in the same series of errors. So not a CMD issue it would seem.

Umm... so stopping and downing the containers and deleting everything and attempting to run the vanilla Wikibase (without any of the WikibaseLexeme bits added; fresh cloned repo and all) is now giving a similar error and not running:

elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T15:07:03,723Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "docker-cluster", "node.name": "86f008daf70e", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]" }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T15:07:03,759Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "86f008daf70e", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T15:07:03,965Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "docker-cluster", "node.name": "86f008daf70e", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
wikibase-1            | /entrypoint.sh: line 2: $'\r': command not found
wikibase-1            | /entrypoint.sh: line 4: $'\r': command not found
wikibase-1            | /entrypoint.sh: line 46: syntax error near unexpected token `$'do\r''
wikibase-1            | /entrypoint.sh: line 46: `    for var in "${req'ired_vars[@]}"; do
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T15:07:04,141Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "86f008daf70e", "message": "initialized" }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T15:07:04,141Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "86f008daf70e", "message": "starting ..." }
elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T15:07:04,311Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "86f008daf70e", "message": "publish_address {172.18.0.2:9300}, bound_addresses {[::]:9300}" }
wikibase-1 exited with code 0
wikibase-1            | /entrypoint.sh: line 2: $'\r': command not found
wikibase-1            | /entrypoint.sh: line 4: $'\r': command not found
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 7/0
 ✔ Container wbs-deploy-quickstatements-1     Stopped             0.0s
 - Container wbs-deploy-elasticsearch-1       Stopping            0.1s
 ✔ Container wbs-deploy-wikibase-jobrunner-1  Stopped             0.0s
 ✔ Container wbs-deploy-wdqs-frontend-1       Stopped             0.0s
 ✔ Container wbs-deploy-wdqs-updater-1        Stopped             0.0s
 - Container wbs-deploy-traefik-1             Stopping            0.1s
 ✔ Container wbs-deploy-wdqs-proxy-1          Stopped             0.0s
 ✔ Container wbs-deploy-wdqs-1                Stopped             0.0s
 ✔ Container wbs-deploy-wikibase-1            Stopped             0.0s
 - Container wbs-deploy-mysql-1               Stopping            0.1s
dependency failed to start: container wbs-deploy-wikibase-1 is unhealthy

So... umm now I'm up a creek a bit. Is there any way to get the original working? Has there been a change to deploy-3 that I'm running into?

Even completely uninstalling and reinstalling Docker does not help-- error still persists.

Triple-checking line endings in the file... they're all LF (\n) for ./build/Wikibase/entrypoint.sh (and every single other file in the repo-- I made sure by running dos2unix over the whole directory before running); so not sure where the issue is coming from:

entrypoint_sh_line_endings_v2.png (1,705×807 px, 196 KB)

entrypoint_sh_line_endings.png (1,080×315 px, 47 KB)

Error is the same btw:

elasticsearch-1       | {"type": "server", "timestamp": "2024-09-06T17:55:15,079Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "488cac0c49e7", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "VJS_Zk5uSou2c5JKoePHjg", "node.id": "LSaWEll7TpOlSO52J5MQDQ"  }
wikibase-1            | /entrypoint.sh: line 2: $'\r': command not found
wikibase-1            | /entrypoint.sh: line 4: $'\r': command not found
wikibase-1            | /entrypoint.sh: line 46: syntax error near unexpected token `$'do\r''
wikibase-1            | /entrypoint.sh: line 46: `    for var in "${req'ired_vars[@]}"; do
wikibase-1 exited with code 0

Hi @Superraptor123,

What an interesting issue. I think this could be happening: https://github.com/docker/for-win/issues/1340. When you inspect the file with your editors, it is still \LF, but as soon as docker build executes COPY it converts the file to \CR\LF.

Apparently there is a way to prevent that behavior of docker on windows: https://github.com/docker/for-win/issues/1340#issuecomment-716941493

So in your case you probably need to

diff --git a/build/wikibase/.gitattributes b/build/wikibase/.gitattributes
new file mode 100644
index 0000000..3130f72
--- /dev/null
+++ b/build/wikibase/.gitattributes
@@ -0,0 +1,7 @@
+# Ensure correct line endings
+*.sh   eol=lf
+*.env  eol=lf
+*.json eol=lf
+*.ini  eol=lf
+*.php  eol=lf
+

and ./build.sh again.

Hope that helps,
Best,
Robert

Robert,

You are a genius-- okay, I got it up and running by running the following in my Python script (this copies a .gitattributes file into the build/Wikibase directory, builds the image, and then later in the script, docker compose up is run):

# Copy .gitattributes file.
shutil.copy("./.gitattributes", wikibase_release_pipeline_dir+"/build/Wikibase/")

# Convert strings to Unix. Included just in case, but probably not needed. May be overkill.
convert_crlf_to_lf.convert(str(wd))

# Build Docker containers.
chdir(wikibase_release_pipeline_dir)
subprocess.run("./build.sh wikibase", shell=True)
chdir(wd)

The copied .gitattributes file has the following:

* text=auto
*.conf text eol=lf
Dockerfile  eol=lf
*.bat  eol=crlf
*.sh   eol=lf
*.env  eol=lf
*.json eol=lf
*.ini  eol=lf
*.php  eol=lf

Now that I have this running, I'll attempt to add extensions again given your previous instructions and will update here as I test!

Thanks for reporting back. Crossing fingers for the extensions!

Okay-- so I'm working through things and here's the what I've come against so far (noting this is just for WikibaseLexeme, as Kartographer and WikibaseQualityConstraints have more complicated requirements). So essentially I have a Python program that works like this (rough outline of the methods for use in deploy-3, not HEAD):

# Add extension(s) to build file (./variable.env)
wikibase_extensions.modify_variables_env(yaml_dict)

# Write Dockerfile with added extensions.
wikibase_extensions.modify_dockerfile(yaml_dict)

# Write build.sh with added extensions.
wikibase_extensions.modify_build_sh(yaml_dict)

# Add LocalSettings.php fragment(s).
wikibase_extensions.modify_localsettings(yaml_dict)

# Rebuild the Wikibase containers only.
wikibase_extensions.rebuild_wikibase_container(yaml_dict)

# Adjust docker-compose.yml.
wikibase_extensions.modify_docker_compose_yml(yaml_dict)

I have validated each of the methods and their respective files and everything runs like a charm (here's the docker ps as-is):

CONTAINER ID   IMAGE                        COMMAND                   CREATED          STATUS                    PORTS                                      NAMES
1697f70959da   wikibase/wdqs-frontend:1     "/entrypoint.sh ngin…"    19 minutes ago   Up 18 minutes (healthy)   80/tcp                                     wbs-deploy-wdqs-frontend-1
58f2380e7708   wikibase/wdqs-proxy:1        "/bin/sh -c \"/entryp…"   19 minutes ago   Up 18 minutes             80/tcp                                     wbs-deploy-wdqs-proxy-1
02a1f5ad9279   wikibase/wdqs:1              "/entrypoint.sh /run…"    19 minutes ago   Up 18 minutes                                                        wbs-deploy-wdqs-updater-1
2185c2399ec3   wikibase/wdqs:1              "/entrypoint.sh /run…"    19 minutes ago   Up 18 minutes (healthy)                                              wbs-deploy-wdqs-1
fa9b88845df8   wikibase/wikibase:3          "/bin/bash /jobrunne…"    19 minutes ago   Up 18 minutes             80/tcp                                     wbs-deploy-wikibase-jobrunner-1
36e56d3d1d34   wikibase/quickstatements:1   "/bin/bash /entrypoi…"    19 minutes ago   Up 18 minutes (healthy)   80/tcp                                     wbs-deploy-quickstatements-1
d34370db3100   wikibase/wikibase:latest     "/bin/bash /entrypoi…"    19 minutes ago   Up 19 minutes (healthy)   80/tcp                                     wbs-deploy-wikibase-1
57ee492d6de5   wikibase/elasticsearch:1     "/tini -- /usr/local…"    19 minutes ago   Up 19 minutes (healthy)   9200/tcp, 9300/tcp                         wbs-deploy-elasticsearch-1
934b385ed3a1   mariadb:10.11                "docker-entrypoint.s…"    19 minutes ago   Up 19 minutes (healthy)   3306/tcp                                   wbs-deploy-mysql-1
f76a1327a903   traefik:3.1                  "/entrypoint.sh --pr…"    19 minutes ago   Up 19 minutes             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   wbs-deploy-traefik-1

Which looks great and is running healthily (visiting https://wikibase.example.com/ resolves, no problems).

WikibaseLexeme is successfully listed at https://wikibase.example.com/wiki/Special:Version too!

I am having an unrelated issue resolving the SPARQL endpoint, but it doesn't appear to be related to the WikibaseLexeme install. Essentially, for WikibaseIntegrator, I'm getting a bunch of errors akin to:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://wikibase.example.com/proxy/wdqs/bigdata/namespace/wdq/sparql?query=%23Tool%3A+wdi_core+fastrun%0A%0A++++++++PREFIX+wd%3A+%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3E%0A++++++++PREFIX+wdt%3A+%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3E%0A%0A++++++++SELECT+%3Fp+WHERE+%7B%0A++++++++++++%3Fp+wdt%3AP2302+wd%3AQ21502410%0A++++++++%7D%0A++++++++&format=json

I checked in the browser, and sure enough, going to the endpoint (https://wikibase.example.com/proxy/wdqs/bigdata/namespace/wdq/sparql) presents me with:

Not Found
The requested URL was not found on this server.

Apache/2.4.59 (Debian) Server at wikibase.example.com Port 80

I then realized that the SPARQL endpoint may have been relocated as the hosts file was changed (re: https://phabricator.wikimedia.org/T372177). In .env the WDQS_FRONTEND_PUBLIC_HOST is set as the default wdqs-frontend.example.com. When visiting wdqs-frontend.example.com, I get DNS_PROBE_FINISHED_NXDOMAIN as the error in Chrome. Do I have to modify something to get access to the query service front-end or the SPARQL endpoint? Thanks so much!

Hey @roti_WMDE -- just following up on the above! Thanks so much!

• roti_WMDE reopened this task as Open.
• roti_WMDE moved this task from In Review to Done on the Wikibase Suite Team (Sprint-∞) board.
• roti_WMDE moved this task from Done to In Review on the Wikibase Suite Team (Sprint-∞) board.

Hi @Superraptor123,

Thanks a lot for reporting back.

Could you please open a new ticket for the SPARQL endpoint issue? So we can track it separately.

I will close this ticket now as I understand that the procedure worked for you. Feel free to reopen for any followups on the extension installation.

Best,
Robert

I tried this method and I am having an issue with the WDQS image.

By default, WDQS seems to have WIKIBASE_HOST=wikibase which is not as defined in the .env file.

This causes the updater to fail and not incorporate any modifications done to my Wikibase instance.

I understand this must be the Docker-internal hostname to communicate with the wikibase instance, but where else can I change the expected hostname?

This is the error:

org.wikidata.query.rdf.tool.rdf.Munger$BadSubjectException: Unrecognized subjects:  [https://wikibase.example/wiki/Special:EntityData/Q57, https://wikibase.example/entity/Q57].  Expected only sitelinks and subjects starting with http://wikibase/wiki/Special:EntityData/ and [http://wikibase/entity/]

Hi @Larjohn,

This is definitely an interesting issue. Could you please open a new ticket for that so that we can separately track that?

Thanks a lot,
Robert