Page MenuHomePhabricator
Create Task
Maniphest T237248

Convert wdqs docker image to not run blazegraph as root
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Currently the wdqs docker images in the wikibase-docker repo run blazegraph as root and should not.

A new user "blazegraph" should be created in the docker image and used for running the blazegraph process.
In order to remain backward compatible the entry point may have to have checks in place running as root to alter file permissions of files such as the wdqs journal file.
The final blazegraph process however should run as the new user "blazegraph".

Acceptance criteria

  • blazegraph process is not run as root, instead run as the new "blazegraph" user
  • Users previously running blazegraph as root should not have any issues when upgrading to new image (with file permissions etc)
  • All wdqs docker image version are updated
  • Follow best practices for creating users and groups in docker images, this probably means specifying the user ID and group ID when creating the user

Related Objects

Event Timeline

Addshore created this task.Nov 4 2019, 11:54 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptNov 4 2019, 11:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.Nov 4 2019, 11:54 AM
Addshore mentioned this in T234644: Wikimedia Technical Conference 2019 Session: Release "strategies" for MediaWiki and other elements of Wikimedia platform, for safe and efficient deployment and hosting.Nov 4 2019, 2:37 PM
Addshore triaged this task as Medium priority.Nov 5 2019, 12:02 PM
Addshore updated the task description. (Show Details)Nov 5 2019, 1:16 PM
Addshore set the point value for this task to 8.
Addshore moved this task from Ready to estimate to Ready to pick up on the Wikidata-Campsite board.
Addshore moved this task from incoming to consider for next sprint on the Wikidata board.Nov 5 2019, 2:10 PM
Addshore moved this task from Backlog to Doing on the Wikibase-Docker-2017+ board.Nov 6 2019, 8:48 AM
Ladsgroup claimed this task.Dec 16 2019, 1:23 PM
Ladsgroup edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Ladsgroup subscribed.

I have been doing this :D

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptDec 16 2019, 1:23 PM
Ladsgroup added a comment.Dec 16 2019, 2:24 PM

This is mostly done except for existing volumes part. No matter what I do, an existing volume still is owned as root and causing the whole thing to fail:

amsa@amsa-Latitude-7480:~/wikibase-docker$ docker-compose run wdqs /bin/bash
bash-4.4$ ls -l
total 84916
-rw-r--r--    1 blazegra blazegra      3252 Dec 13 12:01 RWStore.properties
-rw-r--r--    1 blazegra blazegra  79796363 Nov 27 12:44 blazegraph-service-0.3.10-SNAPSHOT.war
-rwxrwxr-x    1 blazegra blazegra       448 Jul 17 10:42 createNamespace.sh
drwxr-xr-x    2 root     root          4096 Jul  9 15:56 data
-rw-rw-r--    1 blazegra blazegra      1483 Jul 17 10:42 default.properties
drwxrwxr-x    1 blazegra blazegra      4096 Aug 22 10:09 docs
-rwxrwxr-x    1 blazegra blazegra       344 Jul 17 10:42 forAllCategoryWikis.sh
-rw-rw-r--    1 blazegra blazegra   7074499 Jul 17 11:20 jetty-runner-9.4.12.v20180830.jar
-rw-rw-r--    1 blazegra blazegra      2202 Jul 17 10:42 ldf-config.json
drwxr-xr-x    1 blazegra blazegra      4096 Nov 27 12:52 lib
-rwxrwxr-x    1 blazegra blazegra      1133 Jul 17 10:42 loadCategoryDaily.sh
-rwxrwxr-x    1 blazegra blazegra       949 Jul 17 10:42 loadCategoryDump.sh
-rwxrwxr-x    1 blazegra blazegra       824 Jul 17 10:42 loadData.sh
-rwxrwxr-x    1 blazegra blazegra      1345 Jul 17 10:42 loadRestAPI.sh
-rwxr-xr-x    1 blazegra blazegra       833 Nov 13 16:01 munge.sh
-rw-r--r--    1 blazegra blazegra      2431 Dec 16 14:11 mwservices.json
-rw-rw-r--    1 blazegra blazegra         0 Jul 17 10:42 prefixes.conf
drwxrwxr-x    1 blazegra blazegra      4096 Jul 17 10:42 queries
-rwxrwxr-x    1 blazegra blazegra      3603 Jul 17 10:42 runBlazegraph.sh
-rwxr-xr-x    1 blazegra blazegra      2656 Nov 13 16:01 runUpdate.sh
-rw-rw-r--    1 blazegra blazegra      2901 Jul 17 10:42 services.json
-rw-r--r--    1 blazegra blazegra      2721 Dec 13 12:01 whitelist.txt

Any ideas?

The current WIP is https://github.com/wmde/wikibase-docker/pull/106

Some notes: https://github.com/moby/moby/issues/2259 and https://github.com/docker/compose/issues/5507

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 16 2019, 7:08 PM

Okay after talking to Adam about it in mattermost, We decided to go with the solution of running entry point as root and then changing to not-root later.

Ready for review now.

Addshore moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 7 2020, 10:02 AM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.Jan 7 2020, 10:15 AM
WMDE-leszek subscribed.Jan 9 2020, 3:20 PM

@Addshore: with https://github.com/wmde/wikibase-docker/pull/106 merged, is this done, or does it still require some work?

Addshore added a comment.Jan 14 2020, 7:51 PM

It should be verified that's the newly built by CI docker image works as expected.

Addshore closed this task as Resolved.Feb 6 2020, 9:49 AM
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.Feb 6 2020, 10:15 AM
Addshore reopened this task as Open.Apr 20 2020, 1:41 PM
Addshore removed Ladsgroup as the assignee of this task.
Addshore moved this task from Test (Verification) to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Reverted in https://github.com/wmde/wikibase-docker/pull/117 as this was broken for both images

Addshore claimed this task.May 20 2020, 9:22 AM
Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Addshore. · View Herald TranscriptMay 20 2020, 9:22 AM
Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.May 27 2020, 12:20 PM

https://github.com/wmde/wikibase-docker/pull/125

Addshore moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jun 3 2020, 9:18 AM
Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jun 3 2020, 1:44 PM
Addshore moved this task from consider for next sprint to in progress on the Wikidata board.Jun 11 2020, 10:12 AM
Michael moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jun 17 2020, 1:00 PM
Michael subscribed.

Pull request merged, currently is being built by CI

Tarrow closed this task as Resolved.Jun 30 2020, 4:45 PM
Tarrow moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Tarrow subscribed.

Happy to say this appears to be done.
Tested by:

  • Checking user inside container using docker-compose run
  • checking the image starts and the updater was able to write then I was able to query
  • Breaking the permissions on the volume
  • running the image again and seeing it correctly chown'ed
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL