Page MenuHomePhabricator
Create Task
Maniphest T237707

doc1001 permission problems for doc.wikimedia.org deploy
Closed, ResolvedPublic
Actions

Description

I've not been able to successfully deploy changes here for several months. There is always some permission problem or another.

Latest version:

$ ssh doc1001.eqiad.wmnet  git -C /srv/docroot pull
From https://gerrit.wikimedia.org/r/integration/docroot
   72d72d4..fb27369  master     -> origin/master
Updating 72d72d4..fb27369
warning: unable to unlink org/wikimedia/doc/favicon.php: Permission denied
error: unable to create file org/wikimedia/doc/favicon.ico: Permission denied

As such the following changes are (partly) undeployed, and the git status is dirty on the server:

Details

SubjectRepoBranchLines +/-
doc: stop using wikidev group, use doc-uploader groupoperations/puppetproduction+2 -4
doc: use doc-uploader group for docroot privs, stop using shared=>trueoperations/puppetproduction+2 -3
integration/docroot: Change post-merge message to recommend sudointegration/configmaster+1 -1
rsync: readd incoming and outgoing chmodoperations/puppetproduction+10 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krinkle created this task.Nov 8 2019, 2:18 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 8 2019, 2:18 AM
Peachey88 subscribed.Nov 8 2019, 8:20 AM
hashar added a comment.Nov 8 2019, 9:07 AM

The long story is at T235715#5588924

The issue originates from the migration of doc.wikimedia.org to a new server which was T137890: Relocate CI generated docs and coverage reports. It is pending a couple patches I have attached to that T137890:

hashar claimed this task.Nov 8 2019, 9:38 AM
hashar triaged this task as Medium priority.
hashar edited projects, added Release-Engineering-Team (CI & Testing services); removed Release-Engineering-Team.
hashar added parent tasks: T235715: Add Design Style Guide link to doc.wikimedia.org, T137890: Relocate CI generated docs and coverage reports.
gerritbot added a comment.Nov 8 2019, 9:44 AM

Change 484304 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] rsync: readd incoming and outgoing chmod

https://gerrit.wikimedia.org/r/484304

gerritbot added a project: Patch-For-Review.Nov 8 2019, 9:44 AM
Krinkle merged a task: T222767: integration/docroot error: unable to unlink old 'org/wikimedia/doc/default.html': Permission denied.Nov 25 2019, 11:12 PM
Krinkle added subscribers: thcipriani, • dduvall.
gerritbot added a comment.Dec 3 2019, 8:10 PM

Change 484304 merged by Dzahn:
[operations/puppet@production] rsync: readd incoming and outgoing chmod

https://gerrit.wikimedia.org/r/484304

Dzahn subscribed.Dec 3 2019, 8:16 PM

Both patches are merged now. Git status of /srv/docroot is clean.

I ran git pull as user krinkle:

krinkle@doc1001:/srv/docroot$ git -C /srv/docroot pull
Already up-to-date.
Dzahn added a comment.Dec 3 2019, 8:16 PM

Resolved?

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 9:10 PM
Krinkle closed this task as Resolved.Dec 3 2019, 9:17 PM

Looks like it. I'll find out next time but git-pull works fine now, but that already worked when there were no changes.

Krinkle reopened this task as Open.Dec 22 2019, 10:46 PM

Next time was today, and it's not resolved.

Direct command
$ ssh doc1001.eqiad.wmnet git -C /srv/docroot pull
From https://gerrit.wikimedia.org/r/integration/docroot
   de29f03..77493ca  master     -> origin/master
Updating de29f03..77493ca
error: unable to create symlink org/wikimedia/doc/favicon.ico: Permission denied
Remote shell
krinkle@doc1001:/srv/docroot$ sudo -u doc-uploader git pull      
error: cannot open .git/FETCH_HEAD: Permission denied
Krinkle added a comment.Jan 28 2020, 11:16 PM

https://doc.wikimedia.org/favicon.ico is still 404 Not Found.

Krinkle renamed this task from doc1001 permission problems to doc1001 permission problems for doc.wikimedia.org deploy.Jan 29 2020, 3:33 PM
Krinkle merged a task: T243905: doc.wikimedia.org doc root deployment seems broken.
Krinkle added a subscriber: Addshore.
gerritbot added a comment.Jan 29 2020, 8:54 PM

Change 568600 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] integration/docroot: Change post-merge message to recommend sudo

https://gerrit.wikimedia.org/r/568600

gerritbot added a project: Patch-For-Review.Jan 29 2020, 8:54 PM

Change 568600 merged by jenkins-bot:
[integration/config@master] integration/docroot: Change post-merge message to recommend sudo

https://gerrit.wikimedia.org/r/568600

Maintenance_bot removed a project: Patch-For-Review.Jan 29 2020, 9:10 PM
Stashbot added a comment.Feb 3 2020, 6:58 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-03T18:58:12Z] <mutante> < bblack> !log doc1001: chown -R nobody:wikidev /srv/docroot | < mutante> !doc1001 sudo -u doc-uploader chmod g+w /srv/docroot/org/wikimedia/doc | https://gerrit.wikimedia.org/r/c/operations/puppet/+/484304 | (T237707)

Stashbot added a comment.Feb 3 2020, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-03T19:19:42Z] <mutante> doc1001 - chown -R doc-uploader:doc-uploader /srv/docroot ; temp. disabled puppet (T237707)

gerritbot added a comment.Feb 3 2020, 7:33 PM

Change 569620 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] doc: use doc-uploader group for docroot privs, stop using shared=>true

https://gerrit.wikimedia.org/r/569620

gerritbot added a project: Patch-For-Review.Feb 3 2020, 7:33 PM
gerritbot added a comment.Feb 3 2020, 8:09 PM

Change 569620 merged by Dzahn:
[operations/puppet@production] doc: use doc-uploader group for docroot privs, stop using shared=>true

https://gerrit.wikimedia.org/r/569620

Maintenance_bot removed a project: Patch-For-Review.Feb 3 2020, 8:11 PM
Stashbot added a comment.Feb 3 2020, 8:13 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-03T20:13:56Z] <mutante> doc1001 - re-enabled puppet after merging gerrit:569620 - Git::Clone[integration/docroot]/File[/srv/docroot]/mode: mode changed '2775' to '0755' - Profile::Doc/File[/srv/docroot/org/wikimedia/doc]/group: group changed 'doc-uploader' to 'wikidev', mode changed '0775' to '0755'. needs another follow-up (T237707)

gerritbot added a comment.Feb 3 2020, 8:30 PM

Change 569637 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] doc: stop using wikidev group, use doc-uploader group

https://gerrit.wikimedia.org/r/569637

gerritbot added a project: Patch-For-Review.Feb 3 2020, 8:30 PM
gerritbot added a comment.Feb 3 2020, 8:41 PM

Change 569637 merged by Dzahn:
[operations/puppet@production] doc: stop using wikidev group, use doc-uploader group

https://gerrit.wikimedia.org/r/569637

Dzahn added a comment.Feb 3 2020, 8:49 PM

We ran into permission problems again today. The latest attempt to fix is:

  • stop using the 'wikidev' group in general (default and used too widely)
  • stop setting "shared => true" with git::clone (not needed if we all (and puppet!) use the same user to git clone)
  • stop including umask change for wikidev user (reduce complexity, not needed anymore)
  • doc-uploader is also a group, so doc-uploader:doc-uploader is used as owner for everything below /src/docroot now
  • users are expected to deploy with "sudo -u doc-uploader" as it happened today

Before it was possible to deploy as doc-uploader or as regular user and differences in docs kept causing issues.

Dzahn added a comment.Feb 3 2020, 8:57 PM

So please all deploy with "sudo -u doc-uploader git -C /srv/docroot/ pull" and hopefully that resolves the ticket for real now.

All members of contint-admins have privileges to run any command as doc-uploader'.

Maintenance_bot removed a project: Patch-For-Review.Feb 3 2020, 9:10 PM
WMDE-leszek mentioned this in T243392: Move storybook to doc.wikimedia.org.Feb 5 2020, 10:37 AM
Jdforrester-WMF closed this task as Resolved.Mar 5 2020, 12:00 AM
hashar added a comment.Mar 24 2021, 9:37 AM

The permission madness has been finally addressed by moving deployment of integration/docroot.git to use scap and moving the published doc to a disconnected directory (/srv/doc). There is thus no more any permission clashes.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL