Page MenuHomePhabricator

Clear /srv/.git on contint1001; move integration.wikimedia.org docroot to new location
Open, MediumPublic

Description

The website for https://integration.wikimedia.org is in integration/docroot.git which is cloned at /srv

/srv belongs to root:root and also is a place for different materials such as the jenkins build history or zuul git repositories.

I had to manually hack /srv to be owned by jenkins-slave. It should be made back to root:root AND the .git repo should be cloned one level deeper instead of directly in /srv. A good candidate would be /srv/docroot.

Event Timeline

hashar created this task.Nov 3 2016, 5:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2016, 5:35 PM
Paladox added a subscriber: Paladox.Nov 4 2016, 2:09 AM
hashar triaged this task as Low priority.Nov 18 2016, 3:12 PM
hashar moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.
hashar updated the task description. (Show Details)Dec 19 2018, 7:45 PM

Change 480828 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] scap configuration for deployment

https://gerrit.wikimedia.org/r/480828

Change 480879 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: clone integration/docroot

https://gerrit.wikimedia.org/r/480879

Change 480881 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: relocate from /srv to /srv/docroot

https://gerrit.wikimedia.org/r/480881

hashar claimed this task.Dec 19 2018, 11:00 PM

Will be addressed as we migrate the documentation hosting to a new host. I am intending to have integration/docroot.git to be cloned to /srv/docroot which can thus be owned by someone else than the root user.

Change 480879 merged by Dzahn:
[operations/puppet@production] doc: clone integration/docroot

https://gerrit.wikimedia.org/r/480879

Dzahn added a subscriber: Dzahn.Dec 20 2018, 12:56 AM

puppet created the dir and git clone contents after the merge above:

[doc1001:/srv/docroot] $ ls
total 60K
4.0K drwxrwsr-x 6 nobody wikidev 4.0K Dec 20 00:51 .
4.0K drwxr-xr-x 4 root   root    4.0K Dec 20 00:51 ..
4.0K -rw-rw-r-- 1 nobody wikidev  326 Dec 20 00:51 composer.json
4.0K drwxrwsr-x 8 nobody wikidev 4.0K Dec 20 00:51 .git
4.0K -rw-rw-r-- 1 nobody wikidev  356 Dec 20 00:51 .gitignore
4.0K -rw-rw-r-- 1 nobody wikidev   99 Dec 20 00:51 .gitreview
..

Change 480881 merged by Dzahn:
[operations/puppet@production] doc: relocate from /srv to /srv/docroot

https://gerrit.wikimedia.org/r/480881

Dzahn added a comment.Dec 20 2018, 6:09 PM

moved the files around on doc1001, but this one did not change things on contint* yet

I am leaving contint1001 as is :) The task would thus be solved once we switch doc.wikimedia.org hosting to doc1001.eqiad.wmnet.

Thanks for all the patches

https://doc.wikimedia.org/ is now served by doc1001.eqiad.wmnet. integration/docroot is cloned to /srv/docroot.

On contint1001 the old doc still has to be cleaned out, then we have to migrate https://integration.wikimedia.org/ document root to /srv/docroot as well.

hashar removed hashar as the assignee of this task.Mar 19 2019, 7:36 PM
Dzahn claimed this task.Apr 24 2019, 5:52 PM
Dzahn renamed this task from Clear /srv/.git on contint1001 to Clear /srv/.git on contint1001; move integration.wikimedia.org docroot to new location.Apr 24 2019, 6:06 PM
Dzahn removed Dzahn as the assignee of this task.Apr 24 2019, 6:09 PM
Dzahn added a comment.EditedMay 22 2020, 8:31 AM

As disussed on IRC, i'd rather not use /srv/docroot and would prefer instead if we can clean out the repo called "docroot" to actually just contain files for the docroot.

As opposed to extra files outside the docroot / mixing of doc.wm.org and integration.wm.org which got separated meanwhile and including more files from a common directory that is also outside of it.

Let's simplify and have a single docroot repo for a single site and then we can git clone to /srv/org/wikimedia/integration as expected to work at https://gerrit.wikimedia.org/r/c/operations/puppet/+/595525

During the migration attempts of contint2001 this popped up again when the integration.wikimedia.org docroot was empty and we are now living again with a manual hack that needs to be fixed properly.

Dzahn raised the priority of this task from Low to Medium.May 22 2020, 8:32 AM

Change 595525 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] contint: fix git cloning of docroot for integration.wm.org

https://gerrit.wikimedia.org/r/595525

Change 607041 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Stop autodeployment of integration/docroot

https://gerrit.wikimedia.org/r/607041

Change 595525 abandoned by Hashar:
contint: fix git cloning of docroot for integration.wm.org

Reason:
The way we deploy integration/docroot and maintain its deployment is broken. Instead we will migrate the deployment to use scap (T256005) and have Apache DocumentRoot to point to the deployed path under /srv/deployment. That will also addresses the issue Daniel raised to me: the git workspace is polluted by files published by CI.

https://gerrit.wikimedia.org/r/595525

Change 607076 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] ci: switch integration.wikimedia.org to scap DocumentRoot

https://gerrit.wikimedia.org/r/607076

For doc.wikimedia.org it is a bit scary. The org/wikimedia/doc path serves as a DocumentRoot and has plenty of script there for the homepage. We have CI populate the generated documentation directly into it.

I though I could Alias /doc/ to the scap deployed repository, but then the published documentation will no more be available. So I guess I have to find a solution using mod_rewrite to have file not found to be looked up in the directory under which files are published.

Eventually the published documentations should be relocated to a new place /srv/doc with bacula adjusted to use it as well as the rsync doc module.

It is not that complicated, but not that trivial. I fought a bit with mod_rewrite today but have hit a wall. So I guess I will try again later.

Change 607076 merged by Cwhite:
[operations/puppet@production] ci: switch integration.wikimedia.org to scap DocumentRoot

https://gerrit.wikimedia.org/r/607076

Change 611369 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] ci: switch integration.wikimedia.org to scap DocumentRoot

https://gerrit.wikimedia.org/r/611369

Change 611377 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] Support DOCUMENT_ROOT being a symbolic link

https://gerrit.wikimedia.org/r/611377

Change 611377 merged by jenkins-bot:
[integration/docroot@master] Support DOCUMENT_ROOT being a symbolic link

https://gerrit.wikimedia.org/r/611377

Mentioned in SAL (#wikimedia-operations) [2020-07-28T10:56:33Z] <hashar@deploy1001> Started deploy [integration/docroot@ba85bdf]: Catch up with HEAD and support DOCUMENT_ROOT being a symbolic link for T149924

Mentioned in SAL (#wikimedia-operations) [2020-07-28T10:56:40Z] <hashar@deploy1001> Finished deploy [integration/docroot@ba85bdf]: Catch up with HEAD and support DOCUMENT_ROOT being a symbolic link for T149924 (duration: 00m 06s)

Change 620368 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: switch to scap DocumentRoot

https://gerrit.wikimedia.org/r/620368

Change 620389 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: prepare /srv/doc as the new destination

https://gerrit.wikimedia.org/r/620389

Change 611369 merged by Dzahn:
[operations/puppet@production] ci: switch integration.wikimedia.org to scap DocumentRoot

https://gerrit.wikimedia.org/r/611369

Mentioned in SAL (#wikimedia-operations) [2020-08-20T19:07:22Z] <mutante> switching document root of integration.wikimedia.org to scap (T149924)

We switched the docroot to the scap location today. Is it done, @hashar?

No, still have to switch doc.wikimedia.org document root, but that requires a few more patches. It is not as trivial unfortunately.

Change 620389 merged by Jcrespo:
[operations/puppet@production] doc: prepare /srv/doc as the new destination

https://gerrit.wikimedia.org/r/620389

Change 621707 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: set WMF_DOC_PATH environment variable

https://gerrit.wikimedia.org/r/621707

Change 621720 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] Support published documents outside of the git checkout

https://gerrit.wikimedia.org/r/621720

Change 621707 merged by Dzahn:
[operations/puppet@production] doc: set WMF_DOC_PATH environment variable

https://gerrit.wikimedia.org/r/621707

Mentioned in SAL (#wikimedia-releng) [2020-08-24T07:40:46Z] <hashar> contint2001: removed checkout of integration/docroot.git which was at /srv # T149924

Change 622140 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] build: helpers to start php builtin server

https://gerrit.wikimedia.org/r/622140

Change 622323 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] shared: abstract out realpath check

https://gerrit.wikimedia.org/r/622323

Change 622606 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Stop autodeploy of integration/docroot

https://gerrit.wikimedia.org/r/622606

Change 622606 abandoned by Hashar:
[integration/config@master] Stop autodeploy of integration/docroot

Reason:
dupe of https://gerrit.wikimedia.org/r/c/integration/config/ /607041

https://gerrit.wikimedia.org/r/622606

Change 622140 merged by jenkins-bot:
[integration/docroot@master] build: run integration on a different port

https://gerrit.wikimedia.org/r/622140

Change 622323 merged by jenkins-bot:
[integration/docroot@master] shared: abstract out realpath check

https://gerrit.wikimedia.org/r/622323

Change 621720 merged by jenkins-bot:
[integration/docroot@master] Support published documents outside of the git checkout

https://gerrit.wikimedia.org/r/621720

Mentioned in SAL (#wikimedia-operations) [2020-09-07T13:27:57Z] <hashar@deploy1001> Started deploy [integration/docroot@e4e3af9]: Support published documents outside of the git checkout # T149924

Mentioned in SAL (#wikimedia-operations) [2020-09-07T13:28:03Z] <hashar@deploy1001> Finished deploy [integration/docroot@e4e3af9]: Support published documents outside of the git checkout # T149924 (duration: 00m 05s)

Change 625644 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: relocate published documents to /srv/doc

https://gerrit.wikimedia.org/r/625644

Change 625649 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: stop backup for old doc directory

https://gerrit.wikimedia.org/r/625649

Change 625650 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: remove legacy doc directory

https://gerrit.wikimedia.org/r/625650