Move CI coverage reports out of integration.wikimedia.org to a new domain or doc.wm.o
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Jul 7 2016, 3:39 PM

Description

CI has several web services for public consumption, they are all cohosted on gallium.wikimedia.org which is behind the misc varnish.

As part of the target architecture to phase out gallium (T133300) Jenkins/Zuul will be moved to scandium while the generated doc and coverage reports are to be moved to another host.

Apache proxy routing on Gallium to redirect to Jenkins and Zuul. The current state is:

CI Apache routes - v20160421.1.png (368×684 px, 33 KB)

Webservice	Hostname	Path	Description
CI Homepage Eg Zuul status page	integration	/	/srv/org/wikimedia/integration/docroot
Zuul status.json	integration	/zuul/status.json	Apache mod proxy to Zuul server daemon port 8001
Jenkins	integration	/ci/	Apache mod proxy to Jenkins daemon port 8080
Generated coverage	integration	/cover	/srv/org/wikimedia/integration/cover/
Generated doc	doc	/*	/srv/org/wikimedia/doc/

We are going to phase out gallium (T133300) and as a result split it in two parts:

scandium for Jenkins/Zuul
some webhost for documentation.

Hence the first three entries in the table above would stay on integration.wikimedia.org with misc varnish routing to scandium.eqiad.wmnet instead of gallium.wikimedia.org.

Generated doc on doc.wikimedia.org would be migrated to new host (that is not scandium).

The Generated coverage ( integration.wikimedia.org/cover ) can not be hosted on scandium and hence we have to migrate it out of integration.wikimedia.org.

End target

integration.wikimedia.org routed to scandium:

CI Homepage, Zuul status page
Zuul status.json (with mod proxy)
Jenkins interface (with mod proxy)

doc.wikimedia.org routed to a new host:

Existing doc unchanged
/cover/ moved to it (eg: doc.wikimedia.org/cover/) or we can come up with another domain.

Result of the task

Webservice	Hostname	Path	Description
CI Homepage Eg Zuul status page	integration	/	/srv/org/wikimedia/integration/docroot
Zuul status.json	integration	/zuul/status.json	Apache mod proxy to Zuul server daemon port 8001
Jenkins	integration	/ci/	Apache mod proxy to Jenkins daemon port 8080
Generated coverage	doc	/cover	/srv/org/wikimedia/doc/cover/
Generated doc	doc	/*	/srv/org/wikimedia/doc/

Coverage reports canonical place is now https://doc.wikimedia.org/cover/ ( https://integration.wikimedia.org/cover/ has 301 permanent redirect set).

Details

Subject	Repo	Branch	Lines +/-
doc: Fix Coverage page to inherit doc.wm.o, not integration.wm.o	integration/docroot	master	+4 -4
integration: Add slash to redirect from '/cover'.	integration/docroot	master	+4 -1
Migrate /cover/ under doc.wikimedia.org	integration/docroot	master	+2 -2
.htaccess to redirect /cover/ to doc.wm.o	integration/docroot	master	+1 -0
contint: migrate coverage report under doc.wm.o	operations/puppet	production	+1 -0
Migrate coverage reports to doc.wikimedia.org	integration/config	master	+6 -6

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dzahn	T123525 reduce amount of remaining Ubuntu 12.04 (precise) systems in production
Invalid	None	T124121 [keyresult] Migrate Jenkins to Jessie (gallium -> cobalt)
Resolved	Dzahn	T95757 Phase out gallium.wikimedia.org
Declined	None	T133150 Move gallium to an internal host?
Resolved	hashar	T133300 Target architecture without gallium.wikimedia.org
Resolved	hashar	T139620 Move CI coverage reports out of integration.wikimedia.org to a new domain or doc.wm.o

Event Timeline

hashar created this task.Jul 7 2016, 3:39 PM

Restricted Application added a subscriber: Zppix. · View Herald TranscriptJul 7 2016, 3:39 PM

hashar renamed this task from Split CI to Move CI coverage reports out of integration.wikimedia.org to a new domain or doc.wm.o.Jul 7 2016, 3:40 PM

From gallium, looks like all the coverage reports are static files:

$ find /srv/org/wikimedia/integration/cover -type f|rev|cut -d. -f1|rev|sort|uniq -c|sort -rn
   2332 html
    113 js
     53 css
     15 woff2
     15 woff
     15 ttf
     15 svg
     15 eot
      4 png
      2 php
      1 json
      1 htaccess

With the exception of our glue to browse them:

/srv/org/wikimedia/integration/cover/index.php
/srv/org/wikimedia/integration/cover/dir.php

In T139620#2437809, @hashar wrote:
From gallium, looks like all the coverage reports are static files:
$ find /srv/org/wikimedia/integration/cover -type f|rev|cut -d. -f1|rev|sort|uniq -c|sort -rn
   2332 html
    113 js
     53 css
     15 woff2
     15 woff
     15 ttf
     15 svg
     15 eot
      4 png
      2 php
      1 json
      1 htaccess
With the exception of our glue to browse them:

/srv/org/wikimedia/integration/cover/index.php
/srv/org/wikimedia/integration/cover/dir.php

From the looks of it, those files are just FancyIndexing. Would it be possible to simply replace those with either a static page that is periodically updated, or figure something out with mod_autoindex?

In T139620#2438947, @thcipriani wrote:

From the looks of it, those files are just FancyIndexing. Would it be possible to simply replace those with either a static page that is periodically updated, or figure something out with mod_autoindex?

Most probably yes. It all boils down to invoking the shared/Page.php thingy from https://phabricator.wikimedia.org/diffusion/CIDR/browse/master/shared/Page.php

it is static enough that we can probably extract the HTML to add header/footer for Apache FancyIndexing.

As a short term though, I think the easiest is to migrate the coverage report directly under the existing doc.wikimedia.org domain under /cover/. This way we do not have to figure out a new domain name, add it to misc varnish, add a new vhost etc..

So potentially we would:

HTTP 301 https://integration.wikimedia.org/cover(.*) to https://integration.wikimedia.org/cover$1
In integration/docroot.git move org/wikimedia/integration/cover to the doc root: org/wikimedia/doc/cover
Do the same on gallium
Apply puppet / reload Apache and blam

Then we find a new webhost place for generated material we can "just" migrate doc.wikimedia.org to the new host by changing the misc-varnish backend.

Change 298274 had a related patch set uploaded (by Hashar):
contint: migrate coverage report under doc.wm.o

https://gerrit.wikimedia.org/r/298274

Change 298275 had a related patch set uploaded (by Hashar):
Migrate /cover/ under doc.wikimedia.org

https://gerrit.wikimedia.org/r/298275

hashar mentioned this in rOPUPd94d7368b378: contint: migrate coverage report under doc.wm.o.Jul 11 2016, 1:12 PM

Change 298276 had a related patch set uploaded (by Hashar):
Migrate coverage reports to doc.wikimedia.org

https://gerrit.wikimedia.org/r/298276

Patches:

puppet	301 redirect	https://gerrit.wikimedia.org/r/298274
integration/docroot	Move the docroot	https://gerrit.wikimedia.org/r/298275
JJB conf	Update jobs to point to new docroot path	https://gerrit.wikimedia.org/r/298276

If the move from https://integration.wikimedia.org/cover/ to https://doc.wikimedia.org/cover/ is a good idea. We move the files on gallium, update the jobs then get the puppet patch to land to adjust apache conf. And I think it will be good enough for now.

hashar moved this task from Backlog to Doing on the Continuous-Integration-Infrastructure (phase-out-gallium) board.Jul 11 2016, 1:17 PM

In T139620#2447083, @hashar wrote:

As a short term though, I think the easiest is to migrate the coverage report directly under the existing doc.wikimedia.org domain under /cover/. This way we do not have to figure out a new domain name, add it to misc varnish, add a new vhost etc..

So potentially we would:

HTTP 301 https://integration.wikimedia.org/cover(.*) to https://integration.wikimedia.org/cover$1

In integration/docroot.git move org/wikimedia/integration/cover to the doc root: org/wikimedia/doc/cover

Do the same on gallium

Apply puppet / reload Apache and blam

Then we find a new webhost place for generated material we can "just" migrate doc.wikimedia.org to the new host by changing the misc-varnish backend.

Seems like a sane plan to me. Putting all generated content on the doc domain will make hosting generated content somewhere other than the future jenkins master much easier.

hashar mentioned this in rOPUP64a1b69d7f20: contint: migrate coverage report under doc.wm.o.Jul 12 2016, 11:13 AM

hashar mentioned this in rOPUPf6f3f7b97ee0: contint: migrate coverage report under doc.wm.o.

Thanks Tyler!

@Jdforrester-WMF +1 ed one of the change which I take has a functional approval :-]

I have tested the various changes on my local setup and that seems all good. Poking ops to have the Redirect set in place ( https://gerrit.wikimedia.org/r/298274 ) and I will merge the other patches, update Jenkins jobs and move the existing reports.

Change 298276 merged by jenkins-bot:
Migrate coverage reports to doc.wikimedia.org

https://gerrit.wikimedia.org/r/298276

Change 298275 merged by jenkins-bot:
Migrate /cover/ under doc.wikimedia.org

https://gerrit.wikimedia.org/r/298275

Change 298449 had a related patch set uploaded (by Hashar):
.htaccess to redirect /cover/ to doc.wm.o

https://gerrit.wikimedia.org/r/298449

Change 298449 merged by jenkins-bot:
.htaccess to redirect /cover/ to doc.wm.o

https://gerrit.wikimedia.org/r/298449

Change 298274 abandoned by Hashar:
contint: migrate coverage report under doc.wm.o

Reason:
Done via a .htaccess in the docroot https://gerrit.wikimedia.org/r/#/c/298449/ :)

Sorry for the spam!

https://gerrit.wikimedia.org/r/298274

I think it is done:

https://integration.wikimedia.org/cover/

Redirects to https://doc.wikimedia.org/cover/

Announced on QA and wikitech mailing lists pointing back to this task: https://lists.wikimedia.org/pipermail/qa/2016-July/002513.html

Thank you @thcipriani for the chat / suggestion and @Jdforrester-WMF for the +1 :)

hashar updated the task description. (Show Details)Jul 12 2016, 12:10 PM

hashar added a project: OKR-Work.

Rendering of https://integration.wikimedia.org/cover used to work, but now displays an Apache index. It works with trailing slash, but no longer without. We probably need to copy the slash-add hack from integration to doc.wikimedia.org.

Krinkle reopened this task as Open.Jul 19 2016, 8:08 PM

The magic redirect is produced by Apache via DirectorySlash On and that has been disabled for a year now via puppet patch 07c7f447ad3bb60940d812219411640ae696e95b / T95164

+    # T95164
+    DirectorySlash Off
+    RewriteCond %{REQUEST_FILENAME} -d
+    RewriteRule ^(.+[^/])$ https://doc.wikimedia.org/$1/ [R=301,QSA]

The regex explicitly exclude the trailing slash.

Change 305745 had a related patch set uploaded (by Krinkle):
integration: Add slash to redirect from '/cover'.

https://gerrit.wikimedia.org/r/305745

Change 305745 merged by jenkins-bot:
integration: Add slash to redirect from '/cover'.

https://gerrit.wikimedia.org/r/305745

Krinkle closed this task as Resolved.Aug 19 2016, 11:00 PM

Change 307102 had a related patch set uploaded (by Krinkle):
doc: Fix Coverage page to inherit doc.wm.o, not integration.wm.o

https://gerrit.wikimedia.org/r/307102

Change 307102 merged by jenkins-bot:
doc: Fix Coverage page to inherit doc.wm.o, not integration.wm.o