Page MenuHomePhabricator

Move CI coverage reports out of integration.wikimedia.org to a new domain or doc.wm.o
Closed, ResolvedPublic

Description

CI has several web services for public consumption, they are all cohosted on gallium.wikimedia.org which is behind the misc varnish.

As part of the target architecture to phase out gallium (T133300) Jenkins/Zuul will be moved to scandium while the generated doc and coverage reports are to be moved to another host.

Apache proxy routing on Gallium to redirect to Jenkins and Zuul. The current state is:

WebserviceHostnamePathDescription
CI Homepage Eg Zuul status pageintegration//srv/org/wikimedia/integration/docroot
Zuul status.jsonintegration/zuul/status.jsonApache mod proxy to Zuul server daemon port 8001
Jenkins integration/ci/Apache mod proxy to Jenkins daemon port 8080
Generated coverageintegration/cover/srv/org/wikimedia/integration/cover/
Generated docdoc/*/srv/org/wikimedia/doc/

We are going to phase out gallium (T133300) and as a result split it in two parts:

  1. scandium for Jenkins/Zuul
  2. some webhost for documentation.

Hence the first three entries in the table above would stay on integration.wikimedia.org with misc varnish routing to scandium.eqiad.wmnet instead of gallium.wikimedia.org.

Generated doc on doc.wikimedia.org would be migrated to new host (that is not scandium).

The Generated coverage ( integration.wikimedia.org/cover ) can not be hosted on scandium and hence we have to migrate it out of integration.wikimedia.org.

End target

integration.wikimedia.org routed to scandium:

  • CI Homepage, Zuul status page
  • Zuul status.json (with mod proxy)
  • Jenkins interface (with mod proxy)

doc.wikimedia.org routed to a new host:

  • Existing doc unchanged
  • /cover/ moved to it (eg: doc.wikimedia.org/cover/) or we can come up with another domain.

Result of the task

WebserviceHostnamePathDescription
CI Homepage Eg Zuul status pageintegration//srv/org/wikimedia/integration/docroot
Zuul status.jsonintegration/zuul/status.jsonApache mod proxy to Zuul server daemon port 8001
Jenkinsintegration/ci/Apache mod proxy to Jenkins daemon port 8080
Generated coveragedoc/cover/srv/org/wikimedia/doc/cover/
Generated docdoc/*/srv/org/wikimedia/doc/

Coverage reports canonical place is now https://doc.wikimedia.org/cover/ ( https://integration.wikimedia.org/cover/ has 301 permanent redirect set).

Details

Related Gerrit Patches:

Event Timeline

hashar created this task.Jul 7 2016, 3:39 PM
Restricted Application added a subscriber: Zppix. · View Herald TranscriptJul 7 2016, 3:39 PM
hashar renamed this task from Split CI to Move CI coverage reports out of integration.wikimedia.org to a new domain or doc.wm.o.Jul 7 2016, 3:40 PM
hashar added a comment.EditedJul 7 2016, 4:00 PM

From gallium, looks like all the coverage reports are static files:

$ find /srv/org/wikimedia/integration/cover -type f|rev|cut -d. -f1|rev|sort|uniq -c|sort -rn
   2332 html
    113 js
     53 css
     15 woff2
     15 woff
     15 ttf
     15 svg
     15 eot
      4 png
      2 php
      1 json
      1 htaccess

With the exception of our glue to browse them:

/srv/org/wikimedia/integration/cover/index.php
/srv/org/wikimedia/integration/cover/dir.php

From gallium, looks like all the coverage reports are static files:

$ find /srv/org/wikimedia/integration/cover -type f|rev|cut -d. -f1|rev|sort|uniq -c|sort -rn
   2332 html
    113 js
     53 css
     15 woff2
     15 woff
     15 ttf
     15 svg
     15 eot
      4 png
      2 php
      1 json
      1 htaccess

With the exception of our glue to browse them:
/srv/org/wikimedia/integration/cover/index.php
/srv/org/wikimedia/integration/cover/dir.php

From the looks of it, those files are just FancyIndexing. Would it be possible to simply replace those with either a static page that is periodically updated, or figure something out with mod_autoindex?

From the looks of it, those files are just FancyIndexing. Would it be possible to simply replace those with either a static page that is periodically updated, or figure something out with mod_autoindex?

Most probably yes. It all boils down to invoking the shared/Page.php thingy from https://phabricator.wikimedia.org/diffusion/CIDR/browse/master/shared/Page.php

it is static enough that we can probably extract the HTML to add header/footer for Apache FancyIndexing.


As a short term though, I think the easiest is to migrate the coverage report directly under the existing doc.wikimedia.org domain under /cover/. This way we do not have to figure out a new domain name, add it to misc varnish, add a new vhost etc..

So potentially we would:

  • HTTP 301 https://integration.wikimedia.org/cover(.*) to https://integration.wikimedia.org/cover$1
  • In integration/docroot.git move org/wikimedia/integration/cover to the doc root: org/wikimedia/doc/cover
  • Do the same on gallium
  • Apply puppet / reload Apache and blam

Then we find a new webhost place for generated material we can "just" migrate doc.wikimedia.org to the new host by changing the misc-varnish backend.

Change 298274 had a related patch set uploaded (by Hashar):
contint: migrate coverage report under doc.wm.o

https://gerrit.wikimedia.org/r/298274

Change 298275 had a related patch set uploaded (by Hashar):
Migrate /cover/ under doc.wikimedia.org

https://gerrit.wikimedia.org/r/298275

Change 298276 had a related patch set uploaded (by Hashar):
Migrate coverage reports to doc.wikimedia.org

https://gerrit.wikimedia.org/r/298276

Patches:

puppet301 redirecthttps://gerrit.wikimedia.org/r/298274
integration/docrootMove the docroothttps://gerrit.wikimedia.org/r/298275
JJB confUpdate jobs to point to new docroot path https://gerrit.wikimedia.org/r/298276

If the move from https://integration.wikimedia.org/cover/ to https://doc.wikimedia.org/cover/ is a good idea. We move the files on gallium, update the jobs then get the puppet patch to land to adjust apache conf. And I think it will be good enough for now.

As a short term though, I think the easiest is to migrate the coverage report directly under the existing doc.wikimedia.org domain under /cover/. This way we do not have to figure out a new domain name, add it to misc varnish, add a new vhost etc..
So potentially we would:

  • HTTP 301 https://integration.wikimedia.org/cover(.*) to https://integration.wikimedia.org/cover$1
  • In integration/docroot.git move org/wikimedia/integration/cover to the doc root: org/wikimedia/doc/cover
  • Do the same on gallium
  • Apply puppet / reload Apache and blam

Then we find a new webhost place for generated material we can "just" migrate doc.wikimedia.org to the new host by changing the misc-varnish backend.

Seems like a sane plan to me. Putting all generated content on the doc domain will make hosting generated content somewhere other than the future jenkins master much easier.

Thanks Tyler!

@Jdforrester-WMF +1 ed one of the change which I take has a functional approval :-]

I have tested the various changes on my local setup and that seems all good. Poking ops to have the Redirect set in place ( https://gerrit.wikimedia.org/r/298274 ) and I will merge the other patches, update Jenkins jobs and move the existing reports.

Change 298276 merged by jenkins-bot:
Migrate coverage reports to doc.wikimedia.org

https://gerrit.wikimedia.org/r/298276

Change 298275 merged by jenkins-bot:
Migrate /cover/ under doc.wikimedia.org

https://gerrit.wikimedia.org/r/298275

Change 298449 had a related patch set uploaded (by Hashar):
.htaccess to redirect /cover/ to doc.wm.o

https://gerrit.wikimedia.org/r/298449

Change 298449 merged by jenkins-bot:
.htaccess to redirect /cover/ to doc.wm.o

https://gerrit.wikimedia.org/r/298449

Change 298274 abandoned by Hashar:
contint: migrate coverage report under doc.wm.o

Reason:
Done via a .htaccess in the docroot https://gerrit.wikimedia.org/r/#/c/298449/ :)

Sorry for the spam!

https://gerrit.wikimedia.org/r/298274

hashar closed this task as Resolved.Jul 12 2016, 11:55 AM

I think it is done:

https://integration.wikimedia.org/cover/

Redirects to https://doc.wikimedia.org/cover/

Announced on QA and wikitech mailing lists pointing back to this task: https://lists.wikimedia.org/pipermail/qa/2016-July/002513.html

Thank you @thcipriani for the chat / suggestion and @Jdforrester-WMF for the +1 :)

hashar updated the task description. (Show Details)Jul 12 2016, 12:10 PM

Rendering of https://integration.wikimedia.org/cover used to work, but now displays an Apache index. It works with trailing slash, but no longer without. We probably need to copy the slash-add hack from integration to doc.wikimedia.org.

Krinkle reopened this task as Open.Jul 19 2016, 8:08 PM

The magic redirect is produced by Apache via DirectorySlash On and that has been disabled for a year now via puppet patch 07c7f447ad3bb60940d812219411640ae696e95b / T95164

+    # T95164
+    DirectorySlash Off
+    RewriteCond %{REQUEST_FILENAME} -d
+    RewriteRule ^(.+[^/])$ https://doc.wikimedia.org/$1/ [R=301,QSA]

The regex explicitly exclude the trailing slash.

Change 305745 had a related patch set uploaded (by Krinkle):
integration: Add slash to redirect from '/cover'.

https://gerrit.wikimedia.org/r/305745

Change 305745 merged by jenkins-bot:
integration: Add slash to redirect from '/cover'.

https://gerrit.wikimedia.org/r/305745

Krinkle closed this task as Resolved.Aug 19 2016, 11:00 PM

Change 307102 had a related patch set uploaded (by Krinkle):
doc: Fix Coverage page to inherit doc.wm.o, not integration.wm.o

https://gerrit.wikimedia.org/r/307102

Change 307102 merged by jenkins-bot:
doc: Fix Coverage page to inherit doc.wm.o, not integration.wm.o

https://gerrit.wikimedia.org/r/307102