[Task] move wikiba.se webhosting to wikimedia misc-cluster
Open, Stalled, NormalPublic

Description

Move wikiba.se webhosting to wikimedia misc-cluster. For that we need a way to easily change the content of the site (i.e. deployment), perferably automated after git merge. A post merge hook with jenkins that deploys the site might work.

See also: https://github.com/wmde/Wikiba.se/issues/19

JanZerebecki updated the task description. (Show Details)
JanZerebecki raised the priority of this task from to Normal.
JanZerebecki added a project: Wikidata.
JanZerebecki added a subscriber: JanZerebecki.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 18 2015, 3:29 PM
JanZerebecki moved this task from incoming to ready to go on the Wikidata board.May 18 2015, 3:29 PM
Jonas renamed this task from move wikiba.se webhosting to wikimedia misc-cluster to [Task] move wikiba.se webhosting to wikimedia misc-cluster.Aug 14 2015, 7:46 AM
Jonas set Security to None.
hoo added a comment.Oct 18 2016, 4:40 PM

This implies moving the code to gerrit, doesn't it? That sounds good to me, just want to make sure we're on the same page here.

Jonas added a subscriber: Jonas.Oct 18 2016, 7:26 PM

I don't think it is necessary to move the repo, but I think it would be nice to have a real machine not just FTP.

So I would like to setup a machine and then change the CNAME for wikiba.se to this machine.
On this machine we can then automatically pull the repo and create Doxygen and JSDoc documentation.

hoo added a comment.Oct 19 2016, 3:23 PM

I don't think it is necessary to move the repo, but I think it would be nice to have a real machine not just FTP.

I was talking about moving this to the wikimedia misc cluster (as stated above). I think we should avoid running our own services, if we can.

In T99531#2728995, @hoo wrote:

I don't think it is necessary to move the repo, but I think it would be nice to have a real machine not just FTP.

I was talking about moving this to the wikimedia misc cluster (as stated above). I think we should avoid running our own services, if we can.

+1

The maintenance cost of running this is close to 0. I don't see how depending on a third party, WMF, is going to improve things. Same for moving the Git repo. We have full control over it now, no need to coordinate anything with people outside of WMDE.

That said, I think it'd be nice to have the stuff more automated. Given the little changes to this code I'm not sure it's worth anyones time to actually do that though.

Probably it should go to krypton. I'll ask Ops.

faidon added a subscriber: faidon.Jul 21 2017, 7:30 AM

Hi there! I saw @Ladsgroup's email to the ops list (thanks for bringing that to our attention!), so I'll respond to some of the questions he raised there -- sorry if it sounds a bit incoherent with regards to the context above :)

From an infrastructure point-of-view, hosting this microsite is definitely doable and from the looks of it, pretty easy too. If you folks think that it would better serve our movement and/or make your lives easier, we're up for it.

That said, all of my comments below are from a technical/infrastructure point-of-view. There may be other requirements or consequences to the WMF hosting this site, so I'd have to check with Legal about them (I can do so if/when you decide you want to actually proceed with it). For example, there may be a requirement to have a footer that mentions the Foundation's privacy policy.

Where it'd be hosted in our infrastructure doesn't matter much at this point I think, but it'll probably something like krypton or bromine. We'd likely want it to be fronted by our traffic infrastructure, although there are some complications around that (mainly due to the fact that it's not a *.wikimedia.org, so our regular wildcard HTTPS certificate wouldn't cover it), so we may want to do that as a second step at some later point. It'd be useful to know the order of magnitude of traffic this site receives.

Amir inquired about DNS as well; for that, we'd like the Foundation to own (and pay for) the domain, as well as the domain being served by our nameservers (ns0/1/2.wikimedia.org). This isn't a no exception rule, but a very strong preference. We've hosted sites without operating the DNS, or operating the DNS for domains the Foundation didn't own, and we probably still do that in some cases, but it has always been messy, especially over the years (think of cases we want to e.g. make a change to our NS or MX records and and we need to find whoever is responsible for N number of domains across multiple organizations and staff that may have turned around and lost context).

Finally, Amir asked about whether a security review would be needed, considering this is a static site. It's a bordeline case but there may be bugs for example lying in e.g. client-side code (such as XSS), so it's probably not a bad idea to pass it through a security review regardless? Since it's small and frontend-only, It'll be probably be super quick and won't introduce any delays to this move :)

Let me know if/when you'd like to proceed and then we can sort out the remaining questions on our side. In that case, feel free to tag this or another task with Operations :)

greg added a subscriber: greg.Jul 21 2017, 4:46 PM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 21 2017, 4:46 PM
mark added a subscriber: mark.Jul 24 2017, 1:39 PM
ema moved this task from Triage to Caching on the Traffic board.Aug 1 2017, 12:45 PM

After discussion with Faidon at Wikimania we agreed:

  • hosting can move now
  • domain is registered with WMDE. A move requires work by Abraham, who is super busy. I'll keep bugging him about making it happen but we'll not block on it.
Dzahn claimed this task.Sep 18 2017, 3:51 PM
Dzahn added a subscriber: Dzahn.
Dzahn added a comment.Sep 22 2017, 8:05 PM

Hey @Lydia_Pintscher Happy to work on this and talk to you maybe on IRC as well.

I would say one of the first steps is we request a new Gerrit repo and move the static content from the current Github repo (now that the security review is done).

Then i'll write a small puppet class for this that git clones from the new repo. That class can live on an existing VM we have for "static misc sites" or it could have a dedicated VM if needed.

We can then handle permissions on the content repo so that you guys can review and merge and a puppet run deploys it. That is the easiest way that i have done for some other static things like f.e. annual.wikimedia.org.

That should satisfy "perferably automated after git merge" without post-commit hook.

That sounds great!
I thought @Ladsgroup had already requested the repository. I'll let him chime in.

We moved the repo for the source code from github to gerrit (This is the mirror, list of patches), I think @Dzahn means we need to create a repo for static files generated from the source code because we can't do that directly in prod for various reasons. I'm planning to ask for one in Monday, if anyone with proper rights makes wikibase/wikiba.se-deploy repo sooner, I would be waaay happier :D

Ah gotcha. Makes sense.

Change 382277 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[wikibase/wikiba.se-deploy@master] Start the repo

https://gerrit.wikimedia.org/r/382277

Change 382355 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] start profile for wikiba.se web hosting

https://gerrit.wikimedia.org/r/382355

Dzahn added a comment.Oct 5 2017, 4:47 AM

We have the repo, and i see your change Amir, thank you! Before i merge the content itself i started with puppet code to install Apache site and git pull the static files. I was thinking we test that role class first on a cloud VPS with something like wikibase.wmflabs.org or whatever until we are happy to move it to prod. I also started it as a profile following the new puppet style guide lines, so that jenkins-bot likes it now also with the new style check. I made the server name and admin email variable and in templates/Hiera, so it will be easy to change for labs (just a wiki page edit in Hiera: namespace, not even a commit) and also to move it to prod and the actual domain name.

Dzahn added a comment.Oct 5 2017, 11:51 PM

Next we should figure out:

  • who should have Gerrit permissions for +2/merge on the content repo
  • give them the permissions above (which group, how)
  • decide whether in puppet it should be "ensure => latest" (means merging in Gerrit is enough to deploy and people above can do it without shell, they have to wait until next puppet run though) or "ensure => present" (means puppet will not auto-pull changes and a human with shell access has to run git pull to deploy, this could both be an advantage and a disadvantage). Note that "shell" doesn't necessarily mean "ops" and waiting for them. Especially with a dedicated VM we could also just make a (shell) admin group for that with access to just this VM in prod.
  • create project wikibase in "labs"
  • create instance in project, add new role class to instance, set values for sitename, serveradmin in labs Hiera (wiki pages OR repo) to something like personal email address and wikibase.wmflabs.org
  • click in Wikitech-Horizon web ui to add a Proxy and get wikibase.wmflabs.org connected to this instance
  • ... test it..
  • if all is good .. put role on krypton in prod or on dedicated VM (wonder if it should be dedicated right away or not)
  • add misc-web varnish director to handle wikiba.se in caching layer
  • if still good, add real wikiba.se domain to WMF DNS templates ..
  • profit?!
Dzahn added a comment.EditedOct 5 2017, 11:53 PM

By the way i couldn't just merge Ladsgroup's change in the new content repo as i just have +1 rights there, but not +2. I wanted to just do it as long as we are "labs-only" but before we go prod we might have to ask for a quick security review of the js code.

Change 382277 merged by Ladsgroup:
[wikibase/wikiba.se-deploy@master] Start the repo

https://gerrit.wikimedia.org/r/382277

Next we should figure out:

  • who should have Gerrit permissions for +2/merge on the content repo
  • give them the permissions above (which group, how)

"wikidata" LDAP group already has owner access (I gave it to them), We should add all Ops too but I couldn't find the proper LDAP group.

  • decide whether in puppet it should be "ensure => latest" (means merging in Gerrit is enough to deploy and people above can do it without shell, they have to wait until next puppet run though) or "ensure => present" (means puppet will not auto-pull changes and a human with shell access has to run git pull to deploy, this could both be an advantage and a disadvantage). Note that "shell" doesn't necessarily mean "ops" and waiting for them. Especially with a dedicated VM we could also just make a (shell) admin group for that with access to just this VM in prod.

I would tell Lydia and ask her to decide

  • create project wikibase in "labs"

We already have "wikidata-dev", we can use that.

  • create instance in project, add new role class to instance, set values for sitename, serveradmin in labs Hiera (wiki pages OR repo) to something like personal email address and wikibase.wmflabs.org

I make the instance right now, will work on other things too.

wikibase.eqiad.wmflabs is ready for test (hasn't applied the puppet roles yet). Added the hiera var in https://wikitech.wikimedia.org/wiki/Hiera:Wikidata-dev

  • decide whether in puppet it should be "ensure => latest" (means merging in Gerrit is enough to deploy and people above can do it without shell, they have to wait until next puppet run though) or "ensure => present" (means puppet will not auto-pull changes and a human with shell access has to run git pull to deploy, this could both be an advantage and a disadvantage). Note that "shell" doesn't necessarily mean "ops" and waiting for them. Especially with a dedicated VM we could also just make a (shell) admin group for that with access to just this VM in prod.

Let's stick with ensure => latest. Having to find the person who can actually deploy a change has been the major pain point with the previous setup.

Let's stick with ensure => latest. Having to find the person who can actually deploy a change has been the major pain point with the previous setup.

+1

"wikidata" LDAP group already has owner access (I gave it to them), We should add all Ops too but I couldn't find the proper LDAP group.

This is actually the "wikibase" gerrit group (https://gerrit.wikimedia.org/r/#/admin/projects/wikibase,access) which has the owner set as the wikidata gerrit group (https://gerrit.wikimedia.org/r/#/admin/groups/32,members) which in turn has ldap/wmde in it!
As a result all WMDE tech staff have access.

Change 382355 merged by Dzahn:
[operations/puppet@production] start profile for wikiba.se web hosting

https://gerrit.wikimedia.org/r/382355

Mentioned in SAL (#wikimedia-cloud) [2017-10-06T21:58:30Z] <mutante> added myself to project and projectadmins to work on T99531

Dzahn added a comment.Oct 7 2017, 12:33 AM

--> http://wikibase.wmflabs.org/ :)

Dzahn added a comment.EditedOct 7 2017, 12:40 AM

Let's stick with ensure => latest.

Ok! Done. Removed the TODO comment to talk about it.

"wikidata" LDAP group already has owner access (I gave it to them), We should add all Ops too but I couldn't find the proper LDAP group.

The ops LDAP group should just be called "ops". Maybe we don't even need to have it though.

We already have "wikidata-dev", we can use that.
As a result all WMDE tech staff have access.

Alright.

, set values for sitename, serveradmin in labs Hiera (wiki pages OR repo) to something like personal email address

I tried username@fqdn. So the whole thing would become ladsgroup@wikibase-stretch.wikidata-dev.eqiad.wmflabs heh :) Let me know if it works or you want to change it.
It's in hieradata/labs in repo now: https://gerrit.wikimedia.org/r/#/c/382892/2/hieradata/labs/wikidata-dev/host/wikibase-stretch.yaml

You might hate me for this question but have you considered using an org domain instead of the "se"?

Also, i wonder if we should have a dedicated VM or if it's not necessary. Do you have any stats on the traffic that wikiba.se receives?

You might hate me for this question but have you considered using an org domain instead of the "se"?

Yeah but it was decided against it.

Also, i wonder if we should have a dedicated VM or if it's not necessary. Do you have any stats on the traffic that wikiba.se receives?

If that's just matter of traffic, I'm pretty sure it's not much, I checked and we don't have stats but I'm pretty sure it's less than other services they krypton hosts like grafana. Also we simply can put it behind varnish and since it's static, the real hits will be way less.

Change 389944 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] webserver_misc_static: add profile for wikiba.se

https://gerrit.wikimedia.org/r/389944

Change 389944 merged by Dzahn:
[operations/puppet@production] webserver_misc_static: add profile for wikiba.se

https://gerrit.wikimedia.org/r/389944

Change 389947 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikibase: move hiera parameters to correct location

https://gerrit.wikimedia.org/r/389947

Change 389947 merged by Dzahn:
[operations/puppet@production] wikibase: move hiera parameters to correct location

https://gerrit.wikimedia.org/r/389947

Change 389949 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikibase: another fix to Hiera parameter names

https://gerrit.wikimedia.org/r/389949

Change 389949 merged by Dzahn:
[operations/puppet@production] wikibase: another fix to Hiera parameter names

https://gerrit.wikimedia.org/r/389949

Change 389952 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikibase: include base::firewall vs declaring it

https://gerrit.wikimedia.org/r/389952

Change 389952 merged by Dzahn:
[operations/puppet@production] wikibase: include base::firewall vs declaring it

https://gerrit.wikimedia.org/r/389952

Change 389957 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikibase: include apache class vs declaring it

https://gerrit.wikimedia.org/r/389957

Change 389957 merged by Dzahn:
[operations/puppet@production] wikibase: include apache class vs declaring it

https://gerrit.wikimedia.org/r/389957

Dzahn changed the task status from Open to Stalled.Wed, Nov 8, 12:49 PM
Dzahn added a comment.Wed, Nov 8, 12:58 PM

@Ladsgroup Hi, i saw your IRC ping and continued working on this (see above). Though.. now we'll have to talk about the certificate situation (or actually renaming the site). The issue is that wikiba.se is not one of our canonical domain names and wouldn't be covered by our current certs (*.wikimedia.org and project domains). The preferred solution would be to rename it to wikibase.wikimedia.org and i'd like to talk to you about how bad that would actually be. The other solutions would involve $ and/or non-trivial amount of work.

This is not my decision to make, our PM is not around, I'll ask her when she's back