There are plans underway at this point to support multiple LE certs on our standard cache terminators via the work in T199711 due by EOQ (end of Sept), which would make this whole thing simpler and zero cert cost. I couldn't say for sure how fast we'll shake out all the bugs in such a system after initial deployment, but I'd hope quickly.

In the interim, our best option aside from waiting would be to purchase a commercial DV wikiba.se cert and deploy it on the caches (which requires a little bit of testing, we haven't run multiple SNI certs there in a while now). Nobody's worked on this in a while on our end, mostly for lack of priority/time/focus.

In either case, the first few steps are relatively-trivial and would be the same:

1. Create a wikiba.se microsite in WMF infra (already done by @Dzahn I believe, sourcing from https://gerrit.wikimedia.org/r/plugins/gitiles/wikibase/wikiba.se/+/master )
2. Create a wikiba.se template in our authdns, matching the current data (including current non-WMF server IPs) - any complications here, e.g. MX service is currently to udag.de, we can mirror that setting for now I guess. Any other service hostnames besides wikiba.se and www.wikiba.se pointing at 89.31.143.100?).
3. Move authdns control for wikiba.se over to the WMF nameservers (no-op for users, but allows DV on our end).
4. [Issue commercial DV cert to caches to avoid waiting, and/or deploy automated LE DV cert to caches at a later date], Switch IPs over to WMF.

In T99531#4411395, @abian wrote:

wikiba.se is a bit unstable. Today it has been down for some hours (from ~1:00 UTC to ~5:30 UTC). Last issues were detected on 31 May (~0:00 UTC), 24 May (~19:00-20:00 UTC), 10 May (~11:15 UTC) and 9 May (~19:40 UTC) as far as I know. These issues has always been there and I suppose they depend on the server load or, directly, on the provider. I assume this wouldn't happen if the website was hosted in the Wikimedia cluster.

wikiba.se isn't reachable right now, its performance is so bad that connections time out. I don't know what kind of service-level agreement you have, but the uptime might be on the edge.

In T99531#4513194, @abian wrote:

In T99531#4411395, @abian wrote:

wikiba.se is a bit unstable. Today it has been down for some hours (from ~1:00 UTC to ~5:30 UTC). Last issues were detected on 31 May (~0:00 UTC), 24 May (~19:00-20:00 UTC), 10 May (~11:15 UTC) and 9 May (~19:40 UTC) as far as I know. These issues has always been there and I suppose they depend on the server load or, directly, on the provider. I assume this wouldn't happen if the website was hosted in the Wikimedia cluster.

wikiba.se isn't reachable right now, its performance is so bad that connections time out. I don't know what kind of service-level agreement you have, but the uptime might be on the edge.

@Lydia_Pintscher, @WMDE-leszek I have no idea about this!

In case you want to analyze the situation, wikiba.se is down right now.

Indeed it is:

$ ping wikiba.se

Pinging wikiba.se [89.31.143.100] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 89.31.143.100:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Sounds like whatever it is being hosted on is having issues.

these are the contact options listed on the site at http://wikiba.se/contact/ (in case it's down) to find an admin:

It's also possible to ask for support via the Wikidata project:

Mailing Lists
https://twitter.com/wikidata
Wikidata on irc.freenode.net
Contact the development team

In T99531#4511979, @BBlack wrote:

1. Create a wikiba.se microsite in WMF infra (already done by @Dzahn I believe, sourcing from https://gerrit.wikimedia.org/r/plugins/gitiles/wikibase/wikiba.se/+/master )

Yep, that exists already since quite a while. The class is profile::microsites::wikibase and set up http://wikibase.wmflabs.org/

2. Create a wikiba.se template in our authdns, matching the current data (including current non-WMF server IPs) - any complications here, e.g. MX service is currently to udag.de, we can mirror that setting for now I guess. Any other service hostnames besides wikiba.se and www.wikiba.se pointing at 89.31.143.100?).

@Addshore I think we can move this forward by looking at step 2 above. Can you answer that or paste the zone file by any chance?

Did someone ask for a zone file? I have a zone file! Here, take a zone file! ;-)

Change 473543 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/dns@master] WIP DNM: Add wikiba.se

https://gerrit.wikimedia.org/r/473543

Thanks for the data and the patch! We'll dig into the DNS patch next week and get it merged in so we're serving wikiba.se from our DNS as-is (as in, pointing at your existing server IPs). Then we can do handoff of the domain ownership/registration without causing any interruptions.

That gets us over the first handoff hurdle, at which point it's on Traffic to get wikiba.se certs added to our cache clusters using DNS-based validation (again, server IPs still pointing at current server throughout). We're migrating our existing LE certs to a new solution this quarter ( T207050 ), and after that's done we'll look early in the next at defining these new certs' slightly more-complicated case. Once those are issued and deployed, you'll have some time (if needed!) to test and refine the data in our version of the site is hosting ( https://gerrit.wikimedia.org/r/plugins/gitiles/wikibase/wikiba.se/+/master ), and then we switch server IPs and we're done.

Poke as it is now 1 or 2 weeks since the last movement here.
@BBlack I just cced you on the patch so it appears in your review queue.
I haven't made a dns patch in that repo before so no idea if it is ready / I should remove the DNM from the patch?

Change 473543 merged by BBlack:
[operations/dns@master] Add wikiba.se

https://gerrit.wikimedia.org/r/473543

As far as I remember from an IRC discussion, the next step here is to wait for the WMF to get a cert for the domain, then we can move forward again.

There's still a couple of things that can be done serially at present, one of which is necessary for the cert issuance later:

1. Switch the nameservers for wikiba.se to ns[012].wikimedia.org with your current registrar (United Domains). We have to have this to later issue the cert at all. The cert likely won't be issued until sometime in Jan/Feb.
2. Switch the ownership/registration of wikiba.se to the Foundation and its registrar(s). This isn't required to issue the cert on a technical level, but as a matter of general policy we'll want this done at some point before we're really hosting wikiba.se, and there's nothing blocking it after (1) is done above.

For part 2. of this please contact @CRoslof

Part 1 of this should be actioned by WMDE either this or next week.

wikiba.se is now authoritatively nameserved by ns[012].wikimedia.org.

Well, at least that's what I told the DNS.

confirmed, WMF name servers listed as authority:

 dig wikiba.se | grep -A3 AUTHORITY
..
;; AUTHORITY SECTION:
wikiba.se.		86382	IN	NS	ns2.wikimedia.org.
wikiba.se.		86382	IN	NS	ns1.wikimedia.org.
wikiba.se.		86382	IN	NS	ns0.wikimedia.org.

and responding with 89.31.143.100

 for ns in 0 1 2; do dig A wikiba.se @ns${ns}.wikimedia.org; done | grep -A1 ANSWER
..
;; ANSWER SECTION:
wikiba.se.		333	IN	A	89.31.143.100
..

btw, no reverse DNS for that IP

host 89.31.143.100
Host 100.143.31.89.in-addr.arpa. not found: 3(NXDOMAIN)

Might wanna redirect or replace content on ? http://89.31.143.100/

@MasinAlDujailiWMDE could you work with @CRoslof on getting the domain name transferred to WMF?

In T99531#4864434, @Dzahn wrote:

Might wanna redirect or replace content on ? http://89.31.143.100/

I have no idea about that, but @MasinAlDujailiWMDE might be able to do that.

But I guess not top priority as the hosting should be moving soonish ;)

I guess the next thing on the todo list is the second part of step #1

In T99531#4820326, @BBlack wrote:

1. Switch the nameservers for wikiba.se to ns[012].wikimedia.org with your current registrar (United Domains). We have to have this to later issue the cert at all. The cert likely won't be issued until sometime in Jan/Feb.

Step #2 will be up to @MasinAlDujailiWMDE in coordination with @Lydia_Pintscher, @Franziska_Heine and @Abraham ?

2. Switch the ownership/registration of wikiba.se to the Foundation and its registrar(s). This isn't required to issue the cert on a technical level, but as a matter of general policy we'll want this done at some point before we're really hosting wikiba.se, and there's nothing blocking it after (1) is done above.

In T99531#4820326, @BBlack wrote:

1. Switch the nameservers for wikiba.se to ns[012].wikimedia.org with your current registrar (United Domains). We have to have this to later issue the cert at all. The cert likely won't be issued until sometime in Jan/Feb.

This is done. The cert could be issued now?

In T99531#4864464, @Addshore wrote:

But I guess not top priority as the hosting should be moving soonish ;)

Yea, i realized this myself after making the comment. Not worth it, ignore :)

I guess the next thing on the todo list is the second part of step #1
Step #2 will be up to @MasinAlDujailiWMDE in coordination with @Lydia_Pintscher, @Franziska_Heine and @Abraham ?

ACK, indeed

Transferring the domain name from WMDE to the Foundation requires that WMDE complete an ownership change form. I emailed with @Abraham and the Foundation's domain name registrar about it a while back, but the paperwork was never completed. Let me know when WMDE is ready to move forward with the transfer.

In T99531#4878798, @CRoslof wrote:

Transferring the domain name from WMDE to the Foundation requires that WMDE complete an ownership change form. I emailed with @Abraham and the Foundation's domain name registrar about it a while back, but the paperwork was never completed. Let me know when WMDE is ready to move forward with the transfer.

@MasinAlDujailiWMDE I think this is for you to action?

Okay, so the question that I have now been asked is "why we can't simply do a DNS re-route without changing the owner".
So, why can't we just point the A record to the right place and be done with that?

I guess this is questioning the below and figuring out why that has to happen / if it really does have to happen, what the policy is etc that is being followed.

In T99531#4820326, @BBlack wrote:

2. Switch the ownership/registration of wikiba.se to the Foundation and its registrar(s). This isn't required to issue the cert on a technical level, but as a matter of general policy we'll want this done at some point before we're really hosting wikiba.se, and there's nothing blocking it after (1) is done above.

Pinging @BBlack and @CRoslof as the 2 people that might know the answer?

There are different layers of "handing off" DNS management which are being conflated, but to run through them in order:

1. "Point the A record to the right place" - We don't support this, and can't realistically. We need control of the zone data directly on our nameservers for a variety of technical reasons (e.g. setting policy controls like CAA authorizations, future ESNI-related records, etc), and we don't use simple A-records, we use a dynamic system that hands out any of a number of addresses to the nearest of our global edge datacenters, and these things evolve over time on a technical level. If we're going to host something in our production infrastructure and manage it correctly, we have to move to at least the next level of handoff:

2. Leaving the registration of the domain with WMDE and their registrar, but having the Nameservers pointed at WMF nameservers. This is what we've already done earlier in the ticket and where we're at now. Currently the domain is registered to WMDE (presumably, it's hidden in public view) via registrar "united-domains", and the nameserver values are set to the 3x WMF nameserver hosts (ns0.wikimedia.org, ns1.wikimedia.org, and ns2.wikimedia.org, at specific IP addresses for each). This allows the WMF nameservers and SRE staff to do all the basic technical things referenced above, and is the first logical step before:

3. Switching the registration to WMF's registrar/ownership. This is more on a policy/standards/legal level, and maybe @CRoslof can give more details than me on that front about legal-related things. It would be odd in the general case to be canonicalizing a WMF domain without registrar control though, as it could be swapped out from under us at any time. However, even on a purely technical level it matters to us as well: we have future plans to deploy more authdns servers, change their IPs, and deploy anycasted authdns as well, all of which require WMF to have tight control over the registrar settings for all the domains we host resources for so that we can get through transition periods smoothly as those ns[012] hostnames and their IPs change. It's not scalable for those processes to involve contacting N third parties and having them all indirectly contact their registrars on our behalf, etc.

Change 500695 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikiba.se: add Apache rewrites for www to naked domain

https://gerrit.wikimedia.org/r/500695

wikiba.se can now be viewed in WMF production by editing the local /etc/hosts file with f.e. 91.198.174.192 wikiba.se

Open issues are:

• add HSTS
• currently both naked domain and www.wikiba.se work equally but we should avoid serving the same content from different URLs, so we want to redirect/rewrite one to the other. which way around is an open question
• add docs how to deploy. done -> https://wikitech.wikimedia.org/wiki/Microsites#How_to_deploy

I have contacted @Ladsgroup and he will ping WMDE's product manager for Wikibase.

Change 500711 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikiba.se: add HSTS header with low max_age

https://gerrit.wikimedia.org/r/500711

Change 500715 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] varnish/trafficserver: add regex to cover www.wikiba.se as well

https://gerrit.wikimedia.org/r/500715

Change 500695 merged by Dzahn:
[operations/puppet@production] wikiba.se: add Apache rewrites for www to naked domain

https://gerrit.wikimedia.org/r/500695

Change 500715 merged by Dzahn:
[operations/puppet@production] varnish/trafficserver: add regex to cover www.wikiba.se as well

https://gerrit.wikimedia.org/r/500715

In T155359#5108338, @Dzahn wrote:

Next is deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/500715 for T99531#5077170

• done

and then https://gerrit.wikimedia.org/r/c/operations/puppet/+/500695

• done

and then WMDE and WMF have to agree on the domain transfer ownership (Lydia said she will ping some people to find out)

• still to do

looks like we are not going to do https://gerrit.wikimedia.org/r/c/operations/puppet/+/500711

• stalled / abandon?

Also note since recently we now have wikibase.org (https://gerrit.wikimedia.org/r/c/operations/dns/+/503154)

• I don't assume you guys are interested in switching domains? heh

In T99531#5136277, @Dzahn wrote:

Also note since recently we now have wikibase.org (https://gerrit.wikimedia.org/r/c/operations/dns/+/503154)

• I don't assume you guys are interested in switching domains? heh

I'm in no means in charge to decide anything but maybe we can redirect wikibase.org to wikiba.se?

Re: wikibase.org, adding it as a non-canonical redirection to catch confusion from those that manually type URLs is fine, but we should make sure everyone is clear on which domainname is canonical for this project (I assume https://wikiba.se/) and make sure that's the only one that's published, promoted, and used for links we control and such. It's an important notion that one name is canonical!

Re: the HSTS/HTTPS stuff in https://gerrit.wikimedia.org/r/c/operations/puppet/+/500711 :

• It's policy for canonical domains we support here, so there's no real debate about whether we'll end up with full-value HSTS and the HTTP->HTTPS redirect.
• But we don't need this separate patch at the apache level; we'll handle it in VCL with the same code that handles the other canonical project domains.

Re: handing off registration - I really think we should stop touching this whole project until this gets resolved, which means stalling on the above HSTS/HTTPS work and on the switch of IPs.

This is a policy issue as well, which we've tried to explain politely more than once in this thread, but if it's going to end up being a blocker there's no point expending further effort on this until they figure out what direction they want to go. I think the original statement way back from @faidon was that it was a "very strong preference" that we get ownership transfer, but in fact we'd already made the declaration that it's a policy requirement about a week before that in https://wikitech.wikimedia.org/wiki/HTTPS#For_the_Foundation's_canonical_domainnames , and honestly I really don't want to wade into the mess of having an exception to those rules during all the future improvements we have coming at the DNS and HTTPS layers. Even just applying strong HSTS with ownership issues seems irresponsible of us at best, as the current hosting WMDE has it on lacks HTTPS entirely.

thanks for the write up @BBlack, I am going to take over the domain ownership topic from WMDE side, as it apparently has fallen through the cracks at our end.

@WMDE-leszek Thanks for looking into it! I believe @CRoslof is who you want to coordinate with on our end, whose last statement on this topic back in January was:

In T99531#4878798, @CRoslof wrote:

Transferring the domain name from WMDE to the Foundation requires that WMDE complete an ownership change form. I emailed with @Abraham and the Foundation's domain name registrar about it a while back, but the paperwork was never completed. Let me know when WMDE is ready to move forward with the transfer.

Change 500711 abandoned by Dzahn:
wikiba.se: add HSTS header with low max_age

Reason:
quoting bblack "we don't need this separate patch at the apache level; we'll handle it in VCL with the same code that handles the other canonical project domains." https://phabricator.wikimedia.org/T99531#5137543

https://gerrit.wikimedia.org/r/500711

In T99531#5137543, @BBlack wrote:

Re: wikibase.org, adding it as a non-canonical redirection to catch confusion from those that manually type URLs is fine, but we should make sure everyone is clear on which domainname is canonical for this project (I assume https://wikiba.se/) and make sure that's the only one that's published, promoted, and used for links we control and such. It's an important notion that one name is canonical!

Yes, that's kind of why i wanted to mention it now. It seems to me it's "now or never" to make a choice for or against wikibase.org.

It also seemed like quite an interesting timing that we got wikibase.org right now while we (never?) had it in the past.

My personal opinion is that an .org domain is better and more consistent with our other domains instead of using the TLD of Sweden just to get a shorter name.

But of course i also realize it's already an existing project and cool URIs don't change and that it would complicate things.

Just wanted to point out this would be the moment to decide and we shouldn't regret it later when it's much harder to change.

Re: the HSTS/HTTPS stuff in https://gerrit.wikimedia.org/r/c/operations/puppet/+/500711 :

• It's policy for canonical domains we support here, so there's no real debate about whether we'll end up with full-value HSTS and the HTTP->HTTPS redirect.
• But we don't need this separate patch at the apache level; we'll handle it in VCL with the same code that handles the other canonical project domains.

ACK, understood. I abandoned the Apache change with a link to this. Thanks!

Re: handing off registration - I really think we should stop touching this whole project until this gets resolved, which means stalling on the above HSTS/HTTPS work and on the switch of IPs.

Yep, i consider it stalled for now until the domain part has been figured out. I just wanted to finish the technical part we had identified was missing already.

This is a policy issue as well, which we've tried to explain politely more than once in this thread,

Yes, i have no further details about that yet but Lydia wanted to ping some people. Thanks @WMDE-leszek for taking it on!

@BBlack @Dzahn: I have passed the topic of domain ownership transfer to the C-level ranks here at WMDE, and I have to inform that WMDE is not going to transfer the ownership of wikiba.se domain to WMF.
As I understand your statements above, this basically invalidates the whole idea of moving the hosting of the said website to WMF cluster.

I wish we WMDE folks have clarified this way earlier, before you have had spend quite some effort on working on this. I apologize for that, and I genuinely appreciate all your work on this topic.

It sounds like we should just close this ticket as Declined then @WMDE-leszek ?

If that is really the outcome that would be unfortunate but please leave it open or ideally create a subtask to revert/remove the code that had already been added for this.

It seems like we have been in a Mexican standoff / impasse here since July 2017 which is unfortunate, I don't want to know how many hours have been spent trying to figure this out.

Yes let's leave it open at least until sub tickets are created to tidy up what has already be done etc.
Step 1 I gather is for @MasinAlDujailiWMDE to move the domain back to wmde controlled name servers.

Tbh it seems the main issue is that the c-level communication is not happening on this task and therefore not visible. I am respectfully stepping back from this task now.

As noted in T155359 - WMDE has moved the hosting of this to some other platform, including the DNS hosting (and we never had the whois entry). So this task can resolve as Decline I think (or whatever), but we should use it to track down various revert patches first before we close it up (revert the DNS repo stuff and whatever else we've got going on in various other repos supporting the wikiba.se site).

Change 532972 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove wikiba.se microsite puppetization

https://gerrit.wikimedia.org/r/532972

Change 532973 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS/acme_chief/varnish: remove wikiba.se

https://gerrit.wikimedia.org/r/532973

Change 532975 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] labs.yaml: remove wikibase in cloud vps

https://gerrit.wikimedia.org/r/532975

Change 532976 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: remove wikiba.se backend

https://gerrit.wikimedia.org/r/532976

Change 532976 merged by Dzahn:
[operations/puppet@production] ATS: remove wikiba.se backend

https://gerrit.wikimedia.org/r/532976

Change 532972 merged by Dzahn:
[operations/puppet@production] remove wikiba.se microsite puppetization

https://gerrit.wikimedia.org/r/532972

Change 532975 abandoned by Dzahn:
labs.yaml: remove wikibase in cloud vps

Reason:
maybe still useful for WMDE to test changes before they deploy on their servers

https://gerrit.wikimedia.org/r/532975

In T99531#5414154, @BBlack wrote:

track down various revert patches first before we close it up (revert the DNS repo stuff and whatever else we've got going on in various other repos supporting the wikiba.se site).

In the puppet repo we just have the acme_chief.yaml part and profile/manifests/cache/ssl/wikibase.pp part left now.

CC: @Vgutierrez

Change 534276 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] delete wikiba.se

https://gerrit.wikimedia.org/r/534276

Mentioned in SAL (#wikimedia-operations) [2019-09-17T07:10:45Z] <vgutierrez> getting rid of wikibase TLS certificate & nginx configuration on the text cache cluster - T99531

Change 532973 merged by Vgutierrez:
[operations/puppet@production] ATS/acme_chief/varnish: remove wikiba.se

https://gerrit.wikimedia.org/r/532973

Mentioned in SAL (#wikimedia-operations) [2019-09-17T07:19:53Z] <vgutierrez> depooling cp5007 to ensure that wikibase removal goes as expected - T99531

Mentioned in SAL (#wikimedia-operations) [2019-09-17T07:29:36Z] <vgutierrez> repooling cp5007 without wikibase configuration - T99531

Change 534276 merged by Dzahn:
[operations/dns@master] delete wikiba.se

https://gerrit.wikimedia.org/r/534276

I think we are done with the cleanup in production now and only cloud VPS instance is left. @Addshore Does anyone use that to test changes for Wikibase?

@Dzahn the instance was no longer in use, so I've just deleted it.

Thanks @WMDE-leszek. !

I think we can close this now @BBlack @Vgutierrez I don't see more leftovers to clean up.