Sorry, the indications you give here are in contrast with each other:

you seem to want a full redirect of transparency.wikimedia.org to the new url - but also a redirect to https://transparency.wikimedia.org/historical. While I guess we could avoid doing a redirect for the old pages, that would result in a ton of broken links.

Also: all the links and the images/assets in the old transparency report site are without scope - who would be responsible to fix them?

As I understand it - Legal would like the existing microsite located at transparency.wikimedia.org to be relocated to transparency.wikimedia.org/historical - and then for the top-level transparency.wikimedia.org domain to redirect to the report's new location at https://wikimediafoundation.org/about/transparency/

Summary:

• transparency.wikimedia.org microsite moved to transparency.wikimedia.org/historical
• transparency.wikimedia.org redirected to https://wikimediafoundation.org/about/transparency/

I believe the annual report has a similar setup if that reference helps.

@Varnent: For the redirects: just the main https://transparency.wikimedia.org/ URL? Or also the sub-pages like https://transparency.wikimedia.org/content.html ? I haven't yet looked at the content for the move to /historical/, but I assume it's relatively-simple.

Why didn't we just move the whole of transparency.wikimedia.org over to Automattic and handle the redirects and/or content changes there? It's not like any of it's staying original anyways.

In T230638#5443831, @BBlack wrote:

@Varnent: For the redirects: just the main https://transparency.wikimedia.org/ URL? Or also the sub-pages like https://transparency.wikimedia.org/content.html ? I haven't yet looked at the content for the move to /historical/, but I assume it's relatively-simple.

I think just the main URL and perhaps faq.html - but I do not think there are many external links. I defer to Legal on that one though.

Why didn't we just move the whole of transparency.wikimedia.org over to Automattic and handle the redirects and/or content changes there? It's not like any of it's staying original anyways.

There are costs with each additional site we add, and I believe the plan is to shut the historical site down completely once Design is done working on modules to import the rest of the historical content into the organization website.

One thing that you can already do is create https://transparency.wikimedia.org/historical/ since that is just inside the content repo that is under your control and is a requirement before we can add rewrite/redirect rules to it in the webserver config.

In T230638#5450941, @Dzahn wrote:

One thing that you can already do is create https://transparency.wikimedia.org/historical/ since that is just inside the content repo that is under your control and is a requirement before we can add rewrite/redirect rules to it in the webserver config.

I am unfamiliar with the functional technical setup of the existing microsite, and so not sure how change where it builds. It appears to build itself using a script I think we developed. Also, this microsite's content repo is on Foundation servers - so we all have access to it.

I am confused by this statement. The content repo is explicitely setup this way so that people working on this site can merge content changes without having to contact SRE and when i look at the repo in Gerrit i see that you just merged something in it yourself. https://gerrit.wikimedia.org/r/q/project:wikimedia%252FTransparencyReport

All the changes you can see above, including your own, have been deployed to production servers automatically in the past.

The puppet code is:

16     git::clone { 'wikimedia/TransparencyReport':
17         ensure    => latest,

which means all that happens is that puppet pulls from the repo people have been using all this time. I don't see why that would be different now.

In T230638#5453422, @Dzahn wrote:

All the changes you can see above, including your own, have been deployed to production servers automatically in the past.

The puppet code is:

16     git::clone { 'wikimedia/TransparencyReport':
17         ensure    => latest,

which means all that happens is that puppet pulls from the repo people have been using all this time. I don't see why that would be different now.

I am incredibly confused right now. I have made some additions to the code it builds from. However, that was basically adding a line of HTML. I am not familiar enough with the microsite setup to know how to tell the script to build it in a different place. I suppose I can take a look later, but if you all already know how to do it - is there a reason a comms person needs to do that coding? Again, this is not our project, we are offering some help and were asked to help clarify a technical question. My familiarity with the old site ends at knowing how to update some of the numbers and where to add code for a banner.

In T230638#5453416, @Dzahn wrote:

I am confused by this statement. The content repo is explicitely setup this way so that people working on this site can merge content changes without having to contact SRE and when i look at the repo in Gerrit i see that you just merged something in it yourself. https://gerrit.wikimedia.org/r/q/project:wikimedia%252FTransparencyReport

Right - except we are not talking about content changes - we are trying to change the technical setup of the site.

Some clarifying points:

1. Code - means different things to different people in different contexts. From the SRE perspective, the whole repo is "content", even though there's a build script in there and various browser-related "languages" involved. From the server-side perspective it's all just static content to us.
2. Historically SRE doesn't manage anything inside this repo. Our role in this is simply to provide a static content hosting service that auto-deploys whatever is committed to the repo. Whomever does the content repo editing runs a build script and commits the outputs back to the repo itself as static content.
3. Historically, it seems the primary editors of this repo have been the Product Design team, specifically e.g. @Prtksxna and @Volker_E .

Okay - so basically SRE will not setup the requested configuration until we figure out how to setup the script to output the old site into a subdirectory - correct?

There are two separate things to do here:

1. The move of content to the new /historical/ sub-path: For this, nothing will actually change on the side of this that SRE actually manages (the deployment of the static content in the repo to a microsite static service that hosts the domainname). Somebody (SRE could do it perhaps, but we don't historically, and Product Design seems a more-natural fit since they've worked on it before) has to make changes to the build scripts and/or other contents in the repo to move things to /historical/ and keep the internal links working. If the repo is effectively-dead at this point it may not be worth updating the build system if that's problematic; it may be simpler to alter the existing outputs manually (e.g. git mv the pages into the /historical/ path and then edit any links that need editing with a sed script or whatever).

2. The redirect of https://transparency.wikimedia.org/ to the new foundation blog URL(s): This is separate and doesn't happen inside the above repo, and is on the SRE side of things in terms of technical setup. Probably the above should happen first, but they should happen close in time to avoid user confusion. I'd like us to be explicit about the requirements, though. What I'm hearing/interpreting so far is that the following two single-URL redirects should be configured, but all other possible URLs should continue being served directly from the old transparency site (which means they'll be 404s if anyone has them saved or linked anywhere, since the content switched out to /historical/):

• https://transparency.wikimedia.org/ -> https://wikimediafoundation.org/about/transparency/
• https://transparency.wikimedia.org/faq.html -> https://wikimediafoundation.org/about/transparency/faq/

I would've expected you to perhaps add to that list the other primary pages, e.g.:

• https://transparency.wikimedia.org/privacy.html -> https://wikimediafoundation.org/about/transparency/privacy/
• https://transparency.wikimedia.org/content.html -> https://wikimediafoundation.org/about/transparency/content/
• https://transparency.wikimedia.org/stories.html -> ??? (maybe just the main transparency about page, since there's no equivalent?)

Another option would be to simply redirect *all* URLs on the old site to https://wikimediafoundation.org/about/transparency/ by default, and carve out an exception just for https://transparency.wikimedia.org/historical/ so that it can be linked from the foundation site without creating a loop.

(Also - a side note while I was digging in various things: https://wikimediafoundation.org/advocacy/ has a link to the old transparency site still, should probably be updated to the new one on the main blog (and any other such links that might exist around the site))

Awesome - thank you @BBlack - super helpful and I get now what you all need done. Apologies for the confusion on my part, I thought this was done via backend mapping that basically pointed the repo to a specific URL which could just be updated. I think I understand how what you are indicating is the setup.

I will talk with Legal about the redirect setups, I suspect an all with exception setup makes the most sense.

I think all links to the report point to transparency.wikimedia.org - which will essentially continue to be the URL used (it also appears on the printed report which has already been released). So I think we are okay to leave them assuming we are able to make this change reasonably soon.

On the broader meta-topics: Long-lived canonical URLs are important, and I think that transparency.wikimedia.org seems like a more-natural fit for that (and to continue printing and publishing it). IMHO, the ideal end-game here* would be to move transparency.wikimedia.org to Automattic hosting completely and have it serve the new content directly, as well as the historical parts, and have the blog's links link into it. The currently-outlined (interim?) setup sends confusing social and technical signals (e.g. to search engines) about which of https://transparency.wikimedia.org/ or https://wikimediafoundation.org/about/transparency/ is the canonical location of the content.

• - (within the scope of what's already been decided and ignoring the known pain-points we've communicated before, re: how the privacy and transparency policies of our main sites, documented on these sites, are different from Automattic's privacy and transparency policies in hosting the content about our policies, etc...)

Change 533537 had a related patch set uploaded (by Varnent; owner: Varnent):
[wikimedia/TransparencyReport@master] Testing if copying files to subdirectory will produce desired build - related to T230638

https://gerrit.wikimedia.org/r/533537

In T230638#5453970, @BBlack wrote:

On the broader meta-topics: Long-lived canonical URLs are important, and I think that transparency.wikimedia.org seems like a more-natural fit for that (and to continue printing and publishing it). IMHO, the ideal end-game here* would be to move transparency.wikimedia.org to Automattic hosting completely and have it serve the new content directly, as well as the historical parts, and have the blog's links link into it. The currently-outlined (interim?) setup sends confusing social and technical signals (e.g. to search engines) about which of https://transparency.wikimedia.org/ or https://wikimediafoundation.org/about/transparency/ is the canonical location of the content.

• - (within the scope of what's already been decided and ignoring the known pain-points we've communicated before, re: how the privacy and transparency policies of our main sites, documented on these sites, are different from Automattic's privacy and transparency policies in hosting the content about our policies, etc...)

I agree that is a better long-term setup and is something I can bring up with Automattic. Is it safe to say this is something that could be done relatively easy if the site were hosted internally?

Let me know if you think https://gerrit.wikimedia.org/r/533537 will work - thanks! :)

In T230638#5454212, @Varnent wrote:

I agree that is a better long-term setup and is something I can bring up with Automattic. Is it safe to say this is something that could be done relatively easy if the site were hosted internally?

I'm not sure exactly what you mean by "if the site were hosted internally" here, there's a lot of possible interpretations! My best guess would be that you mean transparency.wm.o continues to be a static-content microsite hosted by SRE and driven by the git repo like today, and you want to take the Automattic-developed content and stuff it into the git repo so that it appears through our hosting of the microsite (wouldn't support any "live" WP features like comments, analytics would be our standard production analytics, but on the plus side it would comply with our normal privacy and transparency policies), and then have the foundation site link into it? That could work, if Automattic can output the static data in a workable form (possibly as gerrit changes like the one you link below, for someone here to +2).

Let me know if you think https://gerrit.wikimedia.org/r/533537 will work - thanks! :)

Probably not, as a quick glance at various HTML files shows absolute site-local links that need the historical prefix added, e.g. line 12 of /build/privacy.html has near the beginning of that long line, the snippet: <link href="/stylesheets/bootstrap.min.css" rel=stylesheet />, which would need editing to reference /historical/stylsheets/boostrap.min.css, or to be a relative link (no leading slash, in this case). There are probably lots of such examples throughout.

Sorry - should have clarified. I meant if in theory down the road the wikimediafoundation.org site was moved to our servers - the setup you are describing would be easy to do? Just getting some context and verifying assumptions before going in to talk with Automattic. :)

Hi all, just wanted to see if there was any further info or clarification needed from Legal. We really appreciate everyone thinking through how to get this done. It sounds like the next step is seeing if this patch works?

After a discussion with @LMixter, @JbuattiWMF and @Varnent we'd like to propose:

1. transparency.wikimedia.org (and all sub-paths) should redirect to wikimediafoundation.org/about/transparency/. This might result in some broken links but that is fine (technically not broken, just not as expected, everything should just redirect the the new page).
2. Create a new sub-domain transparency-archive.wikimedia.org that will serve the old website. This will be a temporary measure till legal (with the help of devs) figures out a way to move all the historical data onto the new website. AFAIK the internal links on the old website should continue to work fine.

I understand (2) might not be ideal due to the added sub-domain, but it'll only be temporary. @Dzahn @BBlack what do you think?

I wonder why #1 should be performed before #2 is performed, and not the other way round (copy/move historical data onto new website, then redirect)?

@Aklapper, my understanding is that the legal team doesn't have the resources to do #2 at the moment, and not having #1 is already causing some confusions (people emailing asking where the latest report is etc.)

Hi @Aklapper, just to confirm what Prateek said, we would need help with both but #1 is more immediate because we are getting questions from the community.

Hello friends, and happy new year. Legal was hoping to check back in on #1 and #2 (but #1 in particular). Thanks so much!

Change 564684 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Set up transparency-archive microsite

https://gerrit.wikimedia.org/r/564684

Change 564685 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Add transparency-archive.wikimedia.org

https://gerrit.wikimedia.org/r/564685

Change 564685 merged by BBlack:
[operations/dns@master] Add transparency-archive.wikimedia.org

https://gerrit.wikimedia.org/r/564685

Change 564686 had a related patch set uploaded (by BBlack; owner: BBlack):
[wikimedia/TransparencyReport@master] Replace absolute self-links with host-relatives

https://gerrit.wikimedia.org/r/564686

Change 564686 merged by BBlack:
[wikimedia/TransparencyReport@master] Replace absolute self-links with host-relatives

https://gerrit.wikimedia.org/r/564686

Change 564684 merged by BBlack:
[operations/puppet@production] Set up transparency-archive microsite

https://gerrit.wikimedia.org/r/564684

Change 564689 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] transparency-archive: correct template name

https://gerrit.wikimedia.org/r/564689

Change 564689 merged by BBlack:
[operations/puppet@production] transparency-archive: correct template name

https://gerrit.wikimedia.org/r/564689

Change 564697 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Update webserver-misc-static cert

https://gerrit.wikimedia.org/r/564697

Change 564697 merged by BBlack:
[operations/puppet@production] Update webserver-misc-static cert

https://gerrit.wikimedia.org/r/564697

Change 564704 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Support missing /historical on transparency sites

https://gerrit.wikimedia.org/r/564704

Change 564705 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Redirect transparency.wm.o -> foundation site

https://gerrit.wikimedia.org/r/564705

Change 564704 merged by BBlack:
[operations/puppet@production] Support missing /historical on transparency sites

https://gerrit.wikimedia.org/r/564704

Change 564706 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Fixup for historical redirect

https://gerrit.wikimedia.org/r/564706

Change 564706 merged by BBlack:
[operations/puppet@production] Fixup for historical redirect

https://gerrit.wikimedia.org/r/564706

Change 564705 merged by BBlack:
[operations/puppet@production] Redirect transparency.wm.o -> foundation site

https://gerrit.wikimedia.org/r/564705

Yes, more or less. The major caveat is some of our caches still have non-redirecting copies of various pages in https://transparency.wikimedia.org/ , but this will sort itself out over the next day or so at the most. To save anyone from trawling through the list of commits above, the changes in effect now are:

1. https://transparency-archive.wikimedia.org/ has all the same content as the original https://transparency.wikimedia.org/
2. I fixed a few absolute links to make sure the transparency-archive copy links to itself and not back to transparency.wikimedia.org
3. https://transparency.wikimedia.org/ (and all subpages) redirects to the foundation site at https://wikimediafoundation.org/about/transparency/
4. Because the foundation site (e.g. https://wikimediafoundation.org/about/transparency/privacy/ ) has links like "please see our archived report" which currently link to https://transparency.wikimedia.org/historical , the "/historical" path has been made an exception: on both the transparency and transparency archive sites, hitting that subpath results in an absolute redirect to https://transparency-archive.wikimedia.org/ (instead of a loop back to the foundation site, in the transparency case).
5. All the new redirects are 302 (Temporary) rather than 301 (Permanent) so that we don't cause issues if/when all these URL layout choices change later

https://gerrit.wikimedia.org/r/c/operations/puppet/+/745637 as part of T218900 means in the unlikely event that you need future changes to this archived historic site you will have to ask SRE to deploy. if that should happen feel free to open a new ticket.

Thanks @Dzahn! I don't have access to see that ticket, but I'll keep this in mind.