Add Annual Report 2017 to git repo
Closed, ResolvedPublic

Description

Requesting a new subdomain to host the upcoming annual report:
https://annual.wikimedia.org/2017/

The subdomain will host the webpages documenting Wikimedia Foundation finances, stories, and donor lists from FY16-17.

Previous Annual Reports are visible at https://annual.wikimedia.org/2016/ or https://15.wikipedia.org

Heather created this task.Nov 7 2017, 7:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 7 2017, 7:23 PM

https://phabricator.wikimedia.org/T151798#2828538 seems to explain the procedure and links to patches from last year, if anyone wants to go ahead here.

(Technically speaking, this task actually seems to be about setting up a URL, not a subdomain? If I understand it correctly.)

Yes, thanks! I was just copying the language. Appreciate the correction.

Oh, and to avoid misunderstandings (or running into deadlines), as I see that this task was handled by the Operations team last time: Does the Annual-Report team need any help resolving this task? If so, please add the corresponding team project via the Add Action...Change Project Tags dropdown so it will appear on that team's workboard and they can get aware of this task. Thanks! :)

Reedy added a subscriber: Reedy.EditedNov 7 2017, 11:57 PM

Yeah, it's not a subdomain.

https://github.com/wikimedia/puppet/blob/d6543a1c8ea0213e9deaab994ae5a2bc20b47d2b/modules/profile/manifests/microsites/annualreport.pp

I think, basically... https://github.com/wikimedia/wikimedia-annualreport or rather, from gerrit, it needs adding to a 2017 folder on git clone ssh://gerrit.wikimedia.org:29418/wikimedia/annualreport

Then... When puppet runs (give it an hour or so), it should just update onto the server automagically :)... So, it *shouldn't* need Operations support

Reedy renamed this task from Add subdomain for Annual Report 2017 to Add Annual Report 2017 to git repo.Nov 7 2017, 11:59 PM
Dzahn claimed this task.Nov 8 2017, 9:43 AM
Dzahn added a comment.EditedNov 8 2017, 10:25 AM

What Reedy said is correct. It just needs uploading the content into a new directory 2017 in the repo linked above.

If there is only HTML/CSS/images we can just do it. Anyone with +2 on the repo can merge it and puppet deploys it.

If there are scripts in it, it will need a security review from security team.

I could create the empty "2017" dir right now but it would just be "mkdir" and not have any content, so we'll just wait for the content.

What we really need here are just the files to upload. Ideally you can just make a zip file and drag & drop it into this ticket. I can take it from there.

The last step, which can happen after uploading the content is just switching the line <meta http-equiv="refresh" content="0; URL=./2016/"> in index.html to point to ./2017/.

Dzahn changed the task status from Open to Stalled.Nov 9 2017, 7:39 AM
Dzahn added a comment.Dec 6 2017, 5:38 PM

There was a meeting about this today, involving the actual developer and security. He confirmed he analyzed our pages of the previous years and it will be just like that, only static HTML/CSS/JS, no tracking, no loading from external sites etc. We don't foresee any issues here. It will be released some time in January.

Dzahn changed the task status from Stalled to Open.Jan 3 2018, 10:25 PM
Dzahn triaged this task as High priority.

@ZMcCune sent the files for security review today via mail/dropbox. Uploading them to Gerrit. Unstalling ticket.

Change 401813 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[wikimedia/annualreport@master] Add annual report 2017 files

https://gerrit.wikimedia.org/r/401813

Dzahn changed the task status from Open to Stalled.EditedJan 8 2018, 9:17 PM

waiting for security review by security team

security team, please take a look at the gerrit link above

waiting for security review by security team

security team, please take a look at the gerrit link above

Cool. I will look at this shortly.

I haven't finished with this yet, but the (relatively minor) concerns I have so far:

  • https://wikimediafoundation.org/wiki/Wikimedia_Foundation_Annual_Report_2016-2017_Privacy_Policy is linked from the footer but is a 404. I trust this will be created before the site goes live?
  • `<link href="//fonts.googleapis.com/css?family=Montserrat:300,400,700" rel="stylesheet">` - This in theory allows google to see who is viewing the annual plan site. Montserrat is a freely licensed font. Can the font be included in the /fonts/ directory of the site so we can avoid disclosing any user info to third parties like google?

Ok, review done.

First off, I wanted to say, this looks really smooth and beautiful. Really good work to everyone involved.

Of these concerns, the only two that are hard blockers are: loading fonts from google and the "BebasNeue by Flat-it" not being open source. In both those cases, its ultimately up to WMF-Legal if that's ok rather than being up to me (For the former, if the site has its own privacy policy, than it could potentially be allowed as part of that. For that latter, that's all a licensing thing, which is of course legal's area)

Security concerns

  • <link href="//fonts.googleapis.com/css?family=Montserrat:300,400,700" - For privacy reasons, no loading resources from external domains.
  • [very minor] Links that have target=_blank (especially to external domains) should set rel=noopener to prevent https://mathiasbynens.github.io/rel-noopener/ (That link suggests type attacks. Similarly openShareWindow using window.open js should set the opener property to null

Other comments/concerns

[This is not officially part of the security review as they are not security issues]

  • The privacy policy link needs to be created (And presumably approved by WMF-Legal )
  • I was kind of surprised to see the no-cache meta tag. Given this is a static site wouldn't we want it to be cached?
  • There is a robots.txt file. It would not be taken into account as its not in the root directory. However its also blocking all search engines. I do not think we would want to block search engines from the annual report website
  • "financials.html" line 146 - svg doctype/xml prolog should probably not be included when embedding svg into html.
  • favicon.ico doesn't seem to have a <link> tag. Thus it will not be used as it is not in the root of the domain.
  • The font "BebasNeue by Flat-it" is not open source ( https://www.myfonts.com/viewlicense?type=web&buildid=3481048 ). Generally using non-open source components is frowned upon and I think requires special approval. In addition to not being Free Software, the license has some additional clauses that at first glance look problematic (IANAL. I'm just flagging this as looks suspicious to me, ultimately this is more something that is WMF-Legal 's area. For all I know, this may have already been discussed and approved)
    • "The comments, showing copyright and other legal information provided in the sample HTML/CSS/Javascript files, for each Licensed Web Font must be retained in your working Website code. " - But the copyright notice is not present in the minimized css files which are actually used by the website.
    • "You must retain the page view tracking code, as supplied in the Web Font Kit, on all Websites that Use the Licensed Web Fonts. " - I don't know what this is about, but sounds problematic.
  • The text of the website is listed as cc-by, but its unclear if the html source and non-vendor js is open source. It would be nice if that was clarified as it is living in our public git repo (e.g. Having a file named COPYRIGHT that clarified the copyright status).
  • "stories.html" line 324 - <a target="_blank" href="http://wikimedia2017.mangrove-creative.com/2017/community.html">WE INVITE YOU TO LEARN MORE ABOUT OUR GRANTS PROGRAMS</a> - this should link just to community.html, not to the mangrove-creative site.

Brian, thank you. We are also really pleased with the site and are happy you like it too! ALL of these notes are good and help us improve the site.

Overall, the legal team has reviewed the site and they also raised the font loading from Google as a "concern" but not a stopper. Given that you have both noted it, I think we should add the fonts TO the site as you suggest.

On the BebasNeue typeface, the design agency wanted to use it for certain graphics so it's use is limited. Because the majority of the site is in our open source brand font (Montserrat) we had accepted this artistic choice as a "limited decorative" element.

The Privacy policy has been created and is here -> https://wikimediafoundation.org/wiki/Wikimedia_Foundation_Annual_Report_Privacy_Policy

Let us know if this policy does not cover the items you have raised.

Also, I think your other notes are very very helpful too. We obviously want that community link staying within the site, and we want search engines to index the site too. On the Favicon, once again, it's something we want.

I am going to request these changes from the vendor now. We'll look to get correct files up next week, and we can push out the intended launch date to Tuesday - Wednesday to make sure it's all working right.

On the BebasNeue typeface, the design agency wanted to use it for certain graphics so it's use is limited. Because the majority of the site is in our open source brand font (Montserrat) we had accepted this artistic choice as a "limited decorative" element.

Is the agency providing the license (ie. buying) so that Wikimedia can use this typeface? For how many visitors? I guess the number of visitors we want to support coming there will be "a lot". Yet, they are using some kind of volume licensing:

The total traffic, measured in page views, of all Websites Using a Licensed Web Font must be no greater than the number of page views specified in your account at located at www.myfonts.com

I'm not sure if this is a raw number of page views or a number of page views per month. In any case, what will happen if/when such number that the agency initially buys is surpassed? Is the agency providing an alternative version that can be switched to? Or will some other developer need to change this in the future in order to remove the usage of this proprietary font? ☹

Including tracking code would of course be unacceptable, but there doesn't seem to be any right now (is the agency violating the font license? Maybe they have a different license from the provider). This should be made clear with them (they are responsible for any violation, etc.).

ZMcCune added a comment.EditedJan 12 2018, 6:24 PM

UPDATE: we will have revised files in the next few hours.

@Bawolff , would you want to check them again to be sure the changes requested are implemented?

UPDATE #2: Files corrected per notes. They are here -> https://drive.google.com/drive/u/1/folders/1sdMrFOTMhOGV9w1Eb-Fhvw_gAt3xIoEG

I am sorry I still don't know how to properly add these to Gerrit. Looking for Greg Varnum to help me do it properly, but if someone else wants to upload it in the meantime I would be much obliged

@ZMcCune @Bawolff. I have some good news on the BebasNeue font. It seems to be floating around some different sites, but it appears that it has been freely licensed under the Open Font License (http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=OFL) which is an acceptable free license for font usage. It shows up under this license in a few places, including the original designers website at http://dharmatype.com/license
So, you've got Legal approval for using that one.

@ZMcCune @Bawolff. I have some good news on the BebasNeue font. It seems to be floating around some different sites, but it appears that it has been freely licensed under the Open Font License (http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=OFL) which is an acceptable free license for font usage. It shows up under this license in a few places, including the original designers website at http://dharmatype.com/license
So, you've got Legal approval for using that one.

Which also means we should host it ourself rather than leaking to Google too

@Reedy Correct. The update site files include a fonts folder to do that.

Greg Varnum and are working on the uploads now.

@ZMcCune @Bawolff. I have some good news on the BebasNeue font. It seems to be floating around some different sites, but it appears that it has been freely licensed under the Open Font License (http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=OFL) which is an acceptable free license for font usage. It shows up under this license in a few places, including the original designers website at http://dharmatype.com/license
So, you've got Legal approval for using that one.

Awesome :D

We kept hitting Phabricator's upload file size limit - so had to split to still be upload to upload via Phab ticket. :)

It'd be much easier just pushing them straight to gerrit... :/

(I'd strongly recommend to give https://www.mediawiki.org/wiki/Gerrit/Tutorial a try here. Phab is not designed as a file hosting service...)

@Reedy & @Bawolff -

Want to apologize for the AR team's general ignorance here on Gerrit. This is the only time we use it!

In the interests of time, could you help us get this properly uploaded and reviewed? We are trying to get this live ASAP for our donors, communities, and staff and I think it will take us more time (likely causing more frustration) for us to Gerrit efficiently.

If so, we are looking for your OK on the revised files and the redirect to annual.wikimedia.org and annual.wikimedia.org/2017 to resolve to this new site.

Let us know what's best to do next. This is our top priority :)

Dzahn added a comment.Jan 16 2018, 6:10 PM

Started to download the zip files then noticed @Reedy was so kind to already upload the files as PS2 on https://gerrit.wikimedia.org/r/#/c/401813/2

They are available there now.

Reedy added a comment.Jan 16 2018, 6:12 PM

@Reedy & @Bawolff -

Want to apologize for the AR team's general ignorance here on Gerrit. This is the only time we use it!

Honestly, if you're already pushing to other git repos... Pushing it gerrit isn't much harder. If you've got a change id, rather than just git push origin you can do git push origin HEAD:refs/for/master

Even at this point... Making a git patch of the commit would've been easier, rather than sending a split zip archive of a full git repo used to build it...

I think, this is harder work all round if you've got to split a zip archive (of a git repo) into many pieces to just upload them..

I'd suspect there would've been numerous people in the office in SF that could've helped make it easier... Less hassle all round :)

Anyway. https://gerrit.wikimedia.org/r/#/c/401813/ updated to include the changes in those zip files...

Dzahn added a comment.Jan 16 2018, 6:20 PM

I'm here to deploy it and switch the index.page to 2017 once you the review is done and you give me the GO signal. Did you just want "ASAP" or a specific release date?

I'd suspect there would've been numerous people in the office in SF that could've helped make it easier... Less hassle all round :)

I might be able to shed some light on a few things. :)

The AR Team was using an outside vendor - who was (to my knowledge) not sharing with our internal team in any method outside of Google Drives. We are going to be changing that moving forward - but happened to be the case here. So the source, for the people now involved, was the link above - so pushing to Gerrit was a bit more involved.

The basic goal had been to help AR Team do this within the tools provided - in this case Phabricator - in part in response to other messages about how they should seek help. In this ticket, they were told to upload the files to this ticket. Wanting to honor that request, those of us in the office who helped, helped them do that. We had pinged in IRC about this, but folks were not around. :)

Moving forward - Comms will work to make sure the vendors doing the tech work are also involved with uploading the files to Gerrit so as to avoid this altogether. However, we should also consider a better method of getting large files from internal teams to Dev teams as most outside Tech cannot use Gerrit, and just going around asking folks inside the office to help is not very effective for either side. There are some of us within teams who use Gerrit for our volunteer work, but unless we happen to be involved with one of these projects (which was not the case here), jumping in last minute is complicated. I think the belief was that Google Drive (since we use that elsewhere within the org) would be okay, but it appears here that is not the case.

@Dzahn We are ready to go live now.

I'm just catching up to this thread now. As a member of the AR team, I simply want to give a BIG thank you to Daniel, Sam, and Brian for all of your work on getting this report up and live. After reading through this task, it clearly has been a tremendous amount of work, and I deeply appreciate it. Our annual report is one of our main ways of sharing our work and impact with our donors, community, and users/readers, so thank you again for making this possible to share online :)

Dzahn added a comment.Jan 16 2018, 6:54 PM

@Dzahn We are ready to go live now.

Ok, once i see a positive review from security team i will simply go ahead and deploy.

I have one question about the code myself:

Can the .htaccess file be deleted? Should it be deleted? It says " Development Environment Settings" which makes it sound like it's not meant to be in Production.

And if special Apache config is needed we would prefer it in the main Apache config and not in an .htaccess.

https://gerrit.wikimedia.org/r/#/c/401813/2/2017/.htaccess

[Note, I am sort of away this week, so have limitted availability]

I still see the <link href="//fonts.googleapis.com/css?family=Montserrat:300,400,700" rel="stylesheet"> in the html files. These need to be removed. Once that is fixed, consider this approved.

It would be nice (but not critical) if the code comment around the BebasNeue font was changed to point to where it states that the font is available under an open font license (instead of the current all rights reserved).

Dzahn added a comment.EditedJan 17 2018, 4:45 PM

Ok, so please also see the comments from bawolff on https://gerrit.wikimedia.org/r/#/c/401813/

Can you amend to make the requested changes? @ZMcCune @Varnent

You could try using the web-based editor, it's easier to amend to an existing change than uploading a new one.

See the "Edit" button on the Gerrit link above, you click that to switch to Edit mode, then you select the files to change and once you are done with all you click "Done Editing" and then "Submit". You don't need any tools or command line for it.

We'll work on this today.

greg added a subscriber: greg.Jan 17 2018, 6:56 PM

Moving forward - Comms will work to make sure the vendors doing the tech work are also involved with uploading the files to Gerrit so as to avoid this altogether.

Suggestion to maybe make it easier for all: get Tech directly involved earlier on with all of these projects as well. That's a very important missing piece in most of these situations. They are technology choices, after all :)

Agreed @greg.

I've reviewed our timeline for this year's project and noted we had a 1hr alignment meeting between this group (security review) + Comms + the tech vendor on December 6, 2017. We can do earlier for next year/future projects, but we'll also need commitments to really make use of these alignment meetings to clarify precisely who and how we should coordinate tech hand-offs. Earlier meetings are only as good as the clarity they create.

*Correction: meeting was 30 minutes. Perhaps meeting length could also be extended to improve results and understanding.

Dzahn added a comment.EditedJan 17 2018, 8:50 PM

I don't think we need longer meetings. Everybody agreed on that meeting. I think the problem was the short time span between uploading of the files and a release date that had already been committed to internally but wasn't known to all involved parties. Security reviews often mean that changes have to be made and it should be expected that severeal rounds of updates may be needed.

Change 404873 had a related patch set uploaded (by Varnent; owner: Chad):
[wikimedia/annualreport@master] Merge "Updates per security review"

https://gerrit.wikimedia.org/r/404873

@Dzahn, @Reedy - let me know if there are any problems with that upload. Pinging @ZMcCune will probably get fastest response from us. :)

Change 404873 abandoned by Catrope:
Merge "Updates per security review"

Reason:
Instead submitted as an amendment to https://gerrit.wikimedia.org/r/#/c/401813/

https://gerrit.wikimedia.org/r/404873

Change 401813 merged by Dzahn:
[wikimedia/annualreport@master] Add annual report 2017 files

https://gerrit.wikimedia.org/r/401813

Change 404877 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[wikimedia/annualreport@master] switch index link from 2016 to 2017

https://gerrit.wikimedia.org/r/404877

I amended that to delete the .htaccess file (since there wasn't really an answer on T179974#3903568)

bawolff had said it's approved once the googleapis links are gone, i confirmed they were gone, merged and deployed on bromine.eqiad.wmnet

Now there is https://annual.wikimedia.org/2017/

(but the index page doesn't link to ./2017/ yet)

I uploaded a change to switch it from 2016 to 2017

https://gerrit.wikimedia.org/r/#/c/404877/

@ZMcCune Do you want me to switch https://annual.wikimedia.org/ now to forward users to https://annual.wikimedia.org/2017/ ?

Dzahn changed the task status from Stalled to Open.Jan 17 2018, 10:47 PM

Amazing! There it is! Looking so good! THANK YOU GREG & BRIAN & DANIEL!

Macro seal-of-approval: That feeling  when the site is live

Yes @Dzahn, we are ready to update annual.wikimedia.org to redirect to the 2017 sites.

Change 404877 merged by Dzahn:
[wikimedia/annualreport@master] switch index link from 2016 to 2017

https://gerrit.wikimedia.org/r/404877

Dzahn added a comment.EditedJan 17 2018, 11:42 PM

Yes @Dzahn, we are ready to update annual.wikimedia.org to redirect to the 2017 sites.

Done.

It might be cached in browsers and on varnish, but not for too long.

https://annual.wikimedia.org/?x

I think this resolves this ticket.

Dzahn closed this task as Resolved.Jan 17 2018, 11:47 PM

Agreed @greg.

I've reviewed our timeline for this year's project and noted we had a 1hr alignment meeting between this group (security review) + Comms + the tech vendor on December 6, 2017. We can do earlier for next year/future projects, but we'll also need commitments to really make use of these alignment meetings to clarify precisely who and how we should coordinate tech hand-offs. Earlier meetings are only as good as the clarity they create.

At the time, I did suggest leaving 3 weeks for security review, just in case. In total the process took about 2 weeks, so we're not over the expected time frame (And honestly, I only even noticed it was ready on the ninth due to winter vacation). That meeting in dec did mention a jan 15 goal for deployment, but honestly I didn't really remember that as the meeting was a long time ago once this came up for security review. Reminding us of the intended deadline once it came up for security review would have been helpful.

Having the vendor put directly in git would probably have quickened the process as it reduces the work we need to do (Thank you Daniel for doing that work)

This month we had much lower availability on the security team, between Darian leaving, christmas vacation and pre-allhands "personal" travel. Normally I probably would have been more on top of reviewing this quicker