@DStrine per our interlock today, the only potential dependency on FR tech is the need to use IP lookup to verify country restrictions for users who have not enabled location permissions.

In T145830#2653597, @JMinor wrote:

@DStrine per our interlock today, the only potential dependency on FR tech is the need to use IP lookup to verify country restrictions for users who have not enabled location permissions.

[For iOS:]
In the "permissions window" where a user agrees to share locations or not, do we need to change the reason why we are asking for location (which we now do, due to Apple) to include "for donation purposes" and not just "show articles nearby".

Also, if IP lookup is not implemented, one solution would be to just not show these banners to those users.

Some additional points for consideration:

• iOS dev will likely need to do initial development given Android bandwidth and commitments
• if we can't do an API based location check, @Dbrant suggested that we could ask for a mobile page and detect if the banner is included, if so the existing system would have determined its an "okay" geo, and we could then request and show the message from the specialized endpoint.
• app teams will implement their own front ends, to be released by mid-November for testing
• would be good to consider this a basic "announce" system for the apps, since these messages could be used for other notices (a la central notice)
• should probably be aggregated into the /feed endpoint, so apps do not have to make a secondary call just for this item

Just some thoughts on the structure of the MCS endpoint:
As Josh mentioned, this should ideally be a generalized "announcement" endpoint that we can extend in the future. Here's how I would imagine the structure, roughly (feel free to comment/improve):

"announce": [
  {
    type: "fundraising",
    text: "Please help Wikipedia by making a donation!",
    textColor: "#fff",
    backColor: "#000",
    image: "https://image.url"
    action: {
        text: "Donate"
        url: "https://fundraising.url?campaign_id=12345&source=app"
    }
  },

  {
    type: "survey",
    text: "Would you like to take a brief survey?",
    action: {
        text: "Go",
        url: "https://survey.url?survey_id=12345&source=app"
    }
  },

  ...
]

• This announcement endpoint can be inlined with the existing "aggregated" endpoint.
• Clients who do not care about these announcements can ignore them safely.
• The server is responsible for the timing of which announcement is delivered at which time.
• The server is responsible for localizing the "text" strings based on the language parameter in the request.
• Ideally, the server would also be responsible for restricting announcements based on the geolocation of the incoming IP request, as well as applying a sampling rate, but realistically these things can be handled client-side for now.
• As a first pass, we can literally hard-code the announcement on the server, and time the deployment and un-deployment of the announcement to coincide with the Fundraising timeline.

This looks great. A minor concern is specifying the colors server-side. I'm not saying I'm opposed to this. We just need to explicitly state what we plan to do for dark mode. If we can just invert the colors, and that looks fine then great.

@Dbrant I would second @bearND's point about color. I think generally MCS endpoints should avoid making stylistic assumptions about how MCS data will be presented.

This announcement endpoint can be inlined with the existing "aggregated" endpoint.

It can and it's convenient for the client to not have to make a separate request. Yet I'm not 100% convinced that putting announcements into the aggregated feed endpoint is the best way, though. Announcements tend to be more temporal in nature and specific for WMF, whereas the feed content is supposed to be rather static and more of generic interest. An external client is probably not interested about surveys or fundraising.
While today's feed content changes throughout the day, we tend to not significantly change the output on prior days. (E.g. change prop may change some summary data associated with some page titles but not the titles themselves.)

Explore feed responses for a particular date are stored and cached. So, if you set a campaign to run from Dec 1 to Dec 31, and you request the feed content on your device on Jan 1 then the first response won't have the announcement but the previous days would have it. The client would need to check for age == 0 for announcements to avoid showing old announcements. I want to make sure that this is clear before we proceed with this.

I'm adding the Services guys to chime in if I misrepresented anything or want to add their opinions to that matter.

The server is responsible for localizing the "text" strings based on the language parameter in the request.

We haven't done localization in MCS yet. That would have to be setup first. Also, I think fundraising messages are best to be handled the same as the rest of the fundraising messages: "only admins are allowed to push new translations to a banner (by setting them to the 'Published' state)".

Ideally, the server would also be responsible for restricting announcements based on the geolocation of the incoming IP request, as well as applying a sampling rate, but realistically these things can be handled client-side for now.

MCS can restrict based on wiki project but that is not granular enough. Maybe with some Varnish tricks or similar? So, doing a client side follow-up request to check for which countries the announcement is allowed is probably best for now.

In conclusion I'm wondering if instead of trying to roll our own solution we should be using the same mechanisms that web uses for this (CentralNotice/Sitenotice). Although I admit that I don't know exactly how they use it.

I'll leave the technical discussion to continue, but on the question of using CentralNotice, it was discussed in detail with FR Tech and we decided to pursue a simple solution based on app client and MCS work, at least for this season's fundraiser. I'm happy to explain more about the reasons for this offline, but suffice it to say that integration with the existing stack may not even be ideal, and even if it is our eventual goal, if we want to do fundraising or notices in the apps any time in the next year we will need to build it ourselves.

In T145830#2690907, @bearND wrote:

This announcement endpoint can be inlined with the existing "aggregated" endpoint.

It can and it's convenient for the client to not have to make a separate request. Yet I'm not 100% convinced that putting announcements into the aggregated feed endpoint is the best way, though.
I'm adding the Services guys to chime in if I misrepresented anything or want to add their opinions to that matter.

There's a couple of requirements it would be hard to satisfy if we incorporate this into the feed endpoint.

The server is responsible for localizing the "text" strings based on the language parameter in the request.

Feed endpoint doesn't have a 'lang' parameter, but it has a domain parameter. If localising by domain is enough then we're fine. Then en.wiki feed would have text in English, ru.wiki in Russian etc. regardless of the users phone lang preferences.

We could add support for Accept-Language header, but feed content is cached in Varnish, so we'd need it to vary on the Accept-Language header which would lead to cache fragmentation. The hit rate for feed is not awesome as it is right now because of the random portion and varying it on one more parameter would make it even worse.

Ideally, the server would also be responsible for restricting announcements based on the geolocation of the incoming IP request, as well as applying a sampling rate, but realistically these things can be handled client-side for now.

This would lead to even greater Varnish cache fragmentation, basically making hit rate go to zero. It's completely unacceptable since feed endpoint is quite expensive and we want to leverage Varnish as much as possible.

The server is responsible for the timing of which announcement is delivered at which time.

As @bearND said, old feed content is stored in RESTBase, so we wouldn't really have control over on the server and would need to implement it on the client, for example by introducing the show_until field. Although we could not store this portion in RB and request it in MCS each time, but that might slow down the feed endpoint and it's not blazing fast even right now. But since it's actually almost static content it might not be that bad.

To sum up, we don't want to fragment Varnish cache for feed endpoint even more, so no server-side customisations per-user is possible. If that's acceptable from the product perspective, then we could incorporate it into existing endpoint.

I would imagine the geo location of the client is/can be used in more than just the context of announcements (Nearby, anybody?), so having the client determining it would be best for this purpose.

I'm not keen on including the announcements in the feed either. Not only does it complicate things on the technical side, but that doesn't sound good to me conceptually either: the feed endpoint aims at delivering current and relevant content to the user/client. IMHO, announcements and banners do not fall in that category (content).

I sort of agree, but the thing is, I'd much rather not have to make a separate network request to get the list of announcements, especially when most of the time there won't be any announcements (so the network request will be for nothing).

Another possibility might be to put something in the aggregated response that says announce=true, which signals to the client that there's an announcement available for retrieval, and we can then query the announcement endpoint separately. This would still require the aggregated endpoint to somehow communicate with the announcement endpoint to determine whether to say announce=true or false.

In T145830#2692466, @Dbrant wrote:

I sort of agree, but the thing is, I'd much rather not have to make a separate network request to get the list of announcements, especially when most of the time there won't be any announcements (so the network request will be for nothing).

If there is nothing to announce, then the argument of making a separate (hopefully, asynchronous) request is void, IMHO, as there is nothing to do. On the other hand, as you showed in your endpoint example, even if there is something to fetch, the payload will be rather small with minimal latency since the client-server connection would already be set up at that point in time.

If we need something quick then I propose we restrict this for just enwiki for the first iteration, so we don't have to worry about localizing the messages. It would also have the benefit of for smaller wikis to not make the extra calls if the app only checks for announcements if the app language is set to 'en'.

This being a "temporary" solution is more a reason to not add it to the aggregated feed.
We in Android app land should check why we're still end up with using HTTP/1.1 and not 2.0 yet. Maybe some leftover from T134817?
That and firing off concurrent requests from the app should alleviate some concerns about an extra request being made.

Just spoke with @bearND after looking over the end points and assessing the work/timing we are proposing the following:

V1: Deploy early November
Create a new endpoint to serve static JSON containing the text, a donation url, and display dates.
Support a subset of wikis/regions. At a minimum we support en, possibly some other wikis.

We could use the text from the banner API as a starting point, but it is very long for mobile, so I assume we will want to trim that for the apps.

V2: Deploy next year
Work to get a mobile text field included in the Central Notice API
Consume the Central Notice API in the MCS and massage the response for Mobile consumption
Serve all regions/wikis served by Central Notice
Look further into the implications of including it in the aggregated feed endpoint

I have a meeting with @JMinor tomorrow and will firm up details then -@Dbrant let me know if this works for you

Yep, looks good. Just to reiterate about V1: it's essential to have a whitelist of physical countries, as well as a whitelist of languages. (and all of this can be statically placed in the response for now)

In T145830#2693432, @bearND wrote:

We in Android app land should check why we're still end up with using HTTP/1.1 and not 2.0 yet. Maybe some leftover from T134817?

We are using HTTP/2. You can verify this by examining Response objects in the debugger. See discussion on PS2 of https://gerrit.wikimedia.org/r/#/c/294306/, and also https://github.com/square/okhttp/issues/2247 (which doesn't actually seem fixed to me...).

In T145830#2696671, @Mholloway wrote:

We are using HTTP/2. You can verify this by examining Response objects in the debugger. See discussion on PS2 of https://gerrit.wikimedia.org/r/#/c/294306/, and also https://github.com/square/okhttp/issues/2247 (which doesn't actually seem fixed to me...).

Thank you. The Retrofit log entries are indeed quite misleading. I'm glad we're using HTTP/2.

@bearND lets discus getting this implemented this week. I can code this one - need to get my feet wet on service dev anyways.

@Fjalapeno sounds good. Let me know if you run into any issues during the implementation. We should agree on the the output format soon, though. Here's an update of the strawman proposal:

"announce": [
  {
    startTime: "2016-12-01T17:11:12Z",
    endTime: "2016-12-06T17:11:12Z",
    type: "fundraising",
    text: "Please help Wikipedia by making a donation!",
    image: "https://image.url"
    action: {
        text: "Donate"
        url: "https://fundraising.url?campaign_id=12345&source=app"
    },
    countries: [
        "US",
        "CA"
    ]
  },

  {
    startTime: "2016-11-01T17:11:12Z",
    endTime: "2016-11-06T17:11:12Z",
    type: "survey",
    text: "Would you like to take a brief survey?",
    action: {
        text: "Go",
        url: "https://survey.url?survey_id=12345&source=app"
    },
    countries: [
        "US",
        "CA",
        "GB"
    ]
  },

  ...
]

Please update as needed.

Look for the GeoIP header to find countries

HTTP/1.1 200 OK
Date: Wed, 19 Oct 2016 16:17:53 GMT
Content-Type: application/json; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/aggregated-feed/0.5.0"
Connection: keep-alive
cache-control: s-maxage=300, max-age=60
etag: 20160608/4736d929-9617-11e6-a6a8-c8d4498725f1
access-control-allow-origin: *
access-control-allow-methods: GET
access-control-allow-headers: accept, content-type
access-control-expose-headers: etag
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
content-security-policy: default-src 'none'; frame-ancestors 'none'
x-content-security-policy: default-src 'none'; frame-ancestors 'none'
x-webkit-csp: default-src 'none'; frame-ancestors 'none'
x-request-id: 4721ca88-9617-11e6-a0a8-88ca12e02ad3
X-Varnish: 495110465, 1881872984 1881370477
Via: 1.1 varnish, 1.1 varnish
Age: 132
X-Cache: cp1054 miss, cp1067 hit/5
X-Cache-Status: hit
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Set-Cookie: WMF-Last-Access=19-Oct-2016;Path=/;HttpOnly;secure;Expires=Sun, 20 Nov 2016 12:00:00 GMT
X-Analytics: https=1;nocookies=1
X-Client-IP: 2601:43:1:c26e:c498:bb13:ccef:db0f
Set-Cookie: GeoIP=US:PA:Philadelphia:39.97:-75.18:v4; Path=/; secure; Domain=.wikipedia.org

@bearND the only other field I would add would be "title"

I assume we send this only when the domain is en.wikipedia.org. For all other domains we would return an empty 204 response. Since this is just temporary, we should also code the apps so that they would only request this for enwiki for now.

We also need an endpoint path for this. A few ideas are:

• notice
• announcement
• announce (haven't seen verbs used in path names yet)
• herald

I feel we should have something like a higher level in the path as well. Something on the same level as /page/, /media/, /feed/, /transforms/ that fits our current scheme. Maybe /feed/ could be used for the announcements but I'm not sure yet. Another idea is /site/.
Ideas:

• /feed/
• /site/

Any better names out there?

In the Reading / Services sync meeting people were in favor of adding the new endpoint under the /feed/ hierarchy.
Due to the experimental nature of this endpoint it's conceivable that it could go away in the future. The services team recommended that the clients handle a potential non-existence of the endpoint gracefully and should consider passing in the path to the announce feed endpoint via remote config.

Change 320712 had a related patch set uploaded (by Fjalapeno):
Add announcements endpoint

https://gerrit.wikimedia.org/r/320712

Change 320712 merged by jenkins-bot:
Add announcements endpoint

https://gerrit.wikimedia.org/r/320712

being deployed this afternoon

Change 322300 had a related patch set uploaded (by Fjalapeno):
Add official survey URL

https://gerrit.wikimedia.org/r/322300

Change 322300 merged by jenkins-bot:
Add official survey URL

https://gerrit.wikimedia.org/r/322300

@Fjalapeno or @JMinor: I was discussing the upcoming donation designs with the Reading design team and we were curious if it's possible to have different images / text for each platform? It looks like we could use two separate campaigns / IDs one for Android and one for iOS, just wanted to confirm that this is an okay approach though. Thanks!

@cmodeo It adds some complexity but, yes, we can run separate announcement for each platform by including destination URL for only one platform in each campaign. Ideally for v1 we would set a pretty simple baseline across platforms, but variance is possible.

Change 324506 had a related patch set uploaded (by Fjalapeno):
Extend survey through December

https://gerrit.wikimedia.org/r/324506

Change 324506 merged by jenkins-bot:
Extend survey through December and update Privacy Policy

https://gerrit.wikimedia.org/r/324506

Change 325335 had a related patch set uploaded (by Fjalapeno):
Add new survey link for android

https://gerrit.wikimedia.org/r/325335

Change 325335 merged by jenkins-bot:
Add new survey link for android

https://gerrit.wikimedia.org/r/325335

Change 325805 had a related patch set uploaded (by Fjalapeno):
Update start end times for surveys

https://gerrit.wikimedia.org/r/325805

Change 325805 merged by jenkins-bot:
Update start end times for surveys

https://gerrit.wikimedia.org/r/325805

I'd consider this resolved at this point... @Fjalapeno I'll let you or bernND do the honors.

Thanks guys!