@AMJohnson Which email(s) could you reproduce this on? I tried a couple of recent seeds and they seemed to work okay. If you could share the link which gives the problem that would be great

I'm also receiving this error message when visiting donate.wikimedia.org or if you click donate from sidebar on WP. But no error if we use a geo-link such as https://donate.wikimedia.org/w/index.php?title=Special:LandingPage&uselang=en&country=US.

The links from donate.wikimedia.org/ The sidebar etc appear to be inputting XX for country at the moment: https://donate.wikimedia.org/w/index.php?title=Special:LandingPage&country=XX&uselang=en&utm_medium=sidebar&utm_source=donate&utm_campaign=C13_en.wikipedia.org

Do we want to add an unbreak now tag @Pcoombe?

Huh, the reason I couldn't reproduce this earlier is because it seems to work when logged in to donatewiki. But not when logged out

https://gerrit.wikimedia.org/r/827566 is being reverted as we speak

Thanks to SRE for engaging with our "did something happen with geoip cookies?" question quickly!

Thank you all for the quick fix! Inviting donors to retry.

@AMJohnson great! Just make sure they don't accidentally try to reload using the error producing url (ones that have "country=XX").

Good Point @greg! When we invite to retry we point them to our standard form at donate.wikimedia.org. Ty again for the quick assists! :)

@XenoRyet howdy! I see this is resolved, but this happened to me today, when I clicked on donate wiki ... so I'm posting just in case?

Steps to reproduce:

1. I navigated to the portal >
2. English >
3. Donate (in the sidebar) and saw the error.

When I refreshed, it was gone and country=US appeared correctly.

After talking with the folks in #wikimedia-sre, they're not aware of any changes that could have caused this latest manifestation of the problem. One explanation is that this could be happening due to browsers blocking 3rd-party-cookies. Firefox, Chrome Incognito and Microsoft Edge block 3rd-party-cookies by default.

As you can see on the screenshot below using Chrome Incognito, the cookie headers coming in which are meant to set the GeoIP cookie are being blocked by the browser (the tooltip on the yellow warning sign)

@BBlack suggested the following potential fix:

19:30 <bblack> one way to reliably solve this in the face of all this, would be to have the Special:FundraiserRedirector use a two-step redirect process
19:31 <bblack> (redirect to itself once with some extra parameter like &phase=1, then strip that away in the final redirect)
19:31 <bblack> the cookie would get set with the first redirect, and consumed by the second
19:32 <-- hauskater (~MarcoAure@wikimedia/marcoaurelio) has quit (Quit: Leaving)

For the full conversation please check out the IRC logs here

Thanks so much, @jgleeson, @BBlack, @Vgutierrez!

I agree the most likely explanation is that cross-domain cookies from the background request to login.wikimedia.org previously worked to set the GeoIP cookie for wikimedia.org, but no longer always do.

The two-step redirect also seems like a straightforward fix to ensure things always work.

In fact, it looks like we are actually doing that in links in Fundraising e-mails, or at least in some samples that I saw. Here's an example:

https://donate.wikimedia.org/?utm_campaign=C2223_Email2&utm_medium=email&appeal=Appeal-JimmyQuote&utm_source=sp71918569&hpc=&uselang=&contact_id=&link_id=1&contact_hash=&monthlyconvert=Default

That link redirects to Special:FundraiserRedirector, so GeoIP cookie set on the first request is sent back from the browser and reaches our PHP code, so that redirects to Special:LandingPage with the appropriate form (and with the URL params preserved, too).

However, the sidebar link goes directly to Special:FundraiserRedirector, causing the GeoIP cookie set in Varnish to not be available in PHP on that request.

@AMJohnson, @Pcoombe, @TSkaff can we confirm that none of the links we send (or have recently sent) in e-mails go directly to Special:FundraiserRedirector? (Any that do will likely be giving at least some users this error.)

Also, for the sidebar link, how can we change that to remove the Special:FundraiserRedirector part?

Finally, another clue about why the cross-domain cookies may not be working: in non-incognito mode in Chrome, the same tooltip shown in @jgleeson's screenshot says the cookie was blocked, and that the reason was a lack of a SameSite attribute.

In the Firefox console, I see a message in the console saying the cookie will soon be blocked for the same reason:

Cookie “GeoIP” does not have a proper “SameSite” attribute value. Soon, cookies without the “SameSite” attribute or with an invalid value will be treated as “Lax”. This means that the cookie will no longer be sent in third-party contexts. If your application depends on this cookie being available in such contexts, please add the “SameSite=None“ attribute to it. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite

Thanks again, all!!!!!

In the Firefox console, I see a message in the console saying the cookie will soon be blocked for the same reason:

Cookie “GeoIP” does not have a proper “SameSite” attribute value. Soon, cookies without the “SameSite” attribute or with an invalid value will be treated as “Lax”. This means that the cookie will no longer be sent in third-party contexts. If your application depends on this cookie being available in such contexts, please add the “SameSite=None“ attribute to it. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite

Thanks again, all!!!!!

Nice find! @AndyRussG

@AndyRussG, and team, Amber is off this week. What I can say is that all our messages contain this link: https://donate.wikimedia.org and when I click on it, it takes me to "Special:LandingPage&country=US" which is where I'm based. Is there anything else I should be looking into? Apologies for not helping much!

Thanks all. I searched the seed emails in my inbox, and none have used Special:FundraiserRedirector, so emails should all be fine.

Updating the sidebar links would be very difficult, as they are set on every wiki separately e.g. https://en.wikipedia.org/wiki/MediaWiki:Sitesupport-url, and need an admin to edit. In the longer term we really ought to centralise this somehow. For now it would be better if we can fix this issue on donatewiki somehow.

I've added some on-wiki javascript to tackle this. If a user reaches Special:LandingPage with country set to "XX" then it will try sending them round once more, keeping all the same parameters except country. On this next try the cookie should be available for use, and they will get the correct country landing page. Tested this with the sidebar links and it worked, although I would appreciate if others could confirm.

https://donate.wikimedia.org/wiki/MediaWiki:Common.js#L-9

Not sure if fr-tech still want to try fixing in the extension as well?

Hey @Pcoombe! oops I didn't see your message until now! there is also a fix for the FundraiserLandingPage extension in review now :) (See the attached subtask: T317427.)

Also, the JS-based fix you put up does work for me! I see it even rewrites the URL--cool! thanks much!!! :)

In T316578#8227652, @AndyRussG wrote:

I see it even rewrites the URL

K I see rather it's redirecting...! heheh it happened quite fast in my browser, hardly noticeable :)

I think the on-wiki solution is great, and it's already deployed and working. (I hear "wiki" means "quick", right?)

I think I'd recommend we still go ahead and deploy the server-side solution, as the server-side redirect will be more performant, with everything happening at the HTTP level and neither page rendering nor JS execution required. I guess it'd be a teeny bit more robust, also, since it doesn't rely on the the "XX" default country configuration.

Do we know if there are any plain Special:FundraiserLandingPage links in the wild, with no country param on them?

Thanks again! :)

Thanks @AndyRussG. Agree that we should go ahead with the server-side solution as being faster and more robust.

I don't think there is any use of Special:FundraiserLandingPage links without a country.

So it looks like we DO have the GeoIP v2 databases available on production: T303464
We should be able to restore the fallback server-side lookup we removed earlier this year, and just use the new dbs

After the GeoIP deployed, I find this bug is been resolved, that we do retry to get the country, please double check if this is still valid, thanks!

In T316578#8215105, @TSkaff wrote:

@XenoRyet howdy! I see this is resolved, but this happened to me today, when I clicked on donate wiki ... so I'm posting just in case?

Steps to reproduce:

1. I navigated to the portal >
2. English >
3. Donate (in the sidebar) and saw the error.

When I refreshed, it was gone and country=US appeared correctly.

I removed the javascript hack from Common.js, and the server-side change seems to be working well. Thanks!