Page MenuHomePhabricator

Persistent login session issue on some wikis (due to XCache not saving data across requests?)
Closed, ResolvedPublic

Description

Since yesterday (not sure if this is a coincidence, but from when someone requested a password reset of an account created with my primary account), I have been logged out of at least Commons, Wikidata, Wikipedia and Wikisource getting the following error message when trying to log back in:

There seems to be a problem with your login session; this action has been canceled as a precaution against session hijacking. Go back to the previous page, reload that page and then try again.

At the same time, I have not been logged out of at least Meta, MediaWiki.org, Wikivoyage and the German Wikipedia, where I can still log out and back in freely. Emptying the cache didn't make a difference, while using my browser's private mode did (perhaps a corrupted cookie?).

Event Timeline

FDMS created this task.Jun 17 2016, 10:03 AM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 17 2016, 10:03 AM
Anomie added a subscriber: Anomie.Jun 17 2016, 3:12 PM

That message typically indicates a problem with your browser's cookie handling. You might try clearing all cookies on the affected sites, and double check that you haven't blocked cookies somehow.

Otherwise, please capture the HTTP requests and responses for a login attempt (both loading Special:UserLogin then submitting the form), redact your password, and post them. You might upload them to Phabricator at https://phabricator.wikimedia.org/file/upload/ and set visibility to only yourself and me.

Tgr added a subscriber: Tgr.Jun 17 2016, 6:41 PM

The internal error message was Metadata has an anonymous user, but a non-anon user was provided.

FDMS added a comment.Jun 17 2016, 10:05 PM
This comment was removed by FDMS.

I have had this error since authmanager was added in may. See T138168 please.

From the screenshot you posted in a since-removed comment, it appears that your browser is not sending the cookies back to the server with the post. But in that case it's odd that the server would be attempting to delete a commonswikiUserID cookie, which makes me suspect that Safari's web inspector doesn't display all headers. This seems to be confirmed by some quick web searches, unfortunately no results on how to get unfiltered headers.

To see what's going on, I could really use the following:

  • The HTTP request sent when first loading Special:UserLogin
  • The HTTP response headers received when first loading Special:UserLogin
  • The HTTP request sent when posting the login form
  • The HTTP response headers received from posting the login form.
    • It should be a redirect, so include all the requests and responses from when the browser follows the redirects too.
  • The HTTP requests and response headers for any requests to Special:CentralAutoLogin during or just after the login process.

There has been some other reports about this as well in mediawiki.org support desk, and all seem to be related to a MediaWiki 1.27 upgrade:

Can't login or create user after upgrade to 1.27

Anomie added a comment.Jul 5 2016, 7:23 PM

As mentioned above, to really debug these sorts of issues it would really help to see the HTTP request and response headers both when first loading Special:UserLogin and when posting the form. If available, the 'session' and 'cookie' log channels for both of those would also be handy (the thread you link has only the logs for an unsuccessful post).

For third-party wikis, it would also be helpful to know if the wiki is using any extensions that have open subtasks of T110291 or any extensions that aren't in Gerrit.

But not everyone know how to get that information.

Http request.

Paladox added a comment.EditedJul 5 2016, 11:32 PM

@Anomie Also trying with out CentralAuth brings the same error. So maybe CentralAuth not the problem.

But defiantly to do with the new auth manager.

How do I debug by doing what you say please. I'm not sure how to do what your asking.

Anomie added a comment.Jul 6 2016, 3:50 PM

In the initial response (line 8), we have

Set-Cookie: wiki_session=pokoij4r7tf4dbub3pjd7k35k588o6u4; path=/; httponly

But when you POST the form back to the server, your browser isn't sending the same session cookie back: (line 73)

Cookie: _ga=GA1.2.2001277872.1467777763; _gat=1; __atuvc=3%7C27; __atuvs=577c82e3216a1121002; wiki_session=jqm1bp9vjhe8i1l5drf13n64dhnlkk47

That would certainly cause the session error.

On the other hand, since the Referer headers in the paste reveal the address of the wiki, I was able to try it myself to see if a random username and password would fail due to session or due to the user not existing. Despite my browser submitting the same cookie on post-back, it still gave the session error.

If you have access to the server, capturing the 'session' and 'cookie' channels at debug log level during the two requests would likely be helpful.

Tribly added a comment.Jul 6 2016, 4:13 PM

If you have access to the server, capturing the 'session' and 'cookie' channels at debug log level during the two requests would likely be helpful.

Could you give me a quick guide on how to do that?

Anomie added a comment.Jul 6 2016, 4:32 PM

https://www.mediawiki.org/wiki/Manual:How_to_debug#Logging has some detail. The TL;DR version is to set $wgDebugLogFile to point to some file the webserver can write, assuming you haven't messed with $wgDebugLogGroups.

Tribly added a comment.Jul 6 2016, 4:50 PM

Cool, thanks! I'll be back with the necessary logs.

Tribly added a comment.Jul 6 2016, 4:57 PM

Should I post my whole debug log or am I looking for a specific part?

This comment was removed by Innosflew.
Tribly added a comment.Jul 6 2016, 5:28 PM
This comment was removed by Tribly.
Tribly added a comment.EditedJul 6 2016, 7:19 PM

Alright, so I tried to switch back to the 1.26 version of mediawiki(I made a backup folder before updating to 1.27, so I only had to rename that folder to html to switch back, and rename the 1.27 folder to something else.) and when I tried to login I got the following error:

Wiki uses cookies to log in users. You have cookies disabled. Please enable them and try again.

I tried to login with other accounts as well and tried other browsers as well but every time I got the same error. So I googled the error and got this post: https://www.mediawiki.org/wiki/Topic:Rg3w5u0e70fs8l4e

The solution to the post was:

The session cookie are not working when session.referer_check = On is set in php.ini.

Maybe there sould be iniset in php.ini or at least a check for this setting

So I go into my php.ini file and searched for session.referer_check and find that there was no value set for session.referer_check and it was like this:

session.referer_check =

So I set session.referer_check to Off:

session.referer_check = Off

Then I restarted apache with:

sudo service apache2 restart

And after that I was able to login again with every account. I switched back to 1.27 and it seems to be working there as well.

Also wanted to try out my wiki with this method with PHP7 but it was not working. Seting the session.referer_check to Off in the PHP7's php.ini had no effect. So I set back pnp.ini to it's default state(session.referer_check =) then I looked through my php extensions and I notified that apcu was missing for PHP7. I installed it and now it seems to be working with PHP7 without even the need to modify the php.ini. I also tried to login in private browsing and it worked. My fellow staff members have also tried to login and it's working for them.

For PHP5 I have the following extensions installed:

apcu.ini
curl.ini
gd.ini
imagick.ini
intl.ini
json.ini
mcrypt.ini
mysql.ini
mysqli.ini
opcache.ini
pdo.ini
pdo_mysql.ini
readline.ini
xcache.ini

And For PHP7 I have the following extensions installed:

apcu.ini
apcu_bc.ini
calendar.ini
ctype.ini
dom.ini
exif.ini
fileinfo.ini
ftp.ini
gettext.ini
iconv.ini
json.ini
mbstring.ini
mysqli.ini
mysqlnd.ini
opcache.ini
pdo.ini
pdo_mysql.ini
phar.ini
posix.ini
readline.ini
shmop.ini
simplexml.ini
sockets.ini
sysvmsg.ini
sysvsem.ini
sysvshm.ini
tokenizer.ini
wddx.ini
xml.ini
xmlreader.ini
xmlwriter.ini
xsl.ini
Anomie added a comment.Jul 6 2016, 7:30 PM

@Tribly: Based on that and what I saw in your paste, it looks like we can chalk yours up to accidentally using a non-persistent cache for the session storage (as well as for all other caching).

Tribly added a comment.Jul 6 2016, 7:44 PM

Accidentally using a non-persistent cache? Could you explain this a little? I'm not sure how I did that, but so I can avoid it in the future.

Anomie added a comment.Jul 6 2016, 8:05 PM

You didn't have APC enabled, so it was using XCache instead. I don't know if XCache as you have it configured just doesn't save data across requests, or if you have multiple servers and the XCache cache isn't shared (i.e. server2 can't see the data in server1 put in XCache). Either way, the result is that session data wasn't properly saved across requests.

Tribly added a comment.Jul 6 2016, 8:36 PM

Oh I see. Thank you for your help.

I have xcache also installed.

It looks to me that the switch in 1.27 to store sessions in object cache by default should have switched also $wgSessionCacheType to NONE by default, so people enable it only if they know what they're doing. Maybe there's a good reason to reinvent the wheel (session storage), but landing this change in a LTS release 1.27 without evaluating the impact on existing and new installations outside of WMF was a bit risky IMHO.

Default settings should be the "it just works" ones, and then tune them up in LocalSettings as needed. I'm wondering if it's also breaking the installer, where those settings can't be overriden.

Anomie added a comment.Jul 7 2016, 2:28 PM

It looks to me that the switch in 1.27 to store sessions in object cache by default should have switched also $wgSessionCacheType to NONE by default, so people enable it only if they know what they're doing.

No, doing that would cause everyone's session handling to break until they set $wgSessionCacheType to something reasonable.

Maybe there's a good reason to reinvent the wheel (session storage),

We need the session data to be tied to the authenticated user, otherwise you're liable to have one user's session data being used while authenticated as a different user.

PHP's session mechanism is very limited in this respect: the session is identified by a single cookie, and everything would have to use that cookie to determine who the authenticated user is. That works if you're using the cookie for sessions, but things like OAuth and CentralAuth's centralauthtoken mechanism had to do some crazy hacks to get around it.

SessionManager allows OAuth, CentralAuth's centralauthtoken, BotPasswords, and so on to work more sanely, since the session ID no longer needs to be tied to a single cookie no matter what mechanism is being used to determine the authenticated user for the request. Unfortunately, that means we can't easily use PHP's filesystem-based storage for the session data, since PHP gives no access to that except via its limited session handling interface.

but landing this change in a LTS release 1.27 without evaluating the impact on existing and new installations outside of WMF was a bit risky IMHO.

What impact? That people with odd settings for $wgMainCacheType need to set $wgSessionCacheType to something sensible, while for most installations (i.e. an installation using the defaults, or any installation that uses a shared persistent cache for $wgMainCacheType) it should just work fine?

Default settings should be the "it just works" ones, and then tune them up in LocalSettings as needed.

The default settings do "just work". I don't know where you got the idea that the default for $wgMainCacheType is not CACHE_NONE, which allows the other caches to fall back to CACHE_DB via CACHE_ANYTHING.

I'm wondering if it's also breaking the installer, where those settings can't be overriden.

There were a few installer bugs (T126177), but they have long since been fixed. The thread you link to seems to be T139559, which has been closed as being PHP bug 69188 (INF == 0 on an old version of Solaris).

Aklapper renamed this task from persistent login session issue on some wikis to Persistent login session issue on some wikis (due to XCache not saving data across requests?).Aug 1 2016, 9:54 AM
Aklapper triaged this task as Low priority.

But I doint use xcache now I use php 5.6 and the problem still happends.

Ejcaputo raised the priority of this task from Low to Needs Triage.Aug 5 2016, 8:07 AM
Ejcaputo added a subscriber: Ejcaputo.

This should not be LOW priority. It is easy to make the problem happen, and it has a big impact when it does. On my system (Ubuntu 16.04, PHP 7), I haven't found a reliable way to prevent the problem from occurring:

  • With "$wgReadOnly='something...'", and "$wgMainCacheType = CACHE_DB" or "CACHE_ANYTHING", the system shows an error message "Database is read-only: something...", and doesn't show the normal login screen
  • Changing "$wgMainCacheType = CACHE_NONE" forces the "highjack" error message when login is attempted
  • Removing $wgReadOnly and using "$wgMainCacheType = CACHE_DB" gets the "highjack" error very infrequently.

Request that the problem be upgraded and a work-around for using $wgReadOnly and/or eliminating the problem completely at least be identified. Maybe the session data can be written to a file, like it was before. I'm thinking that maybe $wgObjectCaches could be used to create a custom cache which would work, but I haven't found sufficient documentation or a good example program yet. However, I did try with "$wgMainCacheType = 'hash'" and that produced the "highjack" error also. It is possible that 'SqlBagOStuff' is a sufficiently good example program to get it to work, I'll try it.

Tgr added a comment.Aug 5 2016, 8:52 PM

With "$wgReadOnly='something...'", and "$wgMainCacheType = CACHE_DB" or "CACHE_ANYTHING", the system shows an error message "Database is read-only: something...", and doesn't show the normal login screen

That's the expected behavior. If you use the DB for session storage and make it read-only, sessions won't work.

Changing "$wgMainCacheType = CACHE_NONE" forces the "highjack" error message when login is attempted

Again, expected behavior. If you blackhole the session storage, sessions won't work.

Removing $wgReadOnly and using "$wgMainCacheType = CACHE_DB" gets the "highjack" error very infrequently.

That sounds like a bug (either with MediaWiki or with your infrastructure). We would need more details to even try to diagnose it.

Maybe the session data can be written to a file, like it was before.

I guess we could try to add a CACHE_FILE type. I have no idea how well the performance of manual file-based caching would hold up to PHP's internal file-based session storage, though.

I'm thinking that maybe $wgObjectCaches could be used to create a custom cache which would work, but I haven't found sufficient documentation or a good example program yet.

What information are you missing?

I guess the docs should spell out that cache implementations need to be BagOStuff subclasses but it's quite obvious just by looking at that the default value of $wgObjectCaches. (And yes, BagOStuff is terrible. I don't think that can be helped by documentation, though.)

However, I did try with "$wgMainCacheType = 'hash'" and that produced the "highjack" error also.

Hash means using a PHP array for caching. Cache types which do not survive the end of the request are of course not great for session storage.

In practice you are probably best off by CACHE_MEMCACHED or 'apc' (which means installing one of those but that's not hard). That's somewhat documented at $wgMainCacheType although no doubt that could be improved.

Thanks for the help in finding a work-around for the problem. Finally, I used memcached to get my wikis to mostly work in read-only mode. The only problem left is a message "Error creating thumbnail: Unable to save thumbnail to destination" which I get for links to .PNG files, but not .SVG files, which I can live with. My settings in LocalSettings.php are:

$wgMainCacheType = CACHE_MEMCACHED;
$wgMemCachedServers = array( "127.0.0.1:11211" );
$wgSessionsInObjectCache = true;
$wgSessionCacheType = CACHE_MEMCACHED;

Still, I think that it would be helpful to mention in the documentation what it takes to get $wgReadOnly to work properly. The manual https://www.mediawiki.org/wiki/Manual:$wgReadOnly says to use:

$wgMessageCacheType = $wgMainCacheType = $wgParserCacheType = CACHE_NONE;
$wgLocalisationCacheConf['storeClass'] = 'LCStoreNull';

which doesn't work.

Tgr added a comment.Aug 9 2016, 10:40 PM

The only problem left is a message "Error creating thumbnail: Unable to save thumbnail to destination" which I get for links to .PNG files, but not .SVG files, which I can live with.

That's probably unrelated; generating a thumbnail should not involve sessions or any other kind of cache. Sounds like a directory configuration error. If it was caused by the 1.27 update, and you are willing to do some debugging, feel free to open a separate task and CC me on it.

$wgMainCacheType = CACHE_MEMCACHED;
$wgMemCachedServers = array( "127.0.0.1:11211" );
$wgSessionsInObjectCache = true;
$wgSessionCacheType = CACHE_MEMCACHED;

You only need the first two; $wgSessionsInObjectCache is deprecated and has no effect in 1.27, and $wgSessionCacheType falls back to $wgMainCacheType.
(I wonder why $wgMemCachedServers doesn't default to [ '127.0.0.1:11211' ]? 11000 is not a standard port for memcached, or even redis.)

Still, I think that it would be helpful to mention in the documentation what it takes to get $wgReadOnly to work properly. The manual https://www.mediawiki.org/wiki/Manual:$wgReadOnly says to use:
$wgMessageCacheType = $wgMainCacheType = $wgParserCacheType = CACHE_NONE;
$wgLocalisationCacheConf['storeClass'] = 'LCStoreNull';

Clarified the docs a bit. Note that login might break anyway because in some cases it needs to write to the (non-cache-related) DB tables. Logout will definitely break.

That problem with the thumbnails only occurs when in read-only mode, otherwise it works ok. I tried looking at it yesterday, but I didn't see anything. I'm not sure which version caused this problem, my wikis were 1.17 and 1.22 previously. I'll look a bit more at it today, and if I see anything, I'll open a new task.

Thanks again!

That problem with the thumbnail error in read-only mode is solved by setting $wgIgnoreImageErrors=true;

Apparently it was done intentionally to avoid generating thumbnail images in read-only mode (see functions transformErrorOutput() and transform() in includes/filerepo/file/File.php.

Recommend that the documentation for $wgReadOnly be modified to reflect this. Should this be a new task?

Tgr added a comment.Aug 10 2016, 9:03 PM

Recommend that the documentation for $wgReadOnly be modified to reflect this. Should this be a new task?

Feel free to edit it, it's a wiki :)

Tgr closed this task as Resolved.Aug 11 2016, 8:47 PM
Tgr claimed this task.

Thanks.

I think there is nothing actionable in this bug (unless someone is willing to debug their issues and provide more details, or test XCache to see when does it persist/share information, in which case it could be added to the documentation) so it can be closed. If someone is willing to provide detailed debug information or can suggest specific changes in MediaWiki, they should feel free to reopen (or file a new task).

Cboltz added a subscriber: Cboltz.Aug 21 2016, 11:17 PM

FWIW, I had this problem too at https://freephile.qualitybox.us/wiki/

Product	Version
MediaWiki	1.28.2 (438c3d6)
PHP	5.6.31 (apache2handler)
MariaDB	5.5.56-MariaDB
ICU	50.1.2
Elasticsearch	2.4.6
Lua	5.1.5

I was using memcached for session storage. Changing session.referer_check = (blank) to session.referer_check = Off in php.ini and restarting Apache did the trick for me. I also restarted memcached for good measure.

MaxSem reopened this task as Open.EditedOct 2 2017, 10:01 PM
MaxSem added a subscriber: MaxSem.

Reopening, I see lots of Metadata has an anonymous user, but a non-anon user was provided for the affected user when investigating this report of a user being unable to log in.

Tgr added a comment.Oct 2 2017, 10:15 PM

Can we have a separate task for that? This one is about some cache backends (not the ones used in WMF production) probably not working reliably.

Anomie closed this task as Resolved.Oct 3 2017, 1:30 AM

Yes, please. This bug started out with a vague complaint about failure to stay logged in on some WMF wikis but not others. The necessary browser request data was asked for but not fully provided. Then the bug got taken over by a vaguely similar issue on a non-WMF wiki, which was diagnosed and resolved and this task was retitled and resolved accordingly.

With that confusing history, let's not reopen this task. And without the data I requested in the first reply, there's likely very little we can do to debug this since so many different things can cause the symptoms described. As Gergő noted on IRC, the log message "Metadata has an anonymous user, but a non-anon user was provided" indicates that the session exists server-side and has the necessary array keys to be valid but indicates an anonymous user, while the cookies indicate a logged-in user. And further, the '-' in the pattern <-:12345:UserName> you mentioned in IRC indicates that neither the CA cookies nor the token cookie were used to validate the user. To take it further than that, we'd need to see the cookies the user's browser is sending and the set-cookies the server is responding with, so please include that information if you file a new task for your issue.