Saying "you are disconnected" while saving my translated text on translatewiki.net

Échec de l’enregistrement de la traduction : Assertion that the user is logged in failed.

Previous issue (similar behaviour):

There has been issues with sessions that caused edits or logging in to fail for some users. The problem started around 20.00 UTC, possible even earlier, on the previous day. Possible fix has been applied, but please notify project admins if you continue to encounter this issue. (Other news …)

Sessions are rejected:
Votre session de connexion semble avoir des problèmes ; cette action a été annulée en prévention d'un piratage de session. Veuillez cliquer sur « Précédent », rechargez la page d'où vous venez, puis réessayez.

... but connexions and translations are accepted on other sites as:

https://www.mediawiki.org/w/index.php?title=Special:Translate&group=page-Manual%3AFAQ&action=page&filter=%21translated&language=fr

Did you try logging out and in again?

... out and in, and in and out, and new tab ,and new browser , and empty cache , and change browse and .... finally quit all.

More fluent at this time [Paris time 17;14] cxion allowed and save done successfully on translatewiki.net.

In T178242#3685951, @Wladek92 wrote:

... out and in, and in and out, and new tab ,and new browser , and empty cache , and change browse and .... finally quit all.

More fluent at this time [Paris time 17;14] cxion allowed and save done successfully on translatewiki.net.

I even tried clearing my cookie and ipconfig /flushdns, but still don't help, now CSRF errors blocked me from using Translate interface. So is this task UBN now?

I changed some configuration after my last comment here. Please try log out & log in once more.

disconection is quasi systematic now after a few minutes of connection and when saving translated text ; its 21:50 Paris time
some translation saves succeed but are not taken into account.
Note that the screen is not refreshed and the indication 'connected' in the upper right corner still remains after the failure poppup

After a connection, free navigation is allowed on any link of the page excepted when you acknowledge a rereading or validate your translated text where it fails.

In T178242#3685962, @Nikerabbit wrote:

I changed some configuration after my last comment here. Please try log out & log in once more.

Still CSRF blocked when editing e.g. InternetArchiveBot and XTools, OK now for most others.

Hi all, this morning [8:30 Paris time] :

After connexion and from a remote, separate PC + cleaned cache + latest Firefox updated:

> first immediate translation is/are accepted; ex: https://translatewiki.net/wiki/iNaturalist:Unblock/fr

BUT a few seconds later the same disconnection process applies. This is very unstable and prevent from continuing the translations for now.

Meta comment: It would be nice if logging would have the following properties:

• Output entries to the general error log when something is broken
• Be clear enough to understand what the error messages mean: are they severe, unexceptional or expected to happen on normal use
• Be clear enough to understand what would be the appropriate action to take: change setting, how to debug more, point out what is likely the cause

None of that is the case currently, and I am at loss how to debug this further.

[13:10 Paris time] more stable over 15minutes long; no disconnexion => better :-)

I had restarted memcached (for debugging) and increased maximum memory in case it was getting full. But I have doubts whether this is the cause because I still see similar log entries.

ok, fast responses for review ack and translations => stable for now. Thanks.

very stable; normal behaviour observed.

new disturbance observed 00.15 Paris time

In T178242#3697458, @Nikerabbit wrote:

Lowering priority due to lack of new reports, but I am not sure this is solved yet.

Still Invalid CSRF token in many messages on the new created Dissemin project:

OMFG! By re-logging recently:

It's not a problem with memcached: restarting hhvm helps. But also other hhvm instance works, and telnet works. So I guess this is a bug with HHVM and/or with MemcachedClient.

Out of curiosity is there a reason you're using the PHP memcache implementation and not the built-in HHVM extension?

Good question. The config we have is simply $wgMainCacheType = CACHE_MEMCACHED;. I would expect that to pick the best implementation. Is memcached-pecl the setting to use the built-in HHVM extension?

instability observed again; Paris time 1:34 PM

Thanks for the report. I have restarted hhvm again as a workaround. It seems it has been going on for a while again:

2017-10-29 19:04:25 web1.translatewiki.net translatewiki_net-bw_: Memcached error: timeout reading line from 127.0.0.1:11211
2017-10-29 19:04:25 web1.translatewiki.net translatewiki_net-bw_: Memcached error: timeout writing to 127.0.0.1:11211

I'm considering adding daily restarts while this issue is being investigated.

I enabled memcached-pecl now for testing, as it took less than 24 hours to start seeing those errors again.

First observattions:

1. Wow it has very noisy logging even with $wgMemcachedDebug = false;
2. Notice: Unable to unserialize: [1]. Unexpected end of buffer during unserialization. in /srv/mediawiki/tags/2017-10-31_06:43:17/includes/libs/objectcache/MemcachedPeclBagOStuff.php on line 143

Those PHP errors went away when I restarted memcached – I suppose those clients use a slightly different way to serialize data.

I haven't seen new errors since then, but this must run for couple of days more without errors before I declare that I have found a working configuration to work around the original issue.

I'm declaring my workaround stable for now, and re-purpose this task as general MediaWiki issue.

Very stable during all this month long . This problem has no longer appeared. Satisfied. Thank you Nk.

so it this task still "high priority"?

That would be for the platform team to decide. My guess would be that they deprioritize HHVM related things.

Should I add Vuln-CSRF here?

@Liuxinyu970226 I do not see any vulnerability mentioned in this task.

@Krinkle But the CSRF token errors really still shows to me, just when I'm trying log out and re-log in twn, frankly trying log in 25 times, 24 sessions are CSRF-blocked, and one "successed", however even this one success, I can only translate not more than 20 messages, and after that I got Invalid CSRF token warning when continue translating (sadly, this is why I'm generally retired on that site). I just wanna ask: where to fix this? (Do not answer me to ipconfig /flushdns, that's unhelpful for anything I met).

@Liuxinyu970226 Translatewiki.net no longer runs on HHVM, so if you are having sessions problems lately, it is something else (nobody has reported such problems, so might want to check your browser first).

In T178242#4654733, @Liuxinyu970226 wrote:

@Krinkle But the CSRF token errors really still shows to me, [..]

An error that fails CSRF when it should pass, means we are rejecting more attempts than we should. That's okay from a security perspective.

It works internally by comparing the token your browser has with the token we store internally. When there is a problem with the cache, the token we store internally is "forgotten", which means for a short time, several "valid" attempts are rejected because we cannot compare the tokens when our internal one is "forgotten".

If you see the opposite - you try something you are not allowed to do, and it succeeds, then it is a security problem.