| ori | |
| Jan 29 2016, 10:58 PM |
| F3359784: stacked.php.png | |
| Feb 15 2016, 2:08 AM |
| F3359124: graph.png | |
| Feb 14 2016, 8:24 PM |
| F3292921: graph.png | |
| Jan 30 2016, 6:51 AM |
| F3291688: usedmem.png | |
| Jan 29 2016, 10:58 PM |
| F3291664: memcaches.png | |
| Jan 29 2016, 10:58 PM |
| F3291699: evict.png | |
| Jan 29 2016, 10:58 PM |
| F3291671: abcd.png | |
| Jan 29 2016, 10:58 PM |
Bytes in on the application server cluster increased by about 40% over the past day:
This matches an increase in bytes out on the memcached / session redis hosts:
It seems that the issue is with redis usage rather than memcached; redis memory usage jumped from ~15mb to >500mb:
This is forcing redis to evict keys:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Deskana | T75616 Tracking: API/backend issues blocking Wikipedia app development | |||
| Resolved | Anomie | T32788 Allow triggering of user password reset email via the API | |||
| Open | None | T90925 General authentication improvements for MediaWiki | |||
| Resolved | Anomie | T48179 Allow a challenge stage during authentication | |||
| Open | None | T5709 Refactoring to make external authentication and identity systems easier | |||
| Resolved | Tgr | T43201 UserLoadFromSession considered evil | |||
| Resolved | Anomie | T67493 Session is started by EditAction (problem for extensions using UserLoadFromSession hook) | |||
| Open | Feature | None | T55156 Provide option to force a login session to end within a certain time | ||
| Open | None | T89459 Modernize MediaWiki authentication system (AuthManager) | |||
| Resolved | Anomie | T123451 Deploy SessionManager and bot passwords | |||
| Resolved | ori | T125267 ~3000% increase in session redis memory usage, causing evictions and session loss |
Change 267387 had a related patch set uploaded (by Anomie):
SessionManager: Don't save sessions until they're persisted
Change 267388 had a related patch set uploaded (by BryanDavis):
SessionManager: Don't save sessions until they're persisted
Change 267388 abandoned by BryanDavis:
SessionManager: Don't save sessions until they're persisted
Change 267388 restored by BryanDavis:
SessionManager: Don't save sessions until they're persisted
Change 267388 had a related patch set uploaded (by BryanDavis):
SessionManager: Don't save non-persisted sessions to backend storage
Change 267387 merged by jenkins-bot:
SessionManager: Don't save non-persisted sessions to backend storage
Change 267388 merged by jenkins-bot:
SessionManager: Don't save non-persisted sessions to backend storage
On investigation @Anomie determined that SessionManager was potentially fetching data from the backing cache multiple times in handling a single request. https://gerrit.wikimedia.org/r/267387 (& the https://gerrit.wikimedia.org/r/267388 cherry-pick) add an in-process cache to stop this. TCP dump traces taken by @ori appeared to show ~7 fetches of the same session for a single request before the patch was applied.
T125194: Possibly getting logged out within the same session after inactivity could be related if evictions are actually up.
What seems to have been happening here is two related problems:
#2 was solved by introducing an in-memory cache, so each session's data would be loaded only once per request. It's possible that data still might be saved multiple times per request, however. We can look into mitigating that if that too is a problem.
#1 was solved by only saving to redis when the session is persisted.[1] Non-persisted sessions are saved in-memory only, as any data they might contain is not needed outside the context of the individual request. If this breaks anything, the solution is for that thing to persist the session like it should have been doing in the first place. We note that the baseline memory usage is still increased by (estimated) about 350–400 bytes per session, due to the extra metadata being stored to support the improved security checks offered by SessionManager.
[1]: A "persisted" session is one that has had cookies sent to the user, or whatever is equivalent.
Baseline is certainly up a bit as expected by increased base session contents, but the runaway growth seems to have been taken care of.
It doesn't look so bad if you look at all the servers instead of just mc1001: https://ganglia.wikimedia.org/latest/stacked.php?m=used_memory&c=Memcached%20eqiad&r=week&st=1455218004&host_regex=
