Page MenuHomePhabricator

Improve memcache logs on mc* hosts
Closed, DeclinedPublic

Description

Memcache is currently not logging anything to /var/log/memcached.log, since on Jessie the systemd unit is not configured to read any of /etc/default/memcached and /etc/memcached.conf.

A possible solution is to force memcache to use the -v parameter, and redirect stdout to a specific file via rsyslog rule. The man says:

-v     Be verbose during the event loop; print out errors and warnings.

-vv    Be even more verbose; same as -v but also print client commands and responses.

-vvv   Be extremely verbose; same of the above and also print internal state transitions.

From https://github.com/memcached/memcached/wiki/ConfiguringServer#commandline-arguments:

-v controls verbosity to STDOUT/STDERR. Multiple -v's increase verbosity. A single one prints extra startup information, and multiple will print increasingly verbose information about requests hitting memcached. If you're curious to see if a test script is doing what you expect it to, running memcached in the foreground with a few verbose switches is a good idea.

From a quick test in deployment-prep -v seems very low volume, and it shouldn't degrade performances, meanwhile -vv logs every request landing to memcache (definitely too high).

Event Timeline

elukey triaged this task as Medium priority.Sep 4 2018, 6:57 AM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 4 2018, 6:57 AM

Change 456096 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] memcached: enable basic logging with the -v parameter

https://gerrit.wikimedia.org/r/456096

Mentioned in SAL (#wikimedia-operations) [2018-09-04T13:06:05Z] <elukey> restart memcached on mc1035 with -v option - T203429

Retagging as MediaWiki-Cache is about the PHP code inside MediaWiki core about caching (the the Memcached client, and the BagOStuff and WANObjectCache libraries). It does not relate to Wikimedia's deployment of that library, nor the actual Memcached service.

To my knowledge, there is no component fitting either of those. So tagging general for now.

Change 456096 abandoned by Elukey:
memcached: enable basic logging with the -v parameter

https://gerrit.wikimedia.org/r/456096

Mentioned in SAL (#wikimedia-operations) [2018-09-05T07:48:49Z] <elukey> re-enable puppet on mc1035 - memcache unit refreshed, mw cache shard wiped - T203429

elukey closed this task as Declined.Sep 5 2018, 7:50 AM

The amount of logs found during the night were really low:

Sep 04 13:06:21 mc1035 systemd[1]: Started memcached daemon.
Sep 04 13:09:20 mc1035 memcached[12681]: Failed to write, and not due to blocking: Broken pipe

Given the fact that adding the -v option would mean restart all the memcache instance that we run in production, I don't see the benefit of having very few logs compared to the huge amount of work required to make it happen, so for the moment I'd decline this task.