Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	greg
	Dec 13 2017, 10:24 PM

Description

Page loads are inconsistently loaded or timing out.

apache-status output:

GGGGGGKGGGGGGGGGGGGGGGGGGGGGKGGGGGGGGGGGGGGGGGGGGGGGGKGGKGGGGGGG
KGGGGGGGGGGGKGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG_GGGGGGGGGGGGGGGG
KKGGGGGGGGGGGGGKGGGGGGGGGGGGGGG_GG_GGGGGGGGGGGKGGGGKGGGGGGGKGGGG
GGGGGKGGKKGKGGGGGGGGGGGGGG_GKGKKKGKGGGKKKGKKWGGG_GG__G__GGG__GGK
G__G_GGGGG_KG___...G.G...KGGGGGGG...GG.....G.GG._GG.G..G.G.G.G..
.....G......G.G.G..........G....K..G..........G.................
................................................................
..

  Scoreboard Key:
  "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
  "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
  "C" Closing connection, "L" Logging, "G" Gracefully finishing,
  "I" Idle cleanup of worker, "." Open slot with no current process

Outstanding issues:

Each httpd process stuck in G state shows the following stacktrace: https://phabricator.wikimedia.org/T182832#3940279. This seems to be due to how Phabricator handles very long json strings. Upgrade Phabricator to the next upstream release seems to be a possible solution due to some recent activity related to json string handling.

httpd regularly segfaults during php_request_shutdown with the following stacktrace: https://phabricator.wikimedia.org/T182832#3947103. This is possibly not related to the above issue, but to https://bugs.php.net/bug.php?id=74702. Upgrade phab1001 to PHP-7.1 seems to be a possible solution.

Details

Subject	Repo	Branch	Lines +/-
phabricator: enable php-fpm in Hiera on both hosts	operations/puppet	production	+3 -2
phabricator: Replace mod_php with php-fpm	operations/puppet	production	+1 K -15
phabricator: disable opcache.fastshutdown	operations/puppet	production	+1 -1
Phabricator: Support php 7.2 under stretch	operations/puppet	production	+77 -17
role::phabricator: add prometheus apache exporter	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		mmodell	T182832 Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state
Duplicate		None	T186620 Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array"
Resolved		jcrespo	T187143 Upcoming phabricator upgrade requires unusually long database migrations
Resolved		mmodell	T187790 Phabricator: Clean up deadlocked apache processes
Resolved		Paladox	T187127 Add support for stretch in the phabricator puppet class
Resolved		Dzahn	T190568 Reimage both phab1001 and phab2001 to stretch / buster
Resolved		mmodell	T137928 Deploy phabricator to phab2001.codfw.wmnet
Resolved		RobH	T137838 setup phab2001.codfw.wmnet (WMF6405)
Resolved		MoritzMuehlenhoff	T138689 upload php-mailparse and python-phabricator to jessie
Resolved		Dzahn	T143363 networking: allow ssh between iridium and phab2001
Invalid		None	T143370 build php5-mailparse for jessie?
Declined		Dzahn	T152132 Setup test domain for phab2001
Resolved		mmodell	T168699 Verify that the codfw lvs is configured correctly for Phabricator
Declined		None	T112776 Implement phabricator database clustering support
Resolved		mmodell	T128009 Next Phabricator Upgrade \| 2016-04-28
Resolved		mmodell	T190572 Prepare a disaster recovery plan for failing over Phabricator
Declined		mmodell	T232883 Make PHD run on the backup phabricator server (phab2001, currently)
Resolved	Request	Cmjohnson	T221391 decommission phab1002/WMF4727
Declined		Dzahn	T232887 The phabricator server, WMF7426, was given to us temporarily, we would like to make it permanent
Resolved		MoritzMuehlenhoff	T235140 package wikimedia-lvs-realserver for buster

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@elukey what about backporting php7.1 from buster into wikimedia-stretch component/php71?

I am running buster's php7.1 packages on phab-stretch.wmflabs.org and it works.

@Paladox: how many packages from buster were required to get it working?

@mmodell hi, i have a list here https://phabricator.wikimedia.org/P6688 (not fully complete)

Change 410245 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Phabricator: Support php 7.1 under stretch

https://gerrit.wikimedia.org/r/410245

Looks like we still have workers in the "G" state:

WWG__._G_GGWGWK___.GGW___W.__G.._G._GGGG.GGG_..GG__..G_G____K_._
G__K______G._W.G_W_G_.._..G._._.____......__.G._._..K._.........
......................._........................................
................................................................
................................................................
................................................................
................................................................

@mmodell full list

P6707 (An Untitled Masterwork)

1	root@phabricator-stretch1:/home/paladox# apt list --installed \| grep php
2
3	WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
4
5	libapache2-mod-php7.1/now 7.1.13-1 amd64 [installed,local]
6	php-apcu/now 5.1.9+4.0.11-1+b1 amd64 [installed,local]
7	php-cli/stable,now 1:7.0+49 all [installed]
8	php-common/stable,now 1:49 all [installed,automatic]
9	php-curl/stable,now 1:7.0+49 all [installed]
10	php-dev/stable,now 1:7.0+49 all [installed]
11	php-gd/stable,now 1:7.0+49 all [installed]
12	php-json/stable,now 1:7.0+49 all [installed]
13	php-ldap/stable,now 1:7.0+49 all [installed]
14	php-mailparse/now 3.0.2+2.1.6-12-gae1ef14-1+b1 amd64 [installed,local]
15	php-mysql/stable,now 1:7.0+49 all [installed]
16	php7.0-cli/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
17	php7.0-common/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
18	php7.0-curl/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
19	php7.0-dev/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
20	php7.0-gd/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
21	php7.0-json/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
22	php7.0-ldap/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
23	php7.0-mysql/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
24	php7.0-opcache/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
25	php7.0-phpdbg/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
26	php7.0-readline/stable,now 7.0.27-0+deb9u1 amd64 [installed,automatic]
27	php7.1/now 7.1.13-1 all [installed,local]
28	php7.1-cli/now 7.1.13-1 amd64 [installed,local]
29	php7.1-common/now 7.1.13-1 amd64 [installed,local]
30	php7.1-curl/now 7.1.13-1 amd64 [installed,local]
31	php7.1-dev/now 7.1.13-1 amd64 [installed,local]
32	php7.1-gd/now 7.1.13-1 amd64 [installed,local]
33	php7.1-json/now 7.1.13-1 amd64 [installed,local]
34	php7.1-mbstring/now 7.1.13-1 amd64 [installed,local]
35	php7.1-mysql/now 7.1.13-1 amd64 [installed,local]
36	php7.1-opcache/now 7.1.13-1 amd64 [installed,local]
37	php7.1-readline/now 7.1.13-1 amd64 [installed,local]

Got a backtrace of one process in G state, seems the same issue:

#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007fc2f16735fb in _L_lock_11305 () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fc2f1671758 in __GI___libc_realloc (oldmem=0x7fc2f199a620 <main_arena>, bytes=bytes@entry=64) at malloc.c:3025
#3  0x00007fc2e44578e0 in printbuf_extend (min_size=<optimized out>, p=0x5628ef5467a0) at printbuf.c:72
#4  printbuf_memappend (p=0x5628ef5467a0,
    buf=buf@entry=0x5628f0a6c6bf "/srv/phab/libext/translations/src\"], \"maniphest.custom-field-definitions\": {\"bug.version\": {\"description\": \"Affected software version\", \"fulltext\": true, \"name\": \"Version\", \"search\": true, \"type\": \"te"..., size=size@entry=33) at printbuf.c:82
#5  0x00007fc2e44561a6 in json_tokener_parse_ex (tok=0x5628ef548780,
    str=0x5628f0a6c6e0 "\"], \"maniphest.custom-field-definitions\": {\"bug.version\": {\"description\": \"Affected software version\", \"fulltext\": true, \"name\": \"Version\", \"search\": true, \"type\": \"text\"}, \"deadline.due\": {\"copy\": tr"..., len=8739) at json_tokener.c:434
#6  0x00007fc2e4661e21 in php_json_decode_ex () from /usr/lib/php5/20131226/json.so
#7  0x00007fc2e4661fa7 in ?? () from /usr/lib/php5/20131226/json.so
#8  0x00007fc2edf31dca in dtrace_execute_internal (execute_data_ptr=<optimized out>, fci=<optimized out>, return_value_used=<optimized out>)
    at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:97
#9  0x00007fc2edff27f0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc2f238fae0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:560
#10 0x00007fc2edf80b40 in execute_ex (execute_data=0x7fc2f238fae0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:363
#11 0x00007fc2edf31c68 in dtrace_execute_ex (execute_data=0x7fc2f238fae0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:73
#12 0x00007fc2edff2d33 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc2f238f988) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:592
#13 0x00007fc2edf80b40 in execute_ex (execute_data=0x7fc2f238f988) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:363
#14 0x00007fc2edf31c68 in dtrace_execute_ex (execute_data=0x7fc2f238f988) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:73
#15 0x00007fc2edff2d33 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc2f238f868) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:592
#16 0x00007fc2edf80b40 in execute_ex (execute_data=0x7fc2f238f868) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:363
#17 0x00007fc2edf31c68 in dtrace_execute_ex (execute_data=0x7fc2f238f868) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:73
#18 0x00007fc2edff2d33 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc2f238f688) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:592
#19 0x00007fc2edf80b40 in execute_ex (execute_data=0x7fc2f238f688) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:363
#20 0x00007fc2edf31c68 in dtrace_execute_ex (execute_data=0x7fc2f238f688) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:73
#21 0x00007fc2edff2d33 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc2f238f4e0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:592
#22 0x00007fc2edf80b40 in execute_ex (execute_data=0x7fc2f238f4e0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:363
#23 0x00007fc2edf31c68 in dtrace_execute_ex (execute_data=0x7fc2f238f4e0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:73
#24 0x00007fc2edff2d33 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc2f238f3e8) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:592
#25 0x00007fc2edf80b40 in execute_ex (execute_data=0x7fc2f238f3e8) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:363
#26 0x00007fc2edf31c68 in dtrace_execute_ex (execute_data=0x7fc2f238f3e8) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:73
#27 0x00007fc2edff2d33 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc2f238f1b8) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:592
#28 0x00007fc2edf80b40 in execute_ex (execute_data=0x7fc2f238f1b8) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:363
#29 0x00007fc2edf31c68 in dtrace_execute_ex (execute_data=0x7fc2f238f1b8) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:73
#30 0x00007fc2edff2d33 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc2f238f0a0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:592
#31 0x00007fc2edf80b40 in execute_ex (execute_data=0x7fc2f238f0a0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_vm_execute.h:363
#32 0x00007fc2edf31c68 in dtrace_execute_ex (execute_data=0x7fc2f238f0a0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend_dtrace.c:73
#33 0x00007fc2edf44850 in zend_execute_scripts (type=-241588704, type@entry=8, retval=0x80, retval@entry=0x0, file_count=87, file_count@entry=3)
    at /build/php5-m8fq6G/php5-5.6.33+dfsg/Zend/zend.c:1341
#34 0x00007fc2ededf940 in php_execute_script (primary_file=primary_file@entry=0x7ffed74915d0) at /build/php5-m8fq6G/php5-5.6.33+dfsg/main/main.c:2613
#35 0x00007fc2edff441a in php_handler (r=<optimized out>) at /build/php5-m8fq6G/php5-5.6.33+dfsg/sapi/apache2handler/sapi_apache2.c:667
#36 0x00005628ee1298a0 in ap_run_handler ()
#37 0x00005628ee129de9 in ap_invoke_handler ()
#38 0x00005628ee13fcc2 in ap_process_async_request ()

Paladox added a comment.Feb 15 2018, 5:56 PM

This comment was removed by Paladox.

In T182832#3976287, @mmodell wrote:

Looks like we still have workers in the "G" state:

WWG__._G_GGWGWK___.GGW___W.__G.._G._GGGG.GGG_..GG__..G_G____K_._
G__K______G._W.G_W_G_.._..G._._.____......__.G._._..K._.........
......................._........................................
................................................................
................................................................
................................................................
................................................................

@mmodell worth to open an upstream task about the issue?

Yeah it's especially odd that it's happening on the exact same json string every time. I'll see if upstream has heard of anything like it

Interesting: I think I may have found a relevant (and long standing) bug report for the deadlock: https://bugs.php.net/bug.php?id=31749, it's even got comments from our own @tstarling who points to https://www.mediawiki.org/wiki/Special:Code/MediaWiki/103433#140

@mmodell "This is finally fixed in PHP-7.1 with "safe timeout handling."

In T182832#3974829, @elukey wrote:

In T182832#3974547, @Dzahn wrote:

I fixed the server status page. it's available again now.

What was the issue? I can see a big hole in https://grafana.wikimedia.org/dashboard/db/phabricator?orgId=1&panelId=7&fullscreen&from=now-30d&to=now :(

https://gerrit.wikimedia.org/r/#/q/I7d8bc4c6ed810502 was merged which introduced the httpd module which copied most things from the apache module, including the server-status file but not the server-status config snippet

https://gerrit.wikimedia.org/r/#/q/I5a56c1ca3cc3e8 was merged which switched phabricator from apache module to the httpd module

The result was that the server-status.conf was in conf-available where it was expected but because the apache module had put it there and there was no puppet code to purge it / it wasn't done manually. The symlink from conf-enabled to conf-available though was removed by puppet and it reloaded Apache, breaking the status page.

https://gerrit.wikimedia.org/r/#/c/410630/ was merged to copy the missing config snippet from apache module to httpd module and that fixed it

I can see this in syslog:
Feb 13 06:28:25 phab1001 diamond[828]: Collector failed!

That is a remnant from when it still used the Apache module but the httpd module doesn't have it on purpose because we want to remove the diamond collector.

I saw that you added the prometheus collector in https://gerrit.wikimedia.org/r/#/c/410740/ which is the real fix , yep :)

@Muehlenhoff hi, what about backporting php71 from either buster or https://packages.sury.org/php/pool/main/p/php7.1/ (buster php 7.1 works on stretch when i installed the debs manually).

https://packages.sury.org/php/pool/main/p/php-mailparse/

https://packages.sury.org/php/pool/main/p/php-apcu/

If it's really as simple as importing the packages from buster and deploying them, then that seems like a fairly ideal solution to me. We would need to do some testing beyond simply installing them on a vps instance and declaring it works!!, however, I don't think there will be any huge issues with compatibility, phabricator has supported 7.1 for a full year now.

Mentioned in SAL (#wikimedia-operations) [2018-02-17T17:33:38Z] <twentyafterfour> restarting apache on phab1001 to clear deadlocked workers. refs T182832

@elukey, @Dzahn: do you think that setting MaxConnectionsPerChild would be worth a try? I somehow doubt it will help but it seems like it might improve the situation.

mmodell merged a task: T186620: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array".Feb 17 2018, 5:37 PM

In T182832#3981076, @mmodell wrote:

@elukey, @Dzahn: do you think that setting MaxConnectionsPerChild would be worth a try? I somehow doubt it will help but it seems like it might improve the situation.

The main issue is that when the deadlock occurs and the httpd process is stuck in G state it will never reach MaxConnectionsPerChild, so no hope (from my point of view) to get some relief in this case. It seem that the (only) permanent solution is to migrate to PHP 7.1..

I posted this upstream: https://discourse.phabricator-community.org/t/phabricator-processes-occasionally-segfaulting-deadlocking-since-september-2017/1142/2 - so far this hasn't generated any clues.

mmodell added a project: Wikimedia-Incident.Feb 20 2018, 2:34 PM

mmodell mentioned this in T187790: Phabricator: Clean up deadlocked apache processes.Feb 20 2018, 3:10 PM

mmodell moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Feb 20 2018, 3:31 PM

mmodell moved this task from Follow-up prevention to Active Situation on the Wikimedia-Incident board.

mmodell closed subtask T187790: Phabricator: Clean up deadlocked apache processes as Resolved.Feb 21 2018, 10:38 PM

In T182832#3982284, @elukey wrote:

The main issue is that when the deadlock occurs and the httpd process is stuck in G state it will never reach MaxConnectionsPerChild, so no hope (from my point of view) to get some relief in this case. It seem that the (only) permanent solution is to migrate to PHP 7.1..

I'll look into creating a component/php71 this week.

Thanks @MoritzMuehlenhoff! Please let me know if there is anything I can do to help.

elukey moved this task from In Progress to Keep an eye on it on the User-Elukey board.Feb 27 2018, 12:56 PM

mmodell moved this task from Backlog to Blocked (externally) on the Release-Engineering-Team (Kanban) board.Feb 27 2018, 3:27 PM

mmodell moved this task from Active Situation to Follow-up prevention on the Wikimedia-Incident board.

mmodell added a subtask: T187127: Add support for stretch in the phabricator puppet class.Mar 13 2018, 1:25 AM

Dzahn closed subtask T187127: Add support for stretch in the phabricator puppet class as Resolved.Mar 19 2018, 10:13 PM

Dzahn reopened subtask T187127: Add support for stretch in the phabricator puppet class as Open.Mar 19 2018, 10:18 PM

Change 410245 merged by Dzahn:
[operations/puppet@production] Phabricator: Support php 7.2 under stretch

https://gerrit.wikimedia.org/r/410245

Paladox closed subtask T187127: Add support for stretch in the phabricator puppet class as Resolved.Mar 23 2018, 7:53 PM

Paladox moved this task from To Triage to Misc on the Phabricator board.

Paladox mentioned this in T190568: Reimage both phab1001 and phab2001 to stretch / buster.Mar 23 2018, 7:58 PM

Paladox added a subtask: T190568: Reimage both phab1001 and phab2001 to stretch / buster.

Change 410767 abandoned by Elukey:
phabricator: disable opcache.fastshutdown

Reason:
Old code review

https://gerrit.wikimedia.org/r/410767

Paladox mentioned this in T191361: Phabricator is loading really slowly.Apr 3 2018, 10:06 PM

[phab1001:~] $ sudo apache-status
               Apache Server Status for 127.0.0.1 (via 127.0.0.1)

   Server Version: Apache/2.4.10 (Debian) PHP/5.6.33-0+deb8u1

   Server MPM: prefork

   Server Built: Sep 20 2017 12:05:38

     ----------------------------------------------------------------------

   Current Time: Tuesday, 03-Apr-2018 22:19:21 UTC

   Restart Time: Sunday, 01-Apr-2018 23:15:12 UTC

   Parent Server Config. Generation: 3

   Parent Server MPM Generation: 2

   Server uptime: 1 day 23 hours 4 minutes 8 seconds

   Server load: 11.64 12.49 11.45

   Total accesses: 1143441 - Total Traffic: 194.7 GB

   CPU Usage: u5746.4 s450.4 cu740.81 cs156.01 - 4.19% CPU load

   6.75 requests/sec - 1.2 MB/second - 178.5 kB/request

   96 requests currently being processed, 50 idle workers

 GWWGGGGGG.WWWWWGW.WGGWW._GGGG_.GGGGW_GWGGG.W_.WWGGGGGW_.__K_W.GG
 .GG__GGG_GGW_G..__G.GG__G_.G__.K.WGWG_._KK_.W_..W.GG__WK_.GGG.GG
 _W....._..W._._GW._W_._WK__.W_.._W___W__._K_...W...._._WK...._..
 __.......................G....._..................W.............
 ................................................................
 ................................................................
 ................................................................
 ..

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process

Mentioned in SAL (#wikimedia-operations) [2018-04-03T22:25:02Z] <mutante> restarting Apache on phab1001 - T182832

This happened again even with the cron that restarts it every sunday. Im not sure if we have time for codfw to get the databases setup by the end of 2018.

with the restart above, it fixed it, but it will only fill up like before.

Platonides added a subscriber: Platonides.Apr 3 2018, 10:33 PM

greg merged a task: T191361: Phabricator is loading really slowly.Apr 4 2018, 1:03 AM

The plan is to reimage phab1001 to Stretch in T190568, unblocking the PHP 7.2 deployment that should fix both issues and get us to a normal/more-stable state.

Paladox mentioned this in T196565: Phabricator is very slow to load.Jun 6 2018, 3:53 PM

So apparently this is resolved... I'm not sure what changed on June 10th but check out this graph:
https://grafana.wikimedia.org/dashboard/db/phabricator?orgId=1&from=now-30d&to=now

apache-idle-workers.png (292×514 px, 36 KB)

Could it be traffic? But also it could happen again.

And now it's back again on July 1st:

Screenshot from 2018-07-18 11-49-22.png (235×758 px, 37 KB)

This looks really strange.

Screenshot from 2018-07-18 11-50-47.png (251×949 px, 42 KB)

I guess that may have been the period where the rate limiter was enabled?

In T182832#4434775, @Paladox wrote:

I guess that may have been the period where the rate limiter was enabled?

Not quite, the timing is close but not exact.

Aklapper moved this task from Misc to Infrastructure on the Phabricator board.Aug 29 2018, 6:02 PM

mmodell added a project: User-MModell.Sep 10 2018, 4:51 PM

mmodell moved this task from Backlog to Blocked or Stalled on the User-MModell board.Sep 10 2018, 4:54 PM

Unassigning but I'm still tracking this on my personal workboard (User-MModell)

Paladox mentioned this in T204421: Phabricator is slow.Sep 15 2018, 9:51 PM

this bug has hit us again T204421

greg merged a task: T204421: Phabricator is slow.Sep 17 2018, 3:16 PM

greg added subscribers: akosiaris, Kizule, MarcoAurelio, Legoktm.

Should we raise the frequency of the restarts?

Change 407958 abandoned by Paladox:
phabricator: Replace mod_php with php-fpm

https://gerrit.wikimedia.org/r/407958

Change 407958 restored by Paladox:
phabricator: Replace mod_php with php-fpm

Reason:
Re opening per _joe_ comment at https://phabricator.wikimedia.org/T208108#4701107

https://gerrit.wikimedia.org/r/407958

Change 407958 abandoned by Paladox:
phabricator: Replace mod_php with php-fpm

Reason:
I've did it in https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /476985/

https://gerrit.wikimedia.org/r/407958

While merging and confirming https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/476985/ which adds php-fpm support on stretch (so should not affect jessie prod yet), i noticed that /etc/apache2/mods-available/mpm_prefork.conf was changed from package defaults to custom tuned values we had set in the past. The latter changes were written in ./conf-available' as opposed to 'mods-available'. (as apache::conf class used to do it). There were 2 competing configs. Since the main config loads "mods-available" after "conf-available" the one in mods-available should have been the relevant one. That means our custom tuning config wasn't actually applied for a while. Now they are again.

19:13 < mutante> !log re-enabling puppet on phabricator, applying change that adds php-fpm support on stretch ..which doesnt affect phab1001 (prod) on jessie.. BUT re-adds tuning config from the past for mpm_prefork.conf (more SpareServers etc) that was not actually applied due to a bug

19:15 < mutante> !log MPM prefork tweaks for high load systems are applied again (apparently they were not since a change in the past that resulted in 2 competing configs in mods-enabled and conf-enabled with the latter one being loaded last and containing the package defaults

We think this happened somewhere around:

https://github.com/wikimedia/puppet/commit/5256156a2832d5bb07884c4f847f41e5842a0a0c#diff-544574f97d57acf91c3517080f5e1b49
https://github.com/wikimedia/puppet/commit/9d22da9cfc0aef4ba6ee9b6e619878e735aec2df#diff-544574f97d57acf91c3517080f5e1b49R8

Dzahn added a project: serviceops.Dec 17 2018, 4:35 PM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:38 PM

Dzahn mentioned this in T215335: requesting WMF7426 as phabricator system in eqiad.Feb 8 2019, 2:14 AM

elukey removed a project: User-Elukey.Apr 16 2019, 11:00 AM

Dzahn changed the status of subtask T190568: Reimage both phab1001 and phab2001 to stretch / buster from Open to Stalled.Apr 17 2019, 10:02 PM

(hopefully) just blocked by T190568 which is blocked on T215335.

ArielGlenn removed a project: Patch-For-Review.May 15 2019, 11:56 AM

Change 510597 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: enable php-fpm

https://gerrit.wikimedia.org/r/510597

gerritbot added a project: Patch-For-Review.May 15 2019, 7:10 PM

Dzahn changed the status of subtask T190568: Reimage both phab1001 and phab2001 to stretch / buster from Stalled to Open.May 23 2019, 4:50 AM

It looks like this one is resolved[1], either by the php 7.2 upgrade or use of php-fpm sapi.

(ambiguitytape)

[1] See flat line on the right side of the graph:

Screenshot from 2019-05-23 11-05-11.png (407×951 px, 43 KB)

https://grafana.wikimedia.org/d/000000587/phabricator?orgId=1&panelId=7&fullscreen&from=now-7d&to=now

mmodell awarded a token.May 23 2019, 4:07 PM

Dzahn awarded a token.May 23 2019, 4:13 PM

Updated image for phab1003:

Screenshot 2019-05-23 at 22.40.57.png (1×2 px, 1 MB)

It looks much much more healthier than phab1001.

Aklapper awarded a token.May 23 2019, 9:58 PM

Change 510597 merged by Dzahn:
[operations/puppet@production] phabricator: enable php-fpm in Hiera on both hosts

https://gerrit.wikimedia.org/r/510597

now also phab2001 has been switched to php-fpm and worker . it matches phab1003 now. ^

Dzahn changed the status of subtask T190568: Reimage both phab1001 and phab2001 to stretch / buster from Open to Stalled.Sep 13 2019, 6:54 PM

Dzahn changed the status of subtask T190568: Reimage both phab1001 and phab2001 to stretch / buster from Stalled to Open.Sep 13 2019, 10:47 PM

Dzahn closed subtask T190568: Reimage both phab1001 and phab2001 to stretch / buster as Resolved.Nov 22 2019, 10:11 PM

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM

	F29226685: Screenshot 2019-05-23 at 22.40.57.png
	May 23 2019, 9:41 PM

	F29223857: Screenshot from 2019-05-23 11-05-11.png
	May 23 2019, 4:07 PM

	F23884895: Screenshot from 2018-07-18 11-50-47.png
	Jul 18 2018, 4:51 PM

	F23884869: Screenshot from 2018-07-18 11-49-22.png
	Jul 18 2018, 4:49 PM

	F13019944: Screen Shot 2018-02-05 at 9.15.17 AM.png
	Feb 5 2018, 8:27 AM

	F11999928: phabricator-apache-workers.png
	Dec 22 2017, 11:31 PM

Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" stateClosed, ResolvedPublicActions

Description

apache-status output:

Details

Related ObjectsSearch...

Event Timeline

Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state
Closed, ResolvedPublic
Actions

Related Objects
Search...