Page MenuHomePhabricator

Implement phabricator database clustering support
Closed, DeclinedPublic

Description

Phabricator just got improved clustering support. See upstream changelog/2016.16 for details.

Key details from the phabricator database cluster documentation:

If you lose the master, Phabricator can degrade automatically into read-only mode and remain available, but can not fully recover without operational intervention unless the master recovers on its own.
Phabricator will not currently send read traffic to replicas unless the master has failed, so configuring a replica will not currently spread any load away from the master. Future versions of Phabricator are expected to be able to distribute some read traffic to replicas.
Phabricator can not currently be configured into a multi-master mode, nor can it be configured to automatically promote a replica to become the new master. There are no current plans to support multi-master mode or autonomous failover, although this may change in the future.


Old Description:

Following up from T109279, we need to keep an eye on phabricator performance and scalability concerns.

Some changes may be necessary in the coming months to ensure that we are adequately prepared to keep phabricator performance at an acceptable level as we continue to rely on phabricator for more critical functions.

There are a few areas where phabricator can be configured for better scalability:

  1. File storage.
    • Currently we have phabricator configured to use the mysql storage engine for file attachments and temporary storage. This stores files directly in a blob column within phabricator's mysql cluster. I'm not aware of any serious performance issues with this method, however, it does add a lot of extra load to the mysql cluster and requires extra database connections from the front-end web servers to the mysql servers.
    • An alternative option would be to move phabricator storage to openstack swift. This can probably be accomplished fairly easily by customizing the amazon s3 engine to work with openstack. This should be trivial, as far as I can tell. The major concern here is that we do not currently use swift for storing sensitive information. Unfortunately, some sensitive information is uploaded to phabricator and whatever storage system we use needs to protect these sensitive files from accidental disclosure.
  2. Repository hosting
    • As we are migrating repositories from gerrit to phabricator, a significant amount of load will be placed on phabricator by requests to git service. Both CPU and Disk I/O could become bottlenecks which could potentially slow developer activity and limit CI testing throughput. The upstream phabricator project seems to have a plan for addressing scalability of SCM hosting, which primarily revolves around implementing a smart protocol proxy in phabricator which can delegate the work to multiple back-end git servers. With repositories distributed across several git servers, performance and scaling should not be a problem. This is, however, currently unimplemented and it could be a while before the upstream gets this working.

Related Objects

Event Timeline

mmodell created this task.Sep 16 2015, 5:14 PM
mmodell claimed this task.
mmodell raised the priority of this task from to Needs Triage.
mmodell updated the task description. (Show Details)
mmodell added a project: Phabricator.
mmodell added subscribers: mmodell, jcrespo, chasemp and 2 others.
Restricted Application added subscribers: scfc, Aklapper. · View Herald TranscriptSep 16 2015, 5:14 PM
mmodell triaged this task as Normal priority.Sep 16 2015, 5:14 PM
jcrespo raised the priority of this task from Normal to High.Dec 16 2015, 12:50 PM

1MariaDB MISC m3 localhost (none) > SHOW PROCESSLIST;
2+------------+-----------------+--------------------+--------------------------+-------------+----------+--------------------------------------------
3| Id | User | Host | db | Command | Time | State
4+------------+-----------------+--------------------+--------------------------+-------------+----------+--------------------------------------------
5[system users, omited for security]
6| 5209491971 | phuser | 10.64.32.150:27963 | phabricator_daemon | Sleep | 4 |
7| 5209492163 | phuser | 10.64.32.150:28188 | phabricator_config | Sleep | 5 |
8| 5209492164 | phuser | 10.64.32.150:28189 | phabricator_config | Sleep | 5 |
9| 5209492165 | phuser | 10.64.32.150:28190 | phabricator_config | Sleep | 5 |
10| 5209492168 | phuser | 10.64.32.150:28193 | phabricator_config | Sleep | 5 |
11| 5209492169 | phuser | 10.64.32.150:28194 | phabricator_config | Sleep | 5 |
12| 5209492171 | phuser | 10.64.32.150:28196 | phabricator_config | Sleep | 5 |
13| 5209492172 | phuser | 10.64.32.150:28197 | phabricator_config | Sleep | 5 |
14| 5209492174 | phuser | 10.64.32.150:28199 | phabricator_config | Sleep | 5 |
15| 5209492175 | phuser | 10.64.32.150:28200 | phabricator_spaces | Sleep | 5 |
16| 5209492177 | phuser | 10.64.32.150:28202 | phabricator_spaces | Sleep | 5 |
17| 5209492181 | phuser | 10.64.32.150:28206 | phabricator_project | Sleep | 1 |
18| 5209492182 | phuser | 10.64.32.150:28207 | phabricator_spaces | Sleep | 5 |
19| 5209492183 | phuser | 10.64.32.150:28208 | phabricator_project | Sleep | 0 |
20| 5209492184 | phuser | 10.64.32.150:28209 | phabricator_config | Sleep | 5 |
21| 5209492185 | phuser | 10.64.32.150:28210 | phabricator_spaces | Sleep | 5 |
22| 5209492186 | phuser | 10.64.32.150:28211 | phabricator_config | Sleep | 5 |
23| 5209492187 | phuser | 10.64.32.150:28213 | phabricator_policy | Sleep | 1 |
24| 5209492191 | phuser | 10.64.32.150:28217 | phabricator_spaces | Sleep | 5 |
25| 5209492192 | phuser | 10.64.32.150:28218 | phabricator_project | Sleep | 1 |
26| 5209492193 | phuser | 10.64.32.150:28219 | phabricator_config | Sleep | 5 |
27| 5209492195 | phuser | 10.64.32.150:28221 | phabricator_project | Sleep | 1 |
28| 5209492196 | phuser | 10.64.32.150:28222 | phabricator_config | Sleep | 5 |
29| 5209492197 | phuser | 10.64.32.150:28223 | phabricator_config | Sleep | 5 |
30| 5209492198 | phuser | 10.64.32.150:28224 | phabricator_spaces | Sleep | 5 |
31| 5209492199 | phuser | 10.64.32.150:28225 | phabricator_project | Sleep | 0 |
32| 5209492200 | phuser | 10.64.32.150:28226 | phabricator_repository | Sleep | 2 |
33| 5209492203 | phuser | 10.64.32.150:28229 | phabricator_policy | Sleep | 1 |
34| 5209492204 | phuser | 10.64.32.150:28230 | phabricator_config | Sleep | 5 |
35| 5209492206 | phuser | 10.64.32.150:28232 | phabricator_config | Sleep | 5 |
36| 5209492208 | phuser | 10.64.32.150:28234 | phabricator_spaces | Sleep | 5 |
37| 5209492209 | phuser | 10.64.32.150:28235 | phabricator_policy | Sleep | 5 |
38| 5209492210 | phuser | 10.64.32.150:28236 | phabricator_policy | Sleep | 0 |
39| 5209492211 | phuser | 10.64.32.150:28237 | phabricator_policy | Sleep | 5 |
40| 5209492213 | phuser | 10.64.32.150:28239 | phabricator_project | Sleep | 5 |
41| 5209492214 | phuser | 10.64.32.150:28240 | phabricator_config | Sleep | 5 |
42| 5209492216 | phuser | 10.64.32.150:28242 | phabricator_spaces | Sleep | 5 |
43| 5209492217 | phuser | 10.64.32.150:28243 | phabricator_spaces | Sleep | 5 |
44| 5209492218 | phuser | 10.64.32.150:28244 | phabricator_repository | Sleep | 2 |
45| 5209492219 | phuser | 10.64.32.150:28245 | phabricator_repository | Sleep | 2 |
46| 5209492220 | phuser | 10.64.32.150:28246 | phabricator_config | Sleep | 5 |
47| 5209492221 | phuser | 10.64.32.150:28247 | phabricator_spaces | Sleep | 5 |
48| 5209492222 | phuser | 10.64.32.150:28248 | phabricator_project | Sleep | 1 |
49| 5209492223 | phuser | 10.64.32.150:28249 | phabricator_spaces | Sleep | 5 |
50| 5209492224 | phuser | 10.64.32.150:28250 | phabricator_repository | Sleep | 2 |
51| 5209492225 | phuser | 10.64.32.150:28251 | phabricator_spaces | Sleep | 5 |
52| 5209492227 | phuser | 10.64.32.150:28253 | phabricator_policy | Sleep | 5 |
53| 5209492231 | phuser | 10.64.32.150:28257 | phabricator_repository | Sleep | 1 |
54| 5209492232 | phuser | 10.64.32.150:28258 | phabricator_project | Sleep | 0 |
55| 5209492233 | phuser | 10.64.32.150:28259 | phabricator_spaces | Sleep | 5 |
56| 5209492234 | phuser | 10.64.32.150:28260 | phabricator_project | Sleep | 5 |
57| 5209492235 | phuser | 10.64.32.150:28261 | phabricator_spaces | Sleep | 5 |
58| 5209492236 | phuser | 10.64.32.150:28262 | phabricator_project | Sleep | 0 |
59| 5209492238 | phuser | 10.64.32.150:28264 | phabricator_repository | Sleep | 0 |
60| 5209492239 | phuser | 10.64.32.150:28265 | phabricator_project | Sleep | 5 |
61| 5209492240 | phuser | 10.64.32.150:28266 | phabricator_spaces | Sleep | 5 |
62| 5209492244 | phuser | 10.64.32.150:28270 | phabricator_project | Sleep | 0 |
63| 5209492245 | phuser | 10.64.32.150:28271 | phabricator_policy | Sleep | 5 |
64| 5209492247 | phuser | 10.64.32.150:28273 | phabricator_policy | Sleep | 0 |
65| 5209492248 | phuser | 10.64.32.150:28274 | phabricator_config | Sleep | 5 |
66| 5209492249 | phuser | 10.64.32.150:28275 | phabricator_policy | Sleep | 0 |
67| 5209492250 | phuser | 10.64.32.150:28276 | phabricator_policy | Sleep | 5 |
68| 5209492251 | phuser | 10.64.32.150:28277 | phabricator_policy | Sleep | 5 |
69| 5209492252 | phuser | 10.64.32.150:28278 | phabricator_project | Sleep | 0 |
70| 5209492253 | phuser | 10.64.32.150:28279 | phabricator_spaces | Sleep | 5 |
71| 5209492255 | phuser | 10.64.32.150:28281 | phabricator_config | Sleep | 5 |
72| 5209492257 | phuser | 10.64.32.150:28283 | phabricator_repository | Sleep | 1 |
73| 5209492258 | phuser | 10.64.32.150:28284 | phabricator_spaces | Sleep | 5 |
74| 5209492259 | phuser | 10.64.32.150:28285 | phabricator_repository | Sleep | 5 |
75| 5209492260 | phuser | 10.64.32.150:28286 | phabricator_project | Sleep | 0 |
76| 5209492261 | phuser | 10.64.32.150:28287 | phabricator_repository | Sleep | 2 |
77| 5209492262 | phuser | 10.64.32.150:28288 | phabricator_policy | Sleep | 0 |
78| 5209492263 | phuser | 10.64.32.150:28289 | phabricator_policy | Sleep | 0 |
79| 5209492264 | phuser | 10.64.32.150:28290 | phabricator_spaces | Sleep | 5 |
80| 5209492265 | phuser | 10.64.32.150:28291 | phabricator_policy | Sleep | 0 |
81| 5209492266 | phuser | 10.64.32.150:28292 | phabricator_project | Sleep | 0 |
82| 5209492267 | phuser | 10.64.32.150:28293 | phabricator_project | Sleep | 1 |
83| 5209492268 | phuser | 10.64.32.150:28294 | phabricator_repository | Sleep | 5 |
84| 5209492269 | phuser | 10.64.32.150:28295 | phabricator_project | Sleep | 5 |
85| 5209492271 | phuser | 10.64.32.150:28297 | phabricator_repository | Sleep | 1 |
86| 5209492272 | phuser | 10.64.32.150:28298 | phabricator_repository | Sleep | 2 |
87| 5209492274 | phuser | 10.64.32.150:28300 | phabricator_repository | Sleep | 2 |
88| 5209492275 | phuser | 10.64.32.150:28301 | phabricator_policy | Sleep | 5 |
89| 5209492276 | phuser | 10.64.32.150:28302 | phabricator_policy | Sleep | 5 |
90| 5209492277 | phuser | 10.64.32.150:28303 | phabricator_policy | Sleep | 5 |
91| 5209492279 | phuser | 10.64.32.150:28305 | phabricator_config | Sleep | 5 |
92| 5209492280 | phuser | 10.64.32.150:28306 | phabricator_repository | Sleep | 1 |
93| 5209492282 | phuser | 10.64.32.150:28308 | phabricator_spaces | Sleep | 5 |
94| 5209492283 | phuser | 10.64.32.150:28309 | phabricator_project | Sleep | 5 |
95| 5209492287 | phuser | 10.64.32.150:28313 | phabricator_project | Sleep | 5 |
96| 5209492288 | phuser | 10.64.32.150:28314 | phabricator_repository | Sleep | 5 |
97| 5209492289 | phuser | 10.64.32.150:28315 | phabricator_repository | Sleep | 2 |
98| 5209492293 | phuser | 10.64.32.150:28319 | phabricator_policy | Sleep | 5 |
99| 5209492294 | phuser | 10.64.32.150:28320 | phabricator_policy | Sleep | 5 |
100| 5209492295 | phuser | 10.64.32.150:28321 | phabricator_config | Sleep | 5 |
101| 5209492296 | phuser | 10.64.32.150:28322 | phabricator_repository | Sleep | 1 |
102| 5209492299 | phuser | 10.64.32.150:28324 | phabricator_repository | Sleep | 5 |
103| 5209492301 | phuser | 10.64.32.150:28327 | phabricator_repository | Sleep | 0 |
104| 5209492302 | phuser | 10.64.32.150:28328 | phabricator_spaces | Sleep | 5 |
105| 5209492303 | phuser | 10.64.32.150:28329 | phabricator_config | Sleep | 5 |
106| 5209492304 | phuser | 10.64.32.150:28330 | phabricator_config | Sleep | 5 |
107| 5209492308 | phuser | 10.64.32.150:28334 | phabricator_project | Sleep | 0 |
108| 5209492310 | phuser | 10.64.32.150:28336 | phabricator_spaces | Sleep | 5 |
109| 5209492311 | phuser | 10.64.32.150:28337 | phabricator_policy | Sleep | 0 |
110| 5209492312 | phuser | 10.64.32.150:28338 | phabricator_project | Sleep | 5 |
111| 5209492314 | phuser | 10.64.32.150:28340 | phabricator_config | Sleep | 5 |
112| 5209492316 | phuser | 10.64.32.150:28342 | phabricator_config | Sleep | 5 |
113| 5209492319 | phuser | 10.64.32.150:28345 | phabricator_policy | Sleep | 5 |
114| 5209492320 | phuser | 10.64.32.150:28346 | phabricator_spaces | Sleep | 5 |
115| 5209492322 | phuser | 10.64.32.150:28348 | phabricator_repository | Sleep | 5 |
116| 5209492323 | phuser | 10.64.32.150:28349 | phabricator_spaces | Sleep | 5 |
117| 5209492325 | phuser | 10.64.32.150:28351 | phabricator_config | Sleep | 5 |
118| 5209492327 | phuser | 10.64.32.150:28353 | phabricator_project | Sleep | 5 |
119| 5209492328 | phuser | 10.64.32.150:28354 | phabricator_repository | Sleep | 1 |
120| 5209492329 | phuser | 10.64.32.150:28355 | phabricator_project | Sleep | 5 |
121| 5209492330 | phuser | 10.64.32.150:28356 | phabricator_policy | Sleep | 5 |
122| 5209492333 | phuser | 10.64.32.150:28359 | phabricator_spaces | Sleep | 5 |
123| 5209492334 | phuser | 10.64.32.150:28360 | phabricator_policy | Sleep | 5 |
124| 5209492335 | phuser | 10.64.32.150:28361 | phabricator_spaces | Sleep | 5 |
125| 5209492337 | phuser | 10.64.32.150:28363 | phabricator_project | Sleep | 0 |
126| 5209492339 | phuser | 10.64.32.150:28365 | phabricator_project | Sleep | 5 |
127| 5209492341 | phuser | 10.64.32.150:28367 | phabricator_repository | Sleep | 0 |
128| 5209492342 | phuser | 10.64.32.150:28368 | phabricator_repository | Sleep | 5 |
129| 5209492343 | phuser | 10.64.32.150:28369 | phabricator_policy | Sleep | 5 |
130| 5209492344 | phuser | 10.64.32.150:28370 | phabricator_spaces | Sleep | 5 |
131| 5209492345 | phuser | 10.64.32.150:28371 | phabricator_policy | Sleep | 5 |
132| 5209492346 | phuser | 10.64.32.150:28372 | phabricator_repository | Sleep | 1 |
133| 5209492347 | phuser | 10.64.32.150:28373 | phabricator_project | Sleep | 5 |
134| 5209492348 | phuser | 10.64.32.150:28374 | phabricator_repository | Sleep | 5 |
135| 5209492349 | phuser | 10.64.32.150:28375 | phabricator_policy | Sleep | 5 |
136| 5209492350 | phuser | 10.64.32.150:28376 | phabricator_repository | Sleep | 5 |
137| 5209492351 | phuser | 10.64.32.150:28377 | phabricator_config | Sleep | 5 |
138| 5209492353 | phuser | 10.64.32.150:28379 | phabricator_config | Sleep | 5 |
139| 5209492354 | phuser | 10.64.32.150:28380 | phabricator_spaces | Sleep | 5 |
140| 5209492355 | phuser | 10.64.32.150:28381 | phabricator_config | Sleep | 5 |
141| 5209492356 | phuser | 10.64.32.150:28382 | phabricator_project | Sleep | 5 |
142| 5209492357 | phuser | 10.64.32.150:28383 | phabricator_config | Sleep | 5 |
143| 5209492358 | phuser | 10.64.32.150:28384 | phabricator_config | Sleep | 5 |
144| 5209492359 | phuser | 10.64.32.150:28386 | phabricator_policy | Sleep | 5 |
145| 5209492360 | phuser | 10.64.32.150:28385 | phabricator_config | Sleep | 5 |
146| 5209492361 | phuser | 10.64.32.150:28387 | phabricator_config | Sleep | 5 |
147| 5209492363 | phuser | 10.64.32.150:28389 | phabricator_config | Sleep | 5 |
148| 5209492364 | phuser | 10.64.32.150:28390 | phabricator_config | Sleep | 5 |
149| 5209492365 | phuser | 10.64.32.150:28391 | phabricator_repository | Sleep | 5 |
150| 5209492367 | phuser | 10.64.32.150:28393 | phabricator_spaces | Sleep | 5 |
151| 5209492369 | phuser | 10.64.32.150:28395 | phabricator_config | Sleep | 5 |
152| 5209492370 | phuser | 10.64.32.150:28396 | phabricator_project | Sleep | 5 |
153| 5209492371 | phuser | 10.64.32.150:28397 | phabricator_spaces | Sleep | 5 |
154| 5209492372 | phuser | 10.64.32.150:28398 | phabricator_spaces | Sleep | 5 |
155| 5209492373 | phuser | 10.64.32.150:28399 | phabricator_spaces | Sleep | 5 |
156| 5209492374 | phuser | 10.64.32.150:28400 | phabricator_policy | Sleep | 5 |
157| 5209492375 | phuser | 10.64.32.150:28401 | phabricator_project | Sleep | 0 |
158| 5209492376 | phuser | 10.64.32.150:28402 | phabricator_project | Sleep | 5 |
159| 5209492377 | phuser | 10.64.32.150:28403 | phabricator_repository | Sleep | 5 |
160| 5209492378 | phuser | 10.64.32.150:28404 | phabricator_policy | Sleep | 5 |
161| 5209492379 | phuser | 10.64.32.150:28405 | phabricator_policy | Sleep | 5 |
162| 5209492380 | phuser | 10.64.32.150:28407 | phabricator_project | Sleep | 5 |
163| 5209492381 | phuser | 10.64.32.150:28408 | phabricator_spaces | Sleep | 5 |
164| 5209492382 | phuser | 10.64.32.150:28409 | phabricator_config | Sleep | 5 |
165| 5209492383 | phuser | 10.64.32.150:28410 | phabricator_project | Sleep | 5 |
166| 5209492384 | phuser | 10.64.32.150:28411 | phabricator_repository | Sleep | 4 |
167| 5209492385 | phuser | 10.64.32.150:28412 | phabricator_config | Sleep | 5 |
168| 5209492386 | phuser | 10.64.32.150:28413 | phabricator_spaces | Sleep | 5 |
169| 5209492387 | phuser | 10.64.32.150:28414 | phabricator_config | Sleep | 5 |
170| 5209492388 | phuser | 10.64.32.150:28415 | phabricator_spaces | Sleep | 5 |
171| 5209492389 | phuser | 10.64.32.150:28417 | phabricator_spaces | Sleep | 5 |
172| 5209492390 | phuser | 10.64.32.150:28416 | phabricator_spaces | Sleep | 5 |
173| 5209492391 | phuser | 10.64.32.150:28419 | phabricator_spaces | Sleep | 5 |
174| 5209492392 | phuser | 10.64.32.150:28420 | phabricator_policy | Sleep | 5 |
175| 5209492393 | phuser | 10.64.32.150:28421 | phabricator_project | Sleep | 4 |
176| 5209492394 | phuser | 10.64.32.150:28422 | phabricator_repository | Sleep | 4 |
177| 5209492395 | phuser | 10.64.32.150:28423 | phabricator_repository | Sleep | 0 |
178| 5209492396 | phuser | 10.64.32.150:28424 | phabricator_project | Sleep | 5 |
179| 5209492397 | phuser | 10.64.32.150:28425 | phabricator_config | Sleep | 5 |
180| 5209492398 | phuser | 10.64.32.150:28426 | phabricator_project | Sleep | 5 |
181| 5209492399 | phuser | 10.64.32.150:28427 | phabricator_policy | Sleep | 5 |
182| 5209492402 | phuser | 10.64.32.150:28430 | phabricator_policy | Sleep | 5 |
183| 5209492403 | phuser | 10.64.32.150:28432 | phabricator_repository | Sleep | 4 |
184| 5209492404 | phuser | 10.64.32.150:28433 | phabricator_project | Sleep | 4 |
185| 5209492405 | phuser | 10.64.32.150:28434 | phabricator_spaces | Sleep | 5 |
186| 5209492406 | phuser | 10.64.32.150:28435 | phabricator_repository | Sleep | 4 |
187| 5209492407 | phuser | 10.64.32.150:28436 | phabricator_policy | Sleep | 5 |
188| 5209492409 | phuser | 10.64.32.150:28438 | phabricator_config | Sleep | 5 |
189| 5209492410 | phuser | 10.64.32.150:28439 | phabricator_policy | Sleep | 5 |
190| 5209492413 | phuser | 10.64.32.150:28442 | phabricator_repository | Sleep | 4 |
191| 5209492414 | phuser | 10.64.32.150:28443 | phabricator_project | Sleep | 4 |
192| 5209492415 | phuser | 10.64.32.150:28444 | phabricator_config | Sleep | 4 |
193| 5209492416 | phuser | 10.64.32.150:28445 | phabricator_config | Sleep | 4 |
194| 5209492417 | phuser | 10.64.32.150:28446 | phabricator_config | Sleep | 4 |
195| 5209492421 | phuser | 10.64.32.150:28448 | phabricator_project | Sleep | 4 |
196| 5209492422 | phuser | 10.64.32.150:28449 | phabricator_config | Sleep | 4 |
197| 5209492424 | phuser | 10.64.32.150:28451 | phabricator_spaces | Sleep | 4 |
198| 5209492426 | phuser | 10.64.32.150:28453 | phabricator_policy | Sleep | 4 |
199| 5209492427 | phuser | 10.64.32.150:28454 | phabricator_config | Sleep | 4 |
200| 5209492429 | phuser | 10.64.32.150:28456 | phabricator_config | Sleep | 4 |
201| 5209492430 | phuser | 10.64.32.150:28457 | phabricator_policy | Sleep | 4 |
202| 5209492432 | phuser | 10.64.32.150:28459 | phabricator_repository | Sleep | 4 |
203| 5209492433 | phuser | 10.64.32.150:28460 | phabricator_repository | Sleep | 4 |
204| 5209492434 | phuser | 10.64.32.150:28461 | phabricator_spaces | Sleep | 4 |
205| 5209492436 | phuser | 10.64.32.150:28463 | phabricator_repository | Sleep | 4 |
206| 5209492437 | phuser | 10.64.32.150:28464 | phabricator_spaces | Sleep | 4 |
207| 5209492438 | phuser | 10.64.32.150:28465 | phabricator_spaces | Sleep | 4 |
208| 5209492439 | phuser | 10.64.32.150:28466 | phabricator_policy | Sleep | 4 |
209| 5209492440 | phuser | 10.64.32.150:28467 | phabricator_config | Sleep | 4 |
210| 5209492441 | phuser | 10.64.32.150:28468 | phabricator_project | Sleep | 4 |
211| 5209492442 | phuser | 10.64.32.150:28469 | phabricator_project | Sleep | 4 |
212| 5209492443 | phuser | 10.64.32.150:28470 | phabricator_repository | Sleep | 4 |
213| 5209492444 | phuser | 10.64.32.150:28471 | phabricator_config | Sleep | 4 |
214| 5209492445 | phuser | 10.64.32.150:28472 | phabricator_project | Sleep | 4 |
215| 5209492446 | phuser | 10.64.32.150:28473 | phabricator_config | Sleep | 4 |
216| 5209492447 | phuser | 10.64.32.150:28474 | phabricator_spaces | Sleep | 4 |
217| 5209492448 | phuser | 10.64.32.150:28475 | phabricator_policy | Sleep | 4 |
218| 5209492449 | phuser | 10.64.32.150:28477 | phabricator_project | Sleep | 4 |
219| 5209492450 | phuser | 10.64.32.150:28478 | phabricator_project | Sleep | 4 |
220| 5209492451 | phuser | 10.64.32.150:28479 | phabricator_policy | Sleep | 4 |
221| 5209492452 | phuser | 10.64.32.150:28480 | phabricator_policy | Sleep | 4 |
222| 5209492453 | phuser | 10.64.32.150:28481 | phabricator_repository | Sleep | 4 |
223| 5209492454 | phuser | 10.64.32.150:28482 | phabricator_policy | Sleep | 4 |
224| 5209492455 | phuser | 10.64.32.150:28483 | phabricator_repository | Sleep | 4 |
225| 5209492456 | phuser | 10.64.32.150:28484 | phabricator_spaces | Sleep | 4 |
226| 5209492457 | phuser | 10.64.32.150:28485 | phabricator_project | Sleep | 4 |
227| 5209492458 | phuser | 10.64.32.150:28486 | phabricator_config | Sleep | 4 |
228| 5209492459 | phuser | 10.64.32.150:28487 | phabricator_repository | Sleep | 4 |
229| 5209492460 | phuser | 10.64.32.150:28488 | phabricator_repository | Sleep | 4 |
230| 5209492461 | phuser | 10.64.32.150:28489 | phabricator_spaces | Sleep | 4 |
231| 5209492462 | phuser | 10.64.32.150:28490 | phabricator_spaces | Sleep | 4 |
232| 5209492463 | phuser | 10.64.32.150:28491 | phabricator_policy | Sleep | 4 |
233| 5209492464 | phuser | 10.64.32.150:28492 | phabricator_spaces | Sleep | 4 |
234| 5209492465 | phuser | 10.64.32.150:28493 | phabricator_config | Sleep | 4 |
235| 5209492466 | phuser | 10.64.32.150:28494 | phabricator_project | Sleep | 0 |
236| 5209492467 | phuser | 10.64.32.150:28495 | phabricator_config | Sleep | 4 |
237| 5209492468 | phuser | 10.64.32.150:28496 | phabricator_spaces | Sleep | 4 |
238| 5209492469 | phuser | 10.64.32.150:28497 | phabricator_project | Sleep | 4 |
239| 5209492470 | phuser | 10.64.32.150:28498 | phabricator_project | Sleep | 4 |
240| 5209492471 | phuser | 10.64.32.150:28499 | phabricator_policy | Sleep | 4 |
241| 5209492472 | phuser | 10.64.32.150:28500 | phabricator_spaces | Sleep | 4 |
242| 5209492473 | phuser | 10.64.32.150:28501 | phabricator_spaces | Sleep | 4 |
243| 5209492474 | phuser | 10.64.32.150:28502 | phabricator_policy | Sleep | 4 |
244| 5209492475 | phuser | 10.64.32.150:28503 | phabricator_project | Sleep | 4 |
245| 5209492476 | phuser | 10.64.32.150:28504 | phabricator_project | Sleep | 4 |
246| 5209492477 | phuser | 10.64.32.150:28505 | phabricator_repository | Sleep | 4 |
247| 5209492478 | phuser | 10.64.32.150:28506 | phabricator_repository | Sleep | 0 |
248| 5209492479 | phuser | 10.64.32.150:28507 | phabricator_policy | Sleep | 4 |
249| 5209492480 | phuser | 10.64.32.150:28508 | phabricator_spaces | Sleep | 4 |
250| 5209492481 | phuser | 10.64.32.150:28509 | phabricator_project | Sleep | 4 |
251| 5209492482 | phuser | 10.64.32.150:28510 | phabricator_repository | Sleep | 4 |
252| 5209492483 | phuser | 10.64.32.150:28511 | phabricator_project | Sleep | 4 |
253| 5209492484 | phuser | 10.64.32.150:28512 | phabricator_spaces | Sleep | 4 |
254| 5209492485 | phuser | 10.64.32.150:28513 | phabricator_repository | Sleep | 4 |
255| 5209492486 | phuser | 10.64.32.150:28514 | phabricator_policy | Sleep | 4 |
256| 5209492487 | phuser | 10.64.32.150:28515 | phabricator_config | Sleep | 4 |
257| 5209492488 | phuser | 10.64.32.150:28516 | phabricator_policy | Sleep | 4 |
258| 5209492489 | phuser | 10.64.32.150:28517 | phabricator_policy | Sleep | 4 |
259| 5209492490 | phuser | 10.64.32.150:28518 | phabricator_repository | Sleep | 4 |
260| 5209492491 | phuser | 10.64.32.150:28519 | phabricator_policy | Sleep | 4 |
261| 5209492492 | phuser | 10.64.32.150:28520 | phabricator_repository | Sleep | 4 |
262| 5209492493 | phuser | 10.64.32.150:28521 | phabricator_repository | Sleep | 4 |
263| 5209492494 | phuser | 10.64.32.150:28522 | phabricator_project | Sleep | 4 |
264| 5209492495 | phuser | 10.64.32.150:28523 | phabricator_policy | Sleep | 4 |
265| 5209492496 | phuser | 10.64.32.150:28524 | phabricator_repository | Sleep | 4 |
266| 5209492497 | phuser | 10.64.32.150:28525 | phabricator_policy | Sleep | 4 |
267| 5209492498 | phuser | 10.64.32.150:28526 | phabricator_spaces | Sleep | 4 |
268| 5209492499 | phuser | 10.64.32.150:28527 | phabricator_spaces | Sleep | 4 |
269| 5209492500 | phuser | 10.64.32.150:28528 | phabricator_project | Sleep | 4 |
270| 5209492501 | phuser | 10.64.32.150:28529 | phabricator_repository | Sleep | 3 |
271| 5209492503 | phuser | 10.64.32.150:28531 | phabricator_repository | Sleep | 4 |
272| 5209492507 | phuser | 10.64.32.150:28535 | phabricator_policy | Sleep | 4 |
273| 5209492508 | phuser | 10.64.32.150:28536 | phabricator_project | Sleep | 4 |
274| 5209492509 | phuser | 10.64.32.150:28538 | phabricator_config | Sleep | 4 |
275| 5209492510 | phuser | 10.64.32.150:28539 | phabricator_policy | Sleep | 4 |
276| 5209492511 | phuser | 10.64.32.150:28540 | phabricator_config | Sleep | 4 |
277| 5209492513 | phuser | 10.64.32.150:28542 | phabricator_repository | Sleep | 3 |
278| 5209492514 | phuser | 10.64.32.150:28543 | phabricator_repository | Sleep | 3 |
279| 5209492516 | phuser | 10.64.32.150:28545 | phabricator_spaces | Sleep | 4 |
280| 5209492520 | phuser | 10.64.32.150:28550 | phabricator_spaces | Sleep | 4 |
281| 5209492521 | phuser | 10.64.32.150:28551 | phabricator_project | Sleep | 3 |
282| 5209492522 | phuser | 10.64.32.150:28552 | phabricator_project | Sleep | 3 |
283| 5209492527 | phuser | 10.64.32.150:28557 | phabricator_config | Sleep | 3 |
284| 5209492529 | phuser | 10.64.32.150:28560 | phabricator_config | Sleep | 3 |
285| 5209492531 | phuser | 10.64.32.150:28562 | phabricator_config | Sleep | 3 |
286| 5209492532 | phuser | 10.64.32.150:28563 | phabricator_config | Sleep | 3 |
287| 5209492533 | phuser | 10.64.32.150:28564 | phabricator_config | Sleep | 3 |
288| 5209492534 | phuser | 10.64.32.150:28565 | phabricator_config | Sleep | 3 |
289| 5209492535 | phuser | 10.64.32.150:28566 | phabricator_config | Sleep | 3 |
290| 5209492536 | phuser | 10.64.32.150:28567 | phabricator_config | Sleep | 3 |
291| 5209492537 | phuser | 10.64.32.150:28568 | phabricator_config | Sleep | 3 |
292| 5209492538 | phuser | 10.64.32.150:28569 | phabricator_config | Sleep | 3 |
293| 5209492539 | phuser | 10.64.32.150:28570 | phabricator_config | Sleep | 3 |
294| 5209492540 | phuser | 10.64.32.150:28571 | phabricator_config | Sleep | 3 |
295| 5209492541 | phuser | 10.64.32.150:28572 | phabricator_config | Sleep | 3 |
296| 5209492542 | phuser | 10.64.32.150:28573 | phabricator_config | Sleep | 3 |
297| 5209492543 | phuser | 10.64.32.150:28574 | phabricator_config | Sleep | 3 |
298| 5209492544 | phuser | 10.64.32.150:28575 | phabricator_config | Sleep | 3 |
299| 5209492545 | phuser | 10.64.32.150:28576 | phabricator_config | Sleep | 3 |
300| 5209492546 | phuser | 10.64.32.150:28577 | phabricator_spaces | Sleep | 3 |
301| 5209492547 | phuser | 10.64.32.150:28578 | phabricator_config | Sleep | 3 |
302| 5209492548 | phuser | 10.64.32.150:28579 | phabricator_policy | Sleep | 3 |
303| 5209492549 | phuser | 10.64.32.150:28580 | phabricator_policy | Sleep | 3 |
304| 5209492550 | phuser | 10.64.32.150:28581 | phabricator_config | Sleep | 3 |
305| 5209492551 | phuser | 10.64.32.150:28582 | phabricator_spaces | Sleep | 3 |
306| 5209492552 | phuser | 10.64.32.150:28583 | phabricator_config | Sleep | 3 |
307| 5209492553 | phuser | 10.64.32.150:28584 | phabricator_spaces | Sleep | 3 |
308| 5209492555 | phuser | 10.64.32.150:28586 | phabricator_repository | Sleep | 3 |
309| 5209492556 | phuser | 10.64.32.150:28587 | phabricator_repository | Sleep | 2 |
310| 5209492557 | phuser | 10.64.32.150:28588 | phabricator_project | Sleep | 3 |
311| 5209492558 | phuser | 10.64.32.150:28589 | phabricator_project | Sleep | 3 |
312| 5209492559 | phuser | 10.64.32.150:28590 | phabricator_spaces | Sleep | 3 |
313| 5209492560 | phuser | 10.64.32.150:28591 | phabricator_spaces | Sleep | 3 |
314| 5209492561 | phuser | 10.64.32.150:28592 | phabricator_spaces | Sleep | 3 |
315| 5209492562 | phuser | 10.64.32.150:28593 | phabricator_spaces | Sleep | 3 |
316| 5209492563 | phuser | 10.64.32.150:28594 | phabricator_spaces | Sleep | 3 |
317| 5209492564 | phuser | 10.64.32.150:28595 | phabricator_spaces | Sleep | 3 |
318| 5209492565 | phuser | 10.64.32.150:28596 | phabricator_project | Sleep | 3 |
319| 5209492566 | phuser | 10.64.32.150:28597 | phabricator_spaces | Sleep | 3 |
320| 5209492567 | phuser | 10.64.32.150:28598 | phabricator_project | Sleep | 3 |
321| 5209492568 | phuser | 10.64.32.150:28599 | phabricator_project | Sleep | 3 |
322| 5209492569 | phuser | 10.64.32.150:28600 | phabricator_project | Sleep | 3 |
323| 5209492570 | phuser | 10.64.32.150:28601 | phabricator_project | Sleep | 3 |
324| 5209492571 | phuser | 10.64.32.150:28602 | phabricator_policy | Sleep | 3 |
325| 5209492572 | phuser | 10.64.32.150:28603 | phabricator_project | Sleep | 3 |
326| 5209492573 | phuser | 10.64.32.150:28604 | phabricator_repository | Sleep | 3 |
327| 5209492574 | phuser | 10.64.32.150:28605 | phabricator_policy | Sleep | 3 |
328| 5209492575 | phuser | 10.64.32.150:28606 | phabricator_policy | Sleep | 3 |
329| 5209492576 | phuser | 10.64.32.150:28607 | phabricator_policy | Sleep | 3 |
330| 5209492577 | phuser | 10.64.32.150:28608 | phabricator_policy | Sleep | 3 |
331| 5209492578 | phuser | 10.64.32.150:28609 | phabricator_project | Sleep | 3 |
332| 5209492579 | phuser | 10.64.32.150:28610 | phabricator_repository | Sleep | 3 |
333| 5209492580 | phuser | 10.64.32.150:28611 | phabricator_repository | Sleep | 3 |
334| 5209492581 | phuser | 10.64.32.150:28612 | phabricator_policy | Sleep | 3 |
335| 5209492582 | phuser | 10.64.32.150:28613 | phabricator_project | Sleep | 3 |
336| 5209492583 | phuser | 10.64.32.150:28614 | phabricator_spaces | Sleep | 3 |
337| 5209492584 | phuser | 10.64.32.150:28615 | phabricator_spaces | Sleep | 3 |
338| 5209492585 | phuser | 10.64.32.150:28616 | phabricator_repository | Sleep | 3 |
339| 5209492586 | phuser | 10.64.32.150:28617 | phabricator_project | Sleep | 3 |
340| 5209492587 | phuser | 10.64.32.150:28618 | phabricator_project | Sleep | 3 |
341| 5209492588 | phuser | 10.64.32.150:28619 | phabricator_policy | Sleep | 3 |
342| 5209492589 | phuser | 10.64.32.150:28620 | phabricator_spaces | Sleep | 3 |
343| 5209492590 | phuser | 10.64.32.150:28621 | phabricator_policy | Sleep | 3 |
344| 5209492592 | phuser | 10.64.32.150:28623 | phabricator_policy | Sleep | 3 |
345| 5209492593 | phuser | 10.64.32.150:28624 | phabricator_repository | Sleep | 2 |
346| 5209492594 | phuser | 10.64.32.150:28625 | phabricator_policy | Sleep | 3 |
347| 5209492595 | phuser | 10.64.32.150:28626 | phabricator_repository | Sleep | 3 |
348| 5209492596 | phuser | 10.64.32.150:28627 | phabricator_project | Sleep | 3 |
349| 5209492597 | phuser | 10.64.32.150:28628 | phabricator_spaces | Sleep | 3 |
350| 5209492599 | phuser | 10.64.32.150:28630 | phabricator_spaces | Sleep | 3 |
351| 5209492601 | phuser | 10.64.32.150:28632 | phabricator_project | Sleep | 3 |
352| 5209492602 | phuser | 10.64.32.150:28633 | phabricator_repository | Sleep | 2 |
353| 5209492604 | phuser | 10.64.32.150:28635 | phabricator_policy | Sleep | 3 |
354| 5209492605 | phuser | 10.64.32.150:28636 | phabricator_project | Sleep | 3 |
355| 5209492606 | phuser | 10.64.32.150:28637 | phabricator_repository | Sleep | 2 |
356| 5209492607 | phuser | 10.64.32.150:28638 | phabricator_spaces | Sleep | 3 |
357| 5209492608 | phuser | 10.64.32.150:28639 | phabricator_policy | Sleep | 3 |
358| 5209492609 | phuser | 10.64.32.150:28640 | phabricator_repository | Sleep | 3 |
359| 5209492610 | phuser | 10.64.32.150:28641 | phabricator_repository | Sleep | 2 |
360| 5209492611 | phuser | 10.64.32.150:28642 | phabricator_project | Sleep | 3 |
361| 5209492612 | phuser | 10.64.32.150:28643 | phabricator_config | Sleep | 3 |
362| 5209492613 | phuser | 10.64.32.150:28644 | phabricator_repository | Sleep | 2 |
363| 5209492616 | phuser | 10.64.32.150:28647 | phabricator_policy | Sleep | 3 |
364| 5209492617 | phuser | 10.64.32.150:28648 | phabricator_config | Sleep | 3 |
365| 5209492618 | phuser | 10.64.32.150:28649 | phabricator_policy | Sleep | 3 |
366| 5209492621 | phuser | 10.64.32.150:28652 | phabricator_policy | Sleep | 3 |
367| 5209492622 | phuser | 10.64.32.150:28653 | phabricator_spaces | Sleep | 3 |
368| 5209492623 | phuser | 10.64.32.150:28654 | phabricator_spaces | Sleep | 3 |
369| 5209492624 | phuser | 10.64.32.150:28655 | phabricator_project | Sleep | 3 |
370| 5209492625 | phuser | 10.64.32.150:28656 | phabricator_repository | Sleep | 2 |
371| 5209492626 | phuser | 10.64.32.150:28657 | phabricator_project | Sleep | 3 |
372| 5209492627 | phuser | 10.64.32.150:28658 | phabricator_policy | Sleep | 3 |
373| 5209492628 | phuser | 10.64.32.150:28659 | phabricator_policy | Sleep | 3 |
374| 5209492629 | phuser | 10.64.32.150:28660 | phabricator_config | Sleep | 3 |
375| 5209492630 | phuser | 10.64.32.150:28661 | phabricator_config | Sleep | 3 |
376| 5209492631 | phuser | 10.64.32.150:28662 | phabricator_repository | Sleep | 2 |
377| 5209492632 | phuser | 10.64.32.150:28663 | phabricator_config | Sleep | 3 |
378| 5209492633 | phuser | 10.64.32.150:28664 | phabricator_config | Sleep | 3 |
379| 5209492634 | phuser | 10.64.32.150:28666 | phabricator_config | Sleep | 3 |
380| 5209492635 | phuser | 10.64.32.150:28667 | phabricator_spaces | Sleep | 3 |
381| 5209492636 | phuser | 10.64.32.150:28668 | phabricator_repository | Sleep | 2 |
382| 5209492637 | phuser | 10.64.32.150:28669 | phabricator_project | Sleep | 3 |
383| 5209492638 | phuser | 10.64.32.150:28670 | phabricator_config | Sleep | 3 |
384| 5209492639 | phuser | 10.64.32.150:28671 | phabricator_config | Sleep | 3 |
385| 5209492640 | phuser | 10.64.32.150:28672 | phabricator_config | Sleep | 3 |
386| 5209492641 | phuser | 10.64.32.150:28673 | phabricator_config | Sleep | 3 |
387| 5209492642 | phuser | 10.64.32.150:28674 | phabricator_config | Sleep | 3 |
388| 5209492643 | phuser | 10.64.32.150:28675 | phabricator_repository | Sleep | 2 |
389| 5209492644 | phuser | 10.64.32.150:28676 | phabricator_policy | Sleep | 3 |
390| 5209492645 | phuser | 10.64.32.150:28678 | phabricator_config | Sleep | 3 |
391| 5209492646 | phuser | 10.64.32.150:28677 | phabricator_config | Sleep | 3 |
392| 5209492647 | phuser | 10.64.32.150:28680 | phabricator_spaces | Sleep | 3 |
393| 5209492648 | phuser | 10.64.32.150:28681 | phabricator_spaces | Sleep | 3 |
394| 5209492649 | phuser | 10.64.32.150:28682 | phabricator_spaces | Sleep | 3 |
395| 5209492650 | phuser | 10.64.32.150:28683 | phabricator_project | Sleep | 2 |
396| 5209492651 | phuser | 10.64.32.150:28684 | phabricator_spaces | Sleep | 3 |
397| 5209492652 | phuser | 10.64.32.150:28685 | phabricator_spaces | Sleep | 3 |
398| 5209492653 | phuser | 10.64.32.150:28686 | phabricator_spaces | Sleep | 3 |
399| 5209492654 | phuser | 10.64.32.150:28687 | phabricator_config | Sleep | 3 |
400| 5209492655 | phuser | 10.64.32.150:28688 | phabricator_config | Sleep | 3 |
401| 5209492656 | phuser | 10.64.32.150:28689 | phabricator_spaces | Sleep | 3 |
402| 5209492657 | phuser | 10.64.32.150:28690 | phabricator_config | Sleep | 3 |
403| 5209492658 | phuser | 10.64.32.150:28691 | phabricator_config | Sleep | 3 |
404| 5209492660 | phuser | 10.64.32.150:28693 | phabricator_project | Sleep | 3 |
405| 5209492661 | phuser | 10.64.32.150:28695 | phabricator_project | Sleep | 2 |
406| 5209492662 | phuser | 10.64.32.150:28694 | phabricator_spaces | Sleep | 3 |
407| 5209492663 | phuser | 10.64.32.150:28696 | phabricator_config | Sleep | 3 |
408| 5209492664 | phuser | 10.64.32.150:28697 | phabricator_spaces | Sleep | 3 |
409| 5209492665 | phuser | 10.64.32.150:28698 | phabricator_config | Sleep | 3 |
410| 5209492666 | phuser | 10.64.32.150:28699 | phabricator_policy | Sleep | 3 |
411| 5209492667 | phuser | 10.64.32.150:28700 | phabricator_spaces | Sleep | 3 |
412| 5209492668 | phuser | 10.64.32.150:28701 | phabricator_config | Sleep | 3 |
413| 5209492669 | phuser | 10.64.32.150:28702 | phabricator_config | Sleep | 3 |
414| 5209492670 | phuser | 10.64.32.150:28703 | phabricator_config | Sleep | 3 |
415| 5209492671 | phuser | 10.64.32.150:28704 | phabricator_config | Sleep | 3 |
416| 5209492672 | phuser | 10.64.32.150:28705 | phabricator_spaces | Sleep | 3 |
417| 5209492673 | phuser | 10.64.32.150:28706 | phabricator_spaces | Sleep | 3 |
418| 5209492674 | phuser | 10.64.32.150:28707 | phabricator_spaces | Sleep | 3 |
419| 5209492675 | phuser | 10.64.32.150:28708 | phabricator_config | Sleep | 3 |
420| 5209492678 | phuser | 10.64.32.150:28711 | phabricator_spaces | Sleep | 3 |
421| 5209492680 | phuser | 10.64.32.150:28713 | phabricator_project | Sleep | 2 |
422| 5209492681 | phuser | 10.64.32.150:28715 | phabricator_spaces | Sleep | 3 |
423| 5209492682 | phuser | 10.64.32.150:28714 | phabricator_spaces | Sleep | 3 |
424| 5209492683 | phuser | 10.64.32.150:28716 | phabricator_spaces | Sleep | 3 |
425| 5209492684 | phuser | 10.64.32.150:28717 | phabricator_config | Sleep | 3 |
426| 5209492685 | phuser | 10.64.32.150:28718 | phabricator_project | Sleep | 2 |
427| 5209492686 | phuser | 10.64.32.150:28719 | phabricator_repository | Sleep | 2 |
428| 5209492687 | phuser | 10.64.32.150:28720 | phabricator_config | Sleep | 3 |
429| 5209492688 | phuser | 10.64.32.150:28721 | phabricator_config | Sleep | 3 |
430| 5209492689 | phuser | 10.64.32.150:28722 | phabricator_config | Sleep | 3 |
431| 5209492691 | phuser | 10.64.32.150:28725 | phabricator_project | Sleep | 2 |
432| 5209492692 | phuser | 10.64.32.150:28726 | phabricator_project | Sleep | 2 |
433| 5209492693 | phuser | 10.64.32.150:28727 | phabricator_project | Sleep | 3 |
434| 5209492694 | phuser | 10.64.32.150:28728 | phabricator_repository | Sleep | 2 |
435| 5209492695 | phuser | 10.64.32.150:28729 | phabricator_policy | Sleep | 3 |
436| 5209492696 | phuser | 10.64.32.150:28731 | phabricator_policy | Sleep | 3 |
437| 5209492697 | phuser | 10.64.32.150:28730 | phabricator_policy | Sleep | 3 |
438| 5209492698 | phuser | 10.64.32.150:28732 | phabricator_project | Sleep | 2 |
439| 5209492699 | phuser | 10.64.32.150:28733 | phabricator_project | Sleep | 2 |
440| 5209492701 | phuser | 10.64.32.150:28735 | phabricator_policy | Sleep | 2 |
441| 5209492702 | phuser | 10.64.32.150:28736 | phabricator_project | Sleep | 2 |
442| 5209492703 | phuser | 10.64.32.150:28737 | phabricator_project | Sleep | 3 |
443| 5209492704 | phuser | 10.64.32.150:28738 | phabricator_policy | Sleep | 3 |
444| 5209492705 | phuser | 10.64.32.150:28740 | phabricator_repository | Sleep | 2 |
445| 5209492706 | phuser | 10.64.32.150:28741 | phabricator_policy | Sleep | 3 |
446| 5209492707 | phuser | 10.64.32.150:28742 | phabricator_repository | Sleep | 2 |
447| 5209492708 | phuser | 10.64.32.150:28743 | phabricator_project | Sleep | 2 |
448| 5209492709 | phuser | 10.64.32.150:28744 | phabricator_repository | Sleep | 2 |
449| 5209492711 | phuser | 10.64.32.150:28746 | phabricator_spaces | Sleep | 3 |
450| 5209492712 | phuser | 10.64.32.150:28747 | phabricator_spaces | Sleep | 3 |
451| 5209492713 | phuser | 10.64.32.150:28748 | phabricator_policy | Sleep | 2 |
452| 5209492714 | phuser | 10.64.32.150:28749 | phabricator_repository | Sleep | 2 |
453| 5209492715 | phuser | 10.64.32.150:28750 | phabricator_repository | Sleep | 1 |
454| 5209492716 | phuser | 10.64.32.150:28751 | phabricator_project | Sleep | 2 |
455| 5209492717 | phuser | 10.64.32.150:28752 | phabricator_repository | Sleep | 2 |
456| 5209492720 | phuser | 10.64.32.150:28755 | phabricator_project | Sleep | 2 |
457| 5209492721 | phuser | 10.64.32.150:28756 | phabricator_policy | Sleep | 2 |
458| 5209492722 | phuser | 10.64.32.150:28757 | phabricator_repository | Sleep | 2 |
459| 5209492727 | phuser | 10.64.32.150:28762 | phabricator_config | Sleep | 2 |
460| 5209492729 | phuser | 10.64.32.150:28764 | phabricator_project | Sleep | 2 |
461| 5209492730 | phuser | 10.64.32.150:28766 | phabricator_policy | Sleep | 2 |
462| 5209492731 | phuser | 10.64.32.150:28765 | phabricator_policy | Sleep | 2 |
463| 5209492732 | phuser | 10.64.32.150:28768 | phabricator_repository | Sleep | 2 |
464| 5209492733 | phuser | 10.64.32.150:28769 | phabricator_repository | Sleep | 1 |
465| 5209492734 | phuser | 10.64.32.150:28770 | phabricator_policy | Sleep | 2 |
466| 5209492735 | phuser | 10.64.32.150:28771 | phabricator_config | Sleep | 2 |
467| 5209492736 | phuser | 10.64.32.150:28772 | phabricator_spaces | Sleep | 2 |
468| 5209492737 | phuser | 10.64.32.150:28774 | phabricator_repository | Sleep | 1 |
469| 5209492738 | phuser | 10.64.32.150:28775 | phabricator_project | Sleep | 2 |
470| 5209492739 | phuser | 10.64.32.150:28776 | phabricator_spaces | Sleep | 2 |
471| 5209492741 | phuser | 10.64.32.150:28778 | phabricator_project | Sleep | 2 |
472| 5209492742 | phuser | 10.64.32.150:28779 | phabricator_policy | Sleep | 2 |
473| 5209492743 | phuser | 10.64.32.150:28780 | phabricator_spaces | Sleep | 2 |
474| 5209492745 | phuser | 10.64.32.150:28782 | phabricator_project | Sleep | 2 |
475| 5209492746 | phuser | 10.64.32.150:28783 | phabricator_policy | Sleep | 2 |
476| 5209492747 | phuser | 10.64.32.150:28784 | phabricator_policy | Sleep | 2 |
477| 5209492748 | phuser | 10.64.32.150:28785 | phabricator_policy | Sleep | 2 |
478| 5209492752 | phuser | 10.64.32.150:28789 | phabricator_spaces | Sleep | 2 |
479| 5209492753 | phuser | 10.64.32.150:28790 | phabricator_spaces | Sleep | 2 |
480| 5209492754 | phuser | 10.64.32.150:28791 | phabricator_spaces | Sleep | 2 |
481| 5209492755 | phuser | 10.64.32.150:28792 | phabricator_spaces | Sleep | 2 |
482| 5209492756 | phuser | 10.64.32.150:28793 | phabricator_spaces | Sleep | 2 |
483| 5209492757 | phuser | 10.64.32.150:28794 | phabricator_project | Sleep | 2 |
484| 5209492758 | phuser | 10.64.32.150:28795 | phabricator_policy | Sleep | 2 |
485| 5209492759 | phuser | 10.64.32.150:28796 | phabricator_config | Sleep | 2 |
486| 5209492760 | phuser | 10.64.32.150:28797 | phabricator_config | Sleep | 2 |
487| 5209492761 | phuser | 10.64.32.150:28798 | phabricator_repository | Sleep | 1 |
488| 5209492764 | phuser | 10.64.32.150:28801 | phabricator_config | Sleep | 2 |
489| 5209492765 | phuser | 10.64.32.150:28802 | phabricator_spaces | Sleep | 2 |
490| 5209492766 | phuser | 10.64.32.150:28800 | phabricator_spaces | Sleep | 2 |
491| 5209492767 | phuser | 10.64.32.150:28803 | phabricator_differential | Sleep | 2 |
492| 5209492768 | phuser | 10.64.32.150:28804 | phabricator_policy | Sleep | 2 |
493| 5209492769 | phuser | 10.64.32.150:28805 | phabricator_project | Sleep | 2 |
494| 5209492770 | phuser | 10.64.32.150:28806 | phabricator_spaces | Sleep | 2 |
495| 5209492771 | phuser | 10.64.32.150:28807 | phabricator_project | Sleep | 1 |
496| 5209492772 | phuser | 10.64.32.150:28808 | phabricator_project | Sleep | 2 |
497| 5209492773 | phuser | 10.64.32.150:28809 | phabricator_repository | Sleep | 1 |
498| 5209492774 | phuser | 10.64.32.150:28810 | phabricator_spaces | Sleep | 2 |
499| 5209492775 | phuser | 10.64.32.150:28811 | phabricator_repository | Sleep | 1 |
500| 5209492776 | phuser | 10.64.32.150:28812 | phabricator_project | Sleep | 2 |
501| 5209492777 | phuser | 10.64.32.150:28813 | phabricator_repository | Sleep | 2 |
502| 5209492778 | phuser | 10.64.32.150:28814 | phabricator_repository | Sleep | 1 |
503| 5209492779 | phuser | 10.64.32.150:28815 | phabricator_repository | Sleep | 2 |
504| 5209492780 | phuser | 10.64.32.150:28816 | phabricator_project | Sleep | 1 |
505| 5209492781 | phuser | 10.64.32.150:28817 | phabricator_project | Sleep | 2 |
506| 5209492783 | phuser | 10.64.32.150:28819 | phabricator_policy | Sleep | 2 |
507| 5209492784 | phuser | 10.64.32.150:28820 | phabricator_project | Sleep | 2 |
508| 5209492785 | phuser | 10.64.32.150:28821 | phabricator_policy | Sleep | 2 |
509| 5209492786 | phuser | 10.64.32.150:28822 | phabricator_repository | Sleep | 0 |
510| 5209492787 | phuser | 10.64.32.150:28823 | phabricator_policy | Sleep | 2 |
511| 5209492788 | phuser | 10.64.32.150:28824 | phabricator_repository | Sleep | 1 |
512| 5209492789 | phuser | 10.64.32.150:28825 | phabricator_policy | Sleep | 2 |
513| 5209492790 | phuser | 10.64.32.150:28827 | phabricator_policy | Sleep | 2 |
514| 5209492791 | phuser | 10.64.32.150:28826 | phabricator_project | Sleep | 1 |
515| 5209492792 | phuser | 10.64.32.150:28828 | phabricator_repository | Sleep | 2 |
516| 5209492794 | phuser | 10.64.32.150:28830 | phabricator_repository | Sleep | 1 |
517| 5209492795 | phuser | 10.64.32.150:28831 | phabricator_repository | Sleep | 1 |
518| 5209492796 | phuser | 10.64.32.150:28832 | phabricator_policy | Sleep | 2 |
519| 5209492797 | phuser | 10.64.32.150:28833 | phabricator_file | Sleep | 2 |
520| 5209492798 | phuser | 10.64.32.150:28834 | phabricator_policy | Sleep | 2 |
521| 5209492799 | phuser | 10.64.32.150:28835 | phabricator_project | Sleep | 1 |
522| 5209492800 | phuser | 10.64.32.150:28836 | phabricator_repository | Sleep | 1 |
523| 5209492801 | phuser | 10.64.32.150:28837 | phabricator_project | Sleep | 2 |
524| 5209492802 | phuser | 10.64.32.150:28838 | phabricator_policy | Sleep | 2 |
525| 5209492804 | phuser | 10.64.32.150:28840 | phabricator_project | Sleep | 2 |
526| 5209492805 | phuser | 10.64.32.150:28841 | phabricator_repository | Sleep | 1 |
527| 5209492806 | phuser | 10.64.32.150:28842 | phabricator_spaces | Sleep | 2 |
528| 5209492810 | phuser | 10.64.32.150:28847 | phabricator_repository | Sleep | 1 |
529| 5209492812 | phuser | 10.64.32.150:28849 | phabricator_policy | Sleep | 2 |
530| 5209492813 | phuser | 10.64.32.150:28850 | phabricator_policy | Sleep | 2 |
531| 5209492814 | phuser | 10.64.32.150:28851 | phabricator_repository | Sleep | 1 |
532| 5209492815 | phuser | 10.64.32.150:28852 | phabricator_repository | Sleep | 2 |
533| 5209492816 | phuser | 10.64.32.150:28853 | phabricator_repository | Sleep | 1 |
534| 5209492817 | phuser | 10.64.32.150:28855 | phabricator_config | Sleep | 2 |
535| 5209492818 | phuser | 10.64.32.150:28857 | phabricator_config | Sleep | 2 |
536| 5209492819 | phuser | 10.64.32.150:28856 | phabricator_config | Sleep | 2 |
537| 5209492820 | phuser | 10.64.32.150:28859 | phabricator_config | Sleep | 2 |
538| 5209492821 | phuser | 10.64.32.150:28860 | phabricator_config | Sleep | 2 |
539| 5209492822 | phuser | 10.64.32.150:28861 | phabricator_spaces | Sleep | 2 |
540| 5209492823 | phuser | 10.64.32.150:28862 | phabricator_differential | Sleep | 2 |
541| 5209492824 | phuser | 10.64.32.150:28864 | phabricator_config | Sleep | 2 |
542| 5209492825 | phuser | 10.64.32.150:28863 | phabricator_differential | Sleep | 1 |
543| 5209492826 | phuser | 10.64.32.150:28865 | phabricator_config | Sleep | 2 |
544| 5209492827 | phuser | 10.64.32.150:28866 | phabricator_project | Sleep | 1 |
545| 5209492828 | phuser | 10.64.32.150:28867 | phabricator_spaces | Sleep | 2 |
546| 5209492829 | phuser | 10.64.32.150:28868 | phabricator_config | Sleep | 2 |
547| 5209492830 | phuser | 10.64.32.150:28870 | phabricator_policy | Sleep | 2 |
548| 5209492836 | phuser | 10.64.32.150:28876 | phabricator_spaces | Sleep | 1 |
549| 5209492837 | phuser | 10.64.32.150:28877 | phabricator_policy | Sleep | 2 |
550| 5209492839 | phuser | 10.64.32.150:28879 | phabricator_config | Sleep | 2 |
551| 5209492841 | phuser | 10.64.32.150:28881 | phabricator_repository | Sleep | 1 |
552| 5209492843 | phuser | 10.64.32.150:28884 | phabricator_policy | Sleep | 2 |
553| 5209492844 | phuser | 10.64.32.150:28885 | phabricator_differential | Sleep | 2 |
554| 5209492845 | phuser | 10.64.32.150:28886 | phabricator_project | Sleep | 1 |
555| 5209492846 | phuser | 10.64.32.150:28887 | phabricator_differential | Sleep | 2 |
556| 5209492847 | phuser | 10.64.32.150:28888 | phabricator_spaces | Sleep | 2 |
557| 5209492848 | phuser | 10.64.32.150:28889 | phabricator_repository | Sleep | 0 |
558| 5209492849 | phuser | 10.64.32.150:28890 | phabricator_config | Sleep | 2 |
559| 5209492850 | phuser | 10.64.32.150:28891 | phabricator_project | Sleep | 1 |
560| 5209492851 | phuser | 10.64.32.150:28892 | phabricator_project | Sleep | 1 |
561| 5209492852 | phuser | 10.64.32.150:28893 | phabricator_spaces | Sleep | 2 |
562| 5209492853 | phuser | 10.64.32.150:28894 | phabricator_spaces | Sleep | 2 |
563| 5209492854 | phuser | 10.64.32.150:28895 | phabricator_policy | Sleep | 2 |
564| 5209492855 | phuser | 10.64.32.150:28896 | phabricator_config | Sleep | 1 |
565| 5209492856 | phuser | 10.64.32.150:28897 | phabricator_policy | Sleep | 1 |
566| 5209492857 | phuser | 10.64.32.150:28898 | phabricator_differential | Sleep | 1 |
567| 5209492858 | phuser | 10.64.32.150:28899 | phabricator_config | Sleep | 1 |
568| 5209492859 | phuser | 10.64.32.150:28900 | phabricator_policy | Sleep | 1 |
569| 5209492860 | phuser | 10.64.32.150:28901 | phabricator_config | Sleep | 1 |
570| 5209492861 | phuser | 10.64.32.150:28902 | phabricator_config | Sleep | 1 |
571| 5209492862 | phuser | 10.64.32.150:28903 | phabricator_repository | Sleep | 1 |
572| 5209492864 | phuser | 10.64.32.150:28905 | phabricator_spaces | Sleep | 1 |
573| 5209492865 | phuser | 10.64.32.150:28906 | phabricator_differential | Sleep | 1 |
574| 5209492866 | phuser | 10.64.32.150:28907 | phabricator_file | Sleep | 1 |
575| 5209492867 | phuser | 10.64.32.150:28908 | phabricator_project | Sleep | 1 |
576| 5209492868 | phuser | 10.64.32.150:28912 | phabricator_config | Sleep | 1 |
577| 5209492869 | phuser | 10.64.32.150:28910 | phabricator_config | Sleep | 1 |
578| 5209492870 | phuser | 10.64.32.150:28909 | phabricator_policy | Sleep | 1 |
579| 5209492871 | phuser | 10.64.32.150:28911 | phabricator_config | Sleep | 1 |
580| 5209492872 | phuser | 10.64.32.150:28913 | phabricator_config | Sleep | 1 |
581| 5209492873 | phuser | 10.64.32.150:28914 | phabricator_policy | Sleep | 1 |
582| 5209492874 | phuser | 10.64.32.150:28915 | phabricator_config | Sleep | 1 |
583| 5209492875 | phuser | 10.64.32.150:28916 | phabricator_config | Sleep | 1 |
584| 5209492876 | phuser | 10.64.32.150:28917 | phabricator_repository | Sleep | 0 |
585| 5209492877 | phuser | 10.64.32.150:28919 | phabricator_config | Sleep | 1 |
586| 5209492878 | phuser | 10.64.32.150:28918 | phabricator_config | Sleep | 1 |
587| 5209492879 | phuser | 10.64.32.150:28920 | phabricator_spaces | Sleep | 1 |
588| 5209492880 | phuser | 10.64.32.150:28921 | phabricator_spaces | Sleep | 1 |
589| 5209492882 | phuser | 10.64.32.150:28923 | phabricator_repository | Sleep | 1 |
590| 5209492883 | phuser | 10.64.32.150:28924 | phabricator_project | Sleep | 1 |
591| 5209492884 | phuser | 10.64.32.150:28925 | phabricator_config | Sleep | 1 |
592| 5209492885 | phuser | 10.64.32.150:28926 | phabricator_spaces | Sleep | 1 |
593| 5209492886 | phuser | 10.64.32.150:28927 | phabricator_spaces | Sleep | 1 |
594| 5209492887 | phuser | 10.64.32.150:28928 | phabricator_spaces | Sleep | 1 |
595| 5209492888 | phuser | 10.64.32.150:28929 | phabricator_spaces | Sleep | 1 |
596| 5209492889 | phuser | 10.64.32.150:28930 | phabricator_spaces | Sleep | 1 |
597| 5209492890 | phuser | 10.64.32.150:28931 | phabricator_file | Sleep | 1 |
598| 5209492892 | phuser | 10.64.32.150:28932 | phabricator_repository | Sleep | 1 |
599| 5209492896 | phuser | 10.64.32.150:28937 | phabricator_spaces | Sleep | 1 |
600| 5209492897 | phuser | 10.64.32.150:28938 | phabricator_spaces | Sleep | 1 |
601| 5209492898 | phuser | 10.64.32.150:28939 | phabricator_repository | Sleep | 1 |
602| 5209492899 | phuser | 10.64.32.150:28940 | phabricator_config | Sleep | 1 |
603| 5209492900 | phuser | 10.64.32.150:28941 | phabricator_repository | Sleep | 1 |
604| 5209492901 | phuser | 10.64.32.150:28942 | phabricator_project | Sleep | 1 |
605| 5209492902 | phuser | 10.64.32.150:28943 | phabricator_project | Sleep | 1 |
606| 5209492903 | phuser | 10.64.32.150:28944 | phabricator_file | Sleep | 1 |
607| 5209492906 | phuser | 10.64.32.150:28947 | phabricator_project | Sleep | 1 |
608| 5209492907 | phuser | 10.64.32.150:28948 | phabricator_config | Sleep | 1 |
609| 5209492908 | phuser | 10.64.32.150:28949 | phabricator_config | Sleep | 1 |
610| 5209492909 | phuser | 10.64.32.150:28950 | phabricator_policy | Sleep | 1 |
611| 5209492910 | phuser | 10.64.32.150:28951 | phabricator_policy | Sleep | 1 |
612| 5209492911 | phuser | 10.64.32.150:28952 | phabricator_spaces | Sleep | 1 |
613| 5209492912 | phuser | 10.64.32.150:28953 | phabricator_config | Sleep | 1 |
614| 5209492913 | phuser | 10.64.32.150:28954 | phabricator_spaces | Sleep | 1 |
615| 5209492914 | phuser | 10.64.32.150:28955 | phabricator_project | Sleep | 1 |
616| 5209492915 | phuser | 10.64.32.150:28956 | phabricator_repository | Sleep | 1 |
617| 5209492917 | phuser | 10.64.32.150:28958 | phabricator_repository | Sleep | 1 |
618| 5209492918 | phuser | 10.64.32.150:28959 | phabricator_project | Sleep | 1 |
619| 5209492919 | phuser | 10.64.32.150:28960 | phabricator_project | Sleep | 1 |
620| 5209492920 | phuser | 10.64.32.150:28961 | phabricator_project | Sleep | 0 |
621| 5209492921 | phuser | 10.64.32.150:28962 | phabricator_project | Sleep | 1 |
622| 5209492923 | phuser | 10.64.32.150:28964 | phabricator_spaces | Sleep | 1 |
623| 5209492926 | phuser | 10.64.32.150:28967 | phabricator_spaces | Sleep | 1 |
624| 5209492927 | phuser | 10.64.32.150:28968 | phabricator_repository | Sleep | 0 |
625| 5209492928 | phuser | 10.64.32.150:28969 | phabricator_file | Sleep | 1 |
626| 5209492929 | phuser | 10.64.32.150:28970 | phabricator_project | Sleep | 1 |
627| 5209492930 | phuser | 10.64.32.150:28971 | phabricator_policy | Sleep | 1 |
628| 5209492933 | phuser | 10.64.32.150:28974 | phabricator_policy | Sleep | 1 |
629| 5209492934 | phuser | 10.64.32.150:28975 | phabricator_policy | Sleep | 1 |
630| 5209492935 | phuser | 10.64.32.150:28976 | phabricator_config | Sleep | 1 |
631| 5209492936 | phuser | 10.64.32.150:28978 | phabricator_file | Sleep | 1 |
632| 5209492937 | phuser | 10.64.32.150:28979 | phabricator_differential | Sleep | 1 |
633| 5209492939 | phuser | 10.64.32.150:28981 | phabricator_maniphest | Sleep | 0 |
634| 5209492940 | phuser | 10.64.32.150:28982 | phabricator_project | Sleep | 1 |
635| 5209492942 | phuser | 10.64.32.150:28984 | phabricator_policy | Sleep | 1 |
636| 5209492943 | phuser | 10.64.32.150:28985 | phabricator_spaces | Sleep | 1 |
637| 5209492944 | phuser | 10.64.32.150:28986 | phabricator_policy | Sleep | 1 |
638| 5209492947 | phuser | 10.64.32.150:28989 | phabricator_repository | Sleep | 0 |
639| 5209492948 | phuser | 10.64.32.150:28990 | phabricator_repository | Sleep | 1 |
640| 5209492949 | phuser | 10.64.32.150:28991 | phabricator_project | Sleep | 1 |
641| 5209492950 | phuser | 10.64.32.150:28992 | phabricator_repository | Sleep | 1 |
642| 5209492951 | phuser | 10.64.32.150:28993 | phabricator_policy | Sleep | 1 |
643| 5209492952 | phuser | 10.64.32.150:28995 | phabricator_policy | Sleep | 1 |
644| 5209492953 | phuser | 10.64.32.150:28994 | phabricator_repository | Sleep | 1 |
645| 5209492954 | phuser | 10.64.32.150:28997 | phabricator_policy | Sleep | 1 |
646| 5209492955 | phuser | 10.64.32.150:28998 | phabricator_spaces | Sleep | 1 |
647| 5209492956 | phuser | 10.64.32.150:28999 | phabricator_file | Sleep | 1 |
648| 5209492957 | phuser | 10.64.32.150:29001 | phabricator_project | Sleep | 1 |
649| 5209492958 | phuser | 10.64.32.150:29000 | phabricator_repository | Sleep | 1 |
650| 5209492959 | phuser | 10.64.32.150:29003 | phabricator_differential | Sleep | 1 |
651| 5209492962 | phuser | 10.64.32.150:29006 | phabricator_spaces | Sleep | 1 |
652| 5209492963 | phuser | 10.64.32.150:29007 | phabricator_repository | Sleep | 0 |
653| 5209492964 | phuser | 10.64.32.150:29008 | phabricator_project | Sleep | 1 |
654| 5209492965 | phuser | 10.64.32.150:29009 | phabricator_project | Sleep | 1 |
655| 5209492966 | phuser | 10.64.32.150:29010 | phabricator_policy | Sleep | 1 |
656| 5209492967 | phuser | 10.64.32.150:29011 | phabricator_repository | Sleep | 1 |
657| 5209492969 | phuser | 10.64.32.150:29013 | phabricator_config | Sleep | 1 |
658| 5209492972 | phuser | 10.64.32.150:29016 | phabricator_config | Sleep | 1 |
659| 5209492973 | phuser | 10.64.32.150:29017 | phabricator_config | Sleep | 1 |
660| 5209492974 | phuser | 10.64.32.150:29018 | phabricator_differential | Sleep | 1 |
661| 5209492975 | phuser | 10.64.32.150:29020 | phabricator_project | Sleep | 1 |
662| 5209492976 | phuser | 10.64.32.150:29021 | phabricator_config | Sleep | 1 |
663| 5209492977 | phuser | 10.64.32.150:29022 | phabricator_spaces | Sleep | 1 |
664| 5209492978 | phuser | 10.64.32.150:29023 | phabricator_project | Sleep | 1 |
665| 5209492979 | phuser | 10.64.32.150:29024 | phabricator_project | Sleep | 1 |
666| 5209492980 | phuser | 10.64.32.150:29025 | phabricator_policy | Sleep | 1 |
667| 5209492981 | phuser | 10.64.32.150:29026 | phabricator_policy | Sleep | 1 |
668| 5209492982 | phuser | 10.64.32.150:29027 | phabricator_config | Sleep | 1 |
669| 5209492983 | phuser | 10.64.32.150:29028 | phabricator_repository | Sleep | 1 |
670| 5209492984 | phuser | 10.64.32.150:29029 | phabricator_spaces | Sleep | 1 |
671| 5209492985 | phuser | 10.64.32.150:29030 | phabricator_policy | Sleep | 1 |
672| 5209492986 | phuser | 10.64.32.150:29031 | phabricator_policy | Sleep | 1 |
673| 5209492987 | phuser | 10.64.32.150:29032 | phabricator_project | Sleep | 1 |
674| 5209492988 | phuser | 10.64.32.150:29033 | phabricator_repository | Sleep | 0 |
675| 5209492989 | phuser | 10.64.32.150:29034 | phabricator_spaces | Sleep | 1 |
676| 5209492990 | phuser | 10.64.32.150:29035 | phabricator_user | Sleep | 1 |
677| 5209492991 | phuser | 10.64.32.150:29036 | phabricator_spaces | Sleep | 1 |
678| 5209492992 | phuser | 10.64.32.150:29037 | phabricator_file | Sleep | 1 |
679| 5209492993 | phuser | 10.64.32.150:29038 | phabricator_user | Sleep | 1 |
680| 5209492994 | phuser | 10.64.32.150:29040 | phabricator_repository | Sleep | 0 |
681| 5209492995 | phuser | 10.64.32.150:29039 | phabricator_repository | Sleep | 1 |
682| 5209492996 | phuser | 10.64.32.150:29041 | phabricator_config | Sleep | 1 |
683| 5209492997 | phuser | 10.64.32.150:29042 | phabricator_project | Sleep | 1 |
684| 5209492998 | phuser | 10.64.32.150:29043 | phabricator_spaces | Sleep | 1 |
685| 5209492999 | phuser | 10.64.32.150:29044 | phabricator_file | Sleep | 1 |
686| 5209493000 | phuser | 10.64.32.150:29045 | phabricator_policy | Sleep | 1 |
687| 5209493001 | phuser | 10.64.32.150:29046 | phabricator_spaces | Sleep | 1 |
688| 5209493002 | phuser | 10.64.32.150:29047 | phabricator_repository | Sleep | 1 |
689| 5209493003 | phuser | 10.64.32.150:29048 | phabricator_policy | Sleep | 1 |
690| 5209493004 | phuser | 10.64.32.150:29049 | phabricator_repository | Sleep | 1 |
691| 5209493005 | phuser | 10.64.32.150:29050 | phabricator_repository | Sleep | 1 |
692| 5209493006 | phuser | 10.64.32.150:29051 | phabricator_policy | Sleep | 1 |
693| 5209493007 | phuser | 10.64.32.150:29052 | phabricator_project | Sleep | 1 |
694| 5209493008 | phuser | 10.64.32.150:29053 | phabricator_project | Sleep | 1 |
695| 5209493009 | phuser | 10.64.32.150:29054 | phabricator_repository | Sleep | 1 |
696| 5209493010 | phuser | 10.64.32.150:29055 | phabricator_policy | Sleep | 1 |
697| 5209493011 | phuser | 10.64.32.150:29056 | phabricator_project | Sleep | 1 |
698| 5209493012 | phuser | 10.64.32.150:29057 | phabricator_policy | Sleep | 1 |
699| 5209493013 | phuser | 10.64.32.150:29059 | phabricator_repository | Sleep | 0 |
700| 5209493014 | phuser | 10.64.32.150:29060 | phabricator_repository | Sleep | 1 |
701| 5209493015 | phuser | 10.64.32.150:29061 | phabricator_config | Sleep | 1 |
702| 5209493016 | phuser | 10.64.32.150:29062 | phabricator_file | Sleep | 1 |
703| 5209493017 | phuser | 10.64.32.150:29063 | phabricator_policy | Sleep | 1 |
704| 5209493018 | phuser | 10.64.32.150:29064 | phabricator_config | Sleep | 1 |
705| 5209493019 | phuser | 10.64.32.150:29065 | phabricator_spaces | Sleep | 1 |
706| 5209493020 | phuser | 10.64.32.150:29066 | phabricator_policy | Sleep | 1 |
707| 5209493021 | phuser | 10.64.32.150:29067 | phabricator_policy | Sleep | 1 |
708| 5209493022 | phuser | 10.64.32.150:29068 | phabricator_repository | Sleep | 1 |
709| 5209493023 | phuser | 10.64.32.150:29069 | phabricator_repository | Sleep | 0 |
710| 5209493024 | phuser | 10.64.32.150:29070 | phabricator_repository | Sleep | 0 |
711| 5209493025 | phuser | 10.64.32.150:29071 | phabricator_spaces | Sleep | 1 |
712| 5209493026 | phuser | 10.64.32.150:29072 | phabricator_repository | Sleep | 0 |
713| 5209493027 | phuser | 10.64.32.150:29073 | phabricator_project | Sleep | 1 |
714| 5209493028 | phuser | 10.64.32.150:29074 | phabricator_repository | Sleep | 1 |
715| 5209493029 | phuser | 10.64.32.150:29075 | phabricator_project | Sleep | 0 |
716| 5209493030 | phuser | 10.64.32.150:29076 | phabricator_spaces | Sleep | 1 |
717| 5209493031 | phuser | 10.64.32.150:29077 | phabricator_spaces | Sleep | 1 |
718| 5209493032 | phuser | 10.64.32.150:29078 | phabricator_policy | Sleep | 1 |
719| 5209493033 | phuser | 10.64.32.150:29079 | phabricator_project | Sleep | 1 |
720| 5209493034 | phuser | 10.64.32.150:29080 | phabricator_project | Sleep | 1 |
721| 5209493035 | phuser | 10.64.32.150:29081 | phabricator_repository | Sleep | 0 |
722| 5209493037 | phuser | 10.64.32.150:29083 | phabricator_policy | Sleep | 1 |
723| 5209493038 | phuser | 10.64.32.150:29084 | phabricator_policy | Sleep | 1 |
724| 5209493039 | phuser | 10.64.32.150:29085 | phabricator_policy | Sleep | 1 |
725| 5209493040 | phuser | 10.64.32.150:29086 | phabricator_policy | Sleep | 1 |
726| 5209493041 | phuser | 10.64.32.150:29087 | phabricator_differential | Sleep | 1 |
727| 5209493042 | phuser | 10.64.32.150:29088 | phabricator_project | Sleep | 0 |
728| 5209493043 | phuser | 10.64.32.150:29090 | phabricator_repository | Sleep | 0 |
729| 5209493044 | phuser | 10.64.32.150:29091 | phabricator_differential | Sleep | 1 |
730| 5209493045 | phuser | 10.64.32.150:29092 | phabricator_cache | Sleep | 1 |
731| 5209493046 | phuser | 10.64.32.150:29093 | phabricator_repository | Sleep | 0 |
732| 5209493047 | phuser | 10.64.32.150:29094 | phabricator_repository | Sleep | 0 |
733| 5209493048 | phuser | 10.64.32.150:29095 | phabricator_repository | Sleep | 0 |
734| 5209493049 | phuser | 10.64.32.150:29096 | phabricator_policy | Sleep | 1 |
735| 5209493050 | phuser | 10.64.32.150:29097 | phabricator_config | Sleep | 1 |
736| 5209493051 | phuser | 10.64.32.150:29098 | phabricator_differential | Sleep | 0 |
737| 5209493052 | phuser | 10.64.32.150:29099 | phabricator_differential | Sleep | 0 |
738| 5209493053 | phuser | 10.64.32.150:29100 | phabricator_repository | Sleep | 0 |
739| 5209493054 | phuser | 10.64.32.150:29101 | phabricator_repository | Sleep | 0 |
740| 5209493055 | phuser | 10.64.32.150:29102 | phabricator_draft | Sleep | 0 |
741| 5209493056 | phuser | 10.64.32.150:29103 | phabricator_repository | Sleep | 0 |
742| 5209493057 | phuser | 10.64.32.150:29104 | phabricator_project | Sleep | 0 |
743| 5209493058 | phuser | 10.64.32.150:29105 | phabricator_file | Sleep | 0 |
744| 5209493059 | phuser | 10.64.32.150:29106 | phabricator_file | Sleep | 0 |
745| 5209493060 | phuser | 10.64.32.150:29107 | phabricator_policy | Sleep | 0 |
746| 5209493061 | phuser | 10.64.32.150:29108 | phabricator_repository | Sleep | 0 |
747| 5209493062 | phuser | 10.64.32.150:29109 | phabricator_file | Sleep | 0 |
748| 5209493063 | phuser | 10.64.32.150:29110 | phabricator_repository | Sleep | 0 |
749| 5209493064 | phuser | 10.64.32.150:29111 | phabricator_spaces | Sleep | 0 |
750| 5209493065 | phuser | 10.64.32.150:29112 | phabricator_search | Sleep | 0 |
751| 5209493066 | phuser | 10.64.32.150:29113 | phabricator_search | Sleep | 0 |
752| 5209493067 | phuser | 10.64.32.150:29114 | phabricator_search | Sleep | 0 |
753| 5209493069 | phuser | 10.64.32.150:29116 | phabricator_file | Sleep | 0 |
754| 5209493070 | phuser | 10.64.32.150:29117 | phabricator_search | Sleep | 0 |
755| 5209493071 | phuser | 10.64.32.150:29118 | phabricator_differential | Sleep | 0 |
756| 5209493072 | phuser | 10.64.32.150:29119 | phabricator_differential | Sleep | 0 |
757| 5209493073 | phuser | 10.64.32.150:29120 | phabricator_file | Sleep | 0 |
758| 5209493074 | phuser | 10.64.32.150:29121 | phabricator_differential | Sleep | 0 |
759| 5209493075 | phuser | 10.64.32.150:29122 | phabricator_config | Sleep | 0 |
760| 5209493076 | phuser | 10.64.32.150:29123 | phabricator_differential | Sleep | 0 |
761| 5209493077 | phuser | 10.64.32.150:29124 | phabricator_file | Sleep | 0 |
762| 5209493078 | phuser | 10.64.32.150:29125 | phabricator_file | Sleep | 0 |
763| 5209493079 | phuser | 10.64.32.150:29126 | phabricator_repository | Sleep | 0 |
764| 5209493080 | phuser | 10.64.32.150:29127 | phabricator_repository | Sleep | 0 |
765| 5209493081 | phuser | 10.64.32.150:29128 | phabricator_repository | Sleep | 0 |
766| 5209493082 | phuser | 10.64.32.150:29129 | phabricator_differential | Sleep | 0 |
767+------------+-----------------+--------------------+--------------------------+-------------+----------+--------------------------------------------
768770 rows in set (0.01 sec)

jcrespo renamed this task from Understand phabricator scalability to Understand phabricator scalability (a.k.a. avoiding Phabricator's 503s).Dec 16 2015, 7:37 PM
jcrespo set Security to None.
jcrespo added a subscriber: StudiesWorld.
demon added a subscriber: demon.Dec 16 2015, 7:51 PM

As we are migrating repositories from gerrit to phabricator, a significant amount of load will be placed on phabricator by requests to git service. Both CPU and Disk I/O could become bottlenecks which could potentially slow developer activity and limit CI testing throughput. The upstream phabricator project seems to have a plan for addressing scalability of SCM hosting, which primarily revolves around implementing a smart protocol proxy in phabricator which can delegate the work to multiple back-end git servers. With repositories distributed across several git servers, performance and scaling should not be a problem. This is, however, currently unimplemented and it could be a while before the upstream gets this working.

In addition to scaling repositories over multiple backends (which needs upstream and we want, as you point out), we should also be more aggressively tuning the underlying git gc operations so they run frequently on large repositories and at least occasionally on smaller ones. This helps disk I/O tremendously. This we can do now as well as when upstream scales this out.

jcrespo lowered the priority of this task from High to Normal.Dec 24 2015, 3:53 PM

This has not happened in the last weeks, back to normal priority, as a long term task, as it is not causing currently mysql connection spikes.

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptDec 24 2015, 3:53 PM

I think the root cause of connection spikes is usually misbehaving search indexers, either hitting pages that they shouldn't (excluded in robots.txt) or hitting us too fast (not respecting the index delay in robots.txt)

I'm not exactly sure what we can do other than manually blocking these on a case by case basis.

upstream has recently added support for directing read queries to a read-only mysql slave, as well as failing over to a backup mysql master. That should go a long way to making phabricator more scalable, at least at the db layer.

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 22 2016, 5:02 AM
mmodell renamed this task from Understand phabricator scalability (a.k.a. avoiding Phabricator's 503s) to Implement phabricator database clustering support.Apr 22 2016, 5:09 AM
mmodell updated the task description. (Show Details)

Phabricator was updated today. Which added support for this.

I've submitted a change to operations/puppet which will enable the phabricator cluster features: https://gerrit.wikimedia.org/r/#/c/300494/

This need good planing to coordinate with our proxy setup and dns configuration to see what happens in case slave or master fails.

I'm not sure how to test it thoroughly because I don't have a replicated db setup to test against. Simply configuring it should be safe, according to the documentation it should behave sanely and transparently with the config change I just submitted.

mmodell updated the task description. (Show Details)Jul 22 2016, 6:21 AM
mmodell updated the task description. (Show Details)

@jcrespo: According to monitoring replicas we should grant REPLICATION CLIENT privileges to the phabricator user so that phabricator can keep track of replication status. Should I file a task to request the additional sql GRANT?

It is ok, but let's do it with an m3-slave dns entry, which I think it is not there.

It seems it is, although not necesarily pointing to the right place.

@mmodell (and/or anyone related to/interested in phab architecture): let's meet- I am all for supporting any clustering efforts and active-active setup, but we already had existing clustering transparent to the application which we need to handle (either to disable it or to change its loginc). We need to coordinate to set the goals and the failover logic. Maybe the proxy is not longer needed; maybe it is but we need to clarify how? We also have 2 datacenters- we should clarify how we intend to use them.

With the many database nodes; and now "clients" supporting phabricator, we should have clear what is the best way to go. This should be easy, we just need to be on the same page.

mmodell added a comment.EditedAug 17 2016, 3:23 PM

Phabricator does not do any active cluster failover - it just monitors the slave status so as described in https://secure.phabricator.com/book/phabricator/article/cluster_databases/ - that article simply describes how to use mysql replication which you are surely more familiar with than I am, but I'm pretty sure it's standard mysql stuff - not really phabricator specific.

@jcrespo: It's not essential for REPLICATION CLIENT to be granted, I believe it will still work without it. The main thing phabricator is able to do is degrade into read-only mode when the master is unavailable.

Which is exactly why I want to talk to you, to answer things such as phab2001 role and would it work if master and slave were the same server? Ping me on IRC when you have the time.

There is no issue with the request you mentioned, I just want to understand the best way to proceed with it.

To give you an idea- if the master fails, we currently failover to the slave (but the slave will not have replication working, then, or if it does, it will be meaningless). Maybe it is not worth having a proxy anymore? Hit me on IRC or we can discuss options by email, whatever is more confortable for you. I want to explain you the options and how to combine them with phabricator clustering features.

@jcrespo: Thanks for further explanation. I'll hit you up on IRC.

Dzahn added a comment.Aug 18 2016, 8:53 PM

fwiw "phab1001.eqiad.wmnet" exists now as a CNAME for iridium as suggested by Jaime

mmodell changed the task status from Open to Stalled.Apr 12 2017, 6:13 PM
mmodell lowered the priority of this task from Normal to Low.

Database clustering is already handled by haproxy. There may be some advantages to the built in clustering support, however, it's not very urgent or high priority.

greg removed mmodell as the assignee of this task.May 17 2017, 2:02 PM
mmodell closed this task as Declined.Fri, Sep 13, 7:08 PM

Database clustering is already handled by haproxy. There may be some advantages to the built in clustering support, however, it's not very urgent or high priority.

Advantages unknown, disadvantages take precedence.