Page MenuHomePhabricator

Deploy new media-list endpoint in RESTBase
Closed, ResolvedPublic

Description

The new media-list endpoint is a variation on the existing media endpoint that contains only media info derived from the page HTML. It is intended to avoid the performance and invalidation problems posed by the existing media endpoint.

Proposed path: /page/media-list/{title}{/revision}

I'll add some local benchmarking numbers below.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 19 2019, 2:01 PM

Local ab numbers when fetching page HTML over HTTP from the public RESTBase endpoint:

mholloway@mholloway:~/code/wikimedia/mobileapps$ ab -c 1 -n 1000 http://localhost:6927/en.wikipedia.org/v1/page/media-list/San_Francisco
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            6927

Document Path:          /en.wikipedia.org/v1/page/media-list/San_Francisco
Document Length:        18015 bytes

Concurrency Level:      1
Time taken for tests:   384.439 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      18934000 bytes
HTML transferred:       18015000 bytes
Requests per second:    2.60 [#/sec] (mean)
Time per request:       384.439 [ms] (mean)
Time per request:       384.439 [ms] (mean, across all concurrent requests)
Transfer rate:          48.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   328  384  27.3    390     522
Waiting:      328  384  27.3    390     522
Total:        328  384  27.3    390     522

Percentage of the requests served within a certain time (ms)
  50%    390
  66%    401
  75%    404
  80%    406
  90%    412
  95%    424
  98%    448
  99%    466
 100%    522 (longest request)

And when grabbing stored page HTML from a local RESTBase:

mholloway@mholloway:~/code/wikimedia/restbase$ ab -c 1 -n 1000 http://localhost:6927/en.wikipedia.org/v1/page/media-list/San_Francisco
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            6927

Document Path:          /en.wikipedia.org/v1/page/media-list/San_Francisco
Document Length:        18015 bytes

Concurrency Level:      1
Time taken for tests:   245.354 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      18925000 bytes
HTML transferred:       18015000 bytes
Requests per second:    4.08 [#/sec] (mean)
Time per request:       245.354 [ms] (mean)
Time per request:       245.354 [ms] (mean, across all concurrent requests)
Transfer rate:          75.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   181  245  41.1    244     844
Waiting:      181  245  41.1    244     844
Total:        181  245  41.1    244     845

Percentage of the requests served within a certain time (ms)
  50%    244
  66%    259
  75%    267
  80%    271
  90%    286
  95%    309
  98%    340
  99%    371
 100%    845 (longest request)

Blocking this on confirmation that it will be used. T225443#5274233

Mentioned in SAL (#wikimedia-operations) [2019-06-24T21:04:07Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@a915f69]: Add /page/media-lint - T226105 - and various other cleanups

Mentioned in SAL (#wikimedia-operations) [2019-06-24T21:23:14Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@a915f69]: Add /page/media-lint - T226105 - and various other cleanups (duration: 19m 08s)

mobrovac closed this task as Resolved.Jun 24 2019, 9:27 PM
mobrovac claimed this task.
mobrovac added a subscriber: mobrovac.

PR merged and deployed, it's now live in production.