This metric would perfectly complement the REST equivalent in https://grafana.wikimedia.org/dashboard/db/api-summary?orgId=1, and as a result give us direct information on overall API use.

Check in which webrequext_source partition data lives (and in the mean time check for api/rest too):

SELECT
  webrequest_source,
  (uri_path LIKE '/api/rest_v1/%') AS rest,
  (uri_path LIKE '/w/api.php%') AS action,
  COUNT(1) as c
FROM wmf.webrequest
WHERE year = 2017
  AND month = 11
  AND day = 20
  AND hour = 17
GROUP BY webrequest_source,
  (uri_path LIKE '/api/rest_v1/%'),
  (uri_path LIKE '/w/api.php%')
ORDER BY
  webrequest_source, rest, action
LIMIT 1000;

webrequest_source	rest	action	c
misc	false	false	799279
misc	false	true	3
text	false	false	201987329
text	false	true	33248371
text	true	false	41854561
upload	false	false	209598372
upload	false	true	109
upload	true	false	5

As expected, we should only consider text partition.

Ping @mobrovac, @Pchelolo, @Gilles , @Anomie, @Krinkle for naming confirmation:
The existing metric is named restbase.requests.varnish_requests, and we plan to go with MediaWiki.api.varnish_request for the new one, reusing the existing graphite domain MediaWiki.api.
Thanks all.

In T176785#3777250, @JAllemandou wrote:

The existing metric is named restbase.requests.varnish_requests, and we plan to go with MediaWiki.api.varnish_request for the new one, reusing the existing graphite domain MediaWiki.api.

s/varnish_request/varnish_requests/ but otherwise the proposal looks sensible to me, +1.

Change 392700 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Grow RestbaseMetrics spark job to count MW API

https://gerrit.wikimedia.org/r/392700

Change 392703 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Change restbase job to also count MW-API requests

https://gerrit.wikimedia.org/r/392703

In T176785#3777250, @JAllemandou wrote:

The existing metric is named restbase.requests.varnish_requests, and we plan to go with MediaWiki.api.varnish_request for the new one, reusing the existing graphite domain MediaWiki.api.

I'd recommend against this. The MediaWiki. prefix is prepended by MediaWiki core to all its statsd messages. I would not expect it to be used by other metric writers. Doing so would make it more likely to cause conflicts, and would also make it more difficult to find or document the source of certain metrics. Perhaps use a prefix relating to the source of the data (e.g. Spark, Oozie, Refinery, or some such).

@Krinkle: I'm happy to use a less functional approach, and have for instance analytics.varnish_requests.restbase and analytics.varnish_requests.mw_api.
Only concern is that it changes the existing metric for restbase. @mobrovac and @Pchelolo, is that a big deal?

Only concern is that it changes the existing metric for restbase. @mobrovac and @Pchelolo, is that a big deal?

That will not be desirable as that would break the existing graphs. We're using the rest base varnish metric in a couple of places for a long time

@Krinkle and @Addshore: After looking once more in graphite, would mw.api.varnish_requests be ok for you guys? This would allow us to keep the restbase metric and have the same naming scheme for both.
If you prefer, we can also rename both, but we'll need to convince @Pchelolo .

As far as I am aware, for those with access to the graphite box it is fairly trivial to move and merge metrics.

"analytics.varnish_requests.restbase and analytics.varnish_requests.mw_api" look the best to me.

In T176785#3780948, @Pchelolo wrote:

That will not be desirable as that would break the existing graphs. We're using the rest base varnish metric in a couple of places for a long time

I don't imagine it would be too hard to update these graphs.
If it is I would vote for leaving the restbase stuff where it is for now but having the mw_api stuff @ "analytics.varnish_requests.mw_api" and keeping any new stuff there.

I'm a bit confused here. Is this task about adding a count for the action API, turning an existing restbase-only count into a restbase+action API count, or adding a restbase+action API count in addition to an existing restbase-only count?

In T176785#3781893, @Anomie wrote:

I'm a bit confused here. Is this task about adding a count for the action API, turning an existing restbase-only count into a restbase+action API count, or adding a restbase+action API count in addition to an existing restbase-only count?

I think the idea is to add a count for requests to action API on varnish level. So in the end we will have 2 separate metrics - one for action API and one for rest API.

@Pchelolo is correct, idea is to have both restbase and mw-action-api hourly varnish-requests counts in graphite.

Ping ! If no other message comes in before tomorrow, I'll move forward with @Addshore suggestion of keeping existing restbase metric and push the new one in analytics.varnish_requests.mw_api (@Ottomata - if you have a better name, please suggest :)

Last patch uses analytics.mw_api.varnish_request as matric name for action-api.
Thanks @Krinkle for the reivew :)

I don't love it, but I don't have a better suggestion. Proceed!

Change 392703 merged by Joal:
[analytics/refinery@master] Change restbase job to also count mw_api requests

https://gerrit.wikimedia.org/r/392703

Change 392700 merged by jenkins-bot:
[analytics/refinery/source@master] Grow RestbaseMetrics spark job to count MW API

https://gerrit.wikimedia.org/r/392700

I guess that has been done since I was able to add the Action API graph to the API summary dashboard: https://grafana.wikimedia.org/dashboard/db/api-summary?panelId=1&fullscreen&orgId=1

Should we resolve the ticket?

@Pchelolo: It has indeed happen.
The tak has been moved to done on our kanban, we'll resolve it after we finalize the discussion :)
Thanks !