Page MenuHomePhabricator

Turnilo: per-second rates for wmf_netflow bytes + packets
Closed, ResolvedPublic

Description

When we're interpreting netflow data, we generally want packets-per-second and bits-per-second (bytes-per-second would also be acceptable but not ideal).

But Turnilo wants to sum up each within a given split window (1 minute / 5 minutes / 1 hour etc), so if you need absolute values or to cross-check with other sources of data, constant unit conversion is necessary.

I did file a FR upstream about this and was informed of the $MillisecondsInInterval config knob, but hadn't gotten around for asking to try it out on wmf_netflow until now :)

Event Timeline

ArielGlenn triaged this task as Medium priority.Sep 28 2020, 9:54 AM

@CDanis I added the following config to turnilo on an-tool1005 (staging instance):

measures:
   - name: bytes
     title: Bytes
     formula: $main.sum($bytes)
   - name: packets
     title: Packets
     formula: $main.sum($packets)
   - name: bytes_second
     title: Bytes / second
     formula: $main.sum($bytes) / ($MillisecondsInInterval / 1000)
   - name: packets_second
     title: Packets / second
     formula: $main.sum($packets) / ($MillisecondsInInterval / 1000)

Does it make sense? If so it is very easy to change Turnilo's config, otherwise let me know what you'd expect to see :)

ssh -L 9091:localhost:9091 an-tool1005.eqiad.wmnet

Yay, that's awesome! You can't imagine how much time this would save!

I changed the config a little bit. Specifically:

  • Bits per second is more useful than Bytes per second - typically how you measure network interfaces
  • Use "bps" and "pps" as names of those dimensions, as that's what they're typically called
  • Added a default selected measure. I don't have strong opinions on which one(s) we should be defaulting to, but having all four (bytes, packets, bps, pps) listed was a bit too much.

Config:

defaultSelectedMeasures: [bps, pps]
measures:
   - name: bytes
     title: Bytes
     formula: $main.sum($bytes)
   - name: packets
     title: Packets
     formula: $main.sum($packets)
   - name: bps
     title: bps
     formula: $main.sum($bytes)*8 / ($MillisecondsInInterval / 1000)
   - name: pps
     title: pps
     formula: $main.sum($packets) / ($MillisecondsInInterval / 1000)

Change 634931 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] turnilo: add bps/pps measure to the wmf_netflow datasource

https://gerrit.wikimedia.org/r/634931

Change 634931 merged by Elukey:
[operations/puppet@production] turnilo: add bps/pps measure to the wmf_netflow datasource

https://gerrit.wikimedia.org/r/634931

elukey added a project: Analytics-Kanban.
elukey moved this task from Next Up to Done on the Analytics-Kanban board.