Update jq to v1.4.0 or higher
Closed, DeclinedPublic

Description

We currently run jq 1.3 which has a bug in it that is causing most non-ascii characters to cause a fatal error mid-way the stream and leave the process with invalid json in its standard output.

I'm encountering this on stat1002 when dealing with output from kafkacat. It's valid json, but contains some kind of special character in one of the strings. Passing it through jq to perform some filtering and aggregation is crashing mid-way.

I narrowed it down to the following line. The issue seems similar to https://github.com/stedolan/jq/issues/352, which was fixed in jq 1.4 (June 2014).

{"..":"..", "user_agent":"Mozilla/5.0 (Linux; U; Android 2.3.6; es-co; XT550 Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Versi�n/4.0 Mobile Safari/533.1"}

Latest version is jq 1.5 (released August 2015). We currently run jq 1.3 (released May 2013).

Related Objects

StatusAssignedTask
DeclinedOttomata
ResolvedOttomata
ResolvedRobH
ResolvedOttomata
DeclinedNone
ResolvedRobH
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
Resolvedmpopov
Resolvedmforns
ResolvedOttomata
ResolvedOttomata
ResolvedAddshore
ResolvedGoranSMilovanovic
Resolvedelukey
DeclinedOttomata
ResolvedOttomata
ResolvedErik_Zachte
ResolvedCmjohnson
ResolvedCmjohnson
Krinkle created this task.Mar 2 2017, 12:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2017, 12:33 AM
Krinkle moved this task from Inbox to Radar on the Performance-Team board.Mar 2 2017, 1:01 AM
Nuria moved this task from Incoming to Wikistats Production on the Analytics board.Mar 2 2017, 5:06 PM
Ottomata triaged this task as Low priority.Mar 6 2017, 6:55 PM
Ottomata claimed this task.
Ottomata added a subscriber: Ottomata.

I'll take this on, low priority though. Remind me about it if you get fidgety! :)

jessie has jq 1.4, so this would also be fixed once stat1002 is migrated to jessie.

@MoritzMuehlenhoff Thanks. Is there a ticket for that?

I've transferred my data to terbium for post-processing for the time being because the python/ua-parser package on stat1002 isn't installed properly (importing the parser class results in a fatal error saying regexes.yaml doesn't exist). The same package however is installed properly on terbium.

$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ua_parser import user_agent_parser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/ua_parser/user_agent_parser.py", line 431, in <module>
    yamlFile = open(yamlPath)
IOError: [Errno 2] No such file or directory: '/usr/lib/python2.7/dist-packages/ua_parser/regexes.yaml'

Coincidentally, terbium also has Trusty and jq 1.4.1.

Oh prefect!

We will soon be replacing stat1002 (T159838) and stat1003 (T159839) with newer hardware. When we do so, we will upgrade these to Jessie.

Nuria added a subscriber: Nuria.Jun 5 2017, 4:26 PM

We are moving to new stats boxes which we will not have this problem.

Nuria closed this task as Declined.Jun 5 2017, 4:26 PM