Update jq to v1.4.0 or higher
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	Krinkle
	Mar 2 2017, 12:33 AM

Description

We currently run jq 1.3 which has a bug in it that is causing most non-ascii characters to cause a fatal error mid-way the stream and leave the process with invalid json in its standard output.

I'm encountering this on stat1002 when dealing with output from kafkacat. It's valid json, but contains some kind of special character in one of the strings. Passing it through jq to perform some filtering and aggregation is crashing mid-way.

I narrowed it down to the following line. The issue seems similar to https://github.com/stedolan/jq/issues/352, which was fixed in jq 1.4 (June 2014).

{"..":"..", "user_agent":"Mozilla/5.0 (Linux; U; Android 2.3.6; es-co; XT550 Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Versi�n/4.0 Mobile Safari/533.1"}

Latest version is jq 1.5 (released August 2015). We currently run jq 1.3 (released May 2013).

Related Objects
Search...

Status	Assigned	Task
Declined	Ottomata	T159392 Update jq to v1.4.0 or higher
Resolved	Ottomata	T152712 Replacement of stat1002 and stat1003
Resolved	RobH	T159838 EQIAD: stat1002 replacement
		Unknown Object (Task)
Resolved	Ottomata	T165368 rack/setup/install replacement to stat1005 (stat1002 replacement)
Declined	None	T151904 User limits for stat machines. Limit space on /home dir and possibly /tmp
Resolved	RobH	T159839 EQIAD: stat1003 replacement
		Unknown Object (Task)
Resolved	Ottomata	T165366 rack/setup/install replacement stat1006 (stat1003 replacement)
Resolved	Ottomata	T168683 Upgrade pandoc package to at least 1.12.3
Resolved	Ottomata	T170471 Move statistics::discovery jobs from stat1002 -> stat1005
Resolved	mpopov	T170494 [EPIC] Reconfigure Discovery-Stats on Analytics Cluster
Resolved	mforns	T173333 Reportupdater outputs files with restricted permissions
Resolved	Ottomata	T174110 Private data access for non-person user that calculates metrics
Resolved	Ottomata	T174465 Puppet admin module should support adding system users to managed groups
Resolved	Addshore	T170472 Move statistics::wmde jobs from stat1002 -> stat1005
Resolved	GoranSMilovanovic	T170664 Migrate WMDE campaign and WDCM analytics from stat1002, 1003 to stat1005, 1006
Resolved	elukey	T170878 Audit users and account expiry dates for stat boxes
Declined	Ottomata	T171072 Install Java 7 in Debian Stretch (to fix sqoop (and possibly other things) on stat1005)
Resolved	Ottomata	T171373 Unmount /a partition from 1002
Resolved	Erik_Zachte	T172032 Pagecounts-ez not generating
Resolved	• Cmjohnson	T173097 Decommission stat1002.eqiad.wmnet
Resolved	• Cmjohnson	T175150 Decommission stat1003.eqiad.wmnet

Event Timeline

Krinkle created this task.Mar 2 2017, 12:33 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2017, 12:33 AM

Krinkle moved this task from Inbox, needs triage to Radar on the Performance-Team board.Mar 2 2017, 1:01 AM

• Nuria moved this task from Incoming to Wikistats on the Analytics board.Mar 2 2017, 5:06 PM

I'll take this on, low priority though. Remind me about it if you get fidgety! :)

jessie has jq 1.4, so this would also be fixed once stat1002 is migrated to jessie.

@MoritzMuehlenhoff Thanks. Is there a ticket for that?

I've transferred my data to terbium for post-processing for the time being because the python/ua-parser package on stat1002 isn't installed properly (importing the parser class results in a fatal error saying regexes.yaml doesn't exist). The same package however is installed properly on terbium.

$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ua_parser import user_agent_parser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/ua_parser/user_agent_parser.py", line 431, in <module>
    yamlFile = open(yamlPath)
IOError: [Errno 2] No such file or directory: '/usr/lib/python2.7/dist-packages/ua_parser/regexes.yaml'

Coincidentally, terbium also has Trusty and jq 1.4.1.

Oh prefect!

We will soon be replacing stat1002 (T159838) and stat1003 (T159839) with newer hardware. When we do so, we will upgrade these to Jessie.

Ottomata added a subtask: T159838: EQIAD: stat1002 replacement.Mar 7 2017, 3:50 PM

RobH closed subtask T159838: EQIAD: stat1002 replacement as Resolved.May 15 2017, 6:00 PM

We are moving to new stats boxes which we will not have this problem.

• Nuria closed this task as Declined.Jun 5 2017, 4:26 PM

Krinkle added a subtask: T152712: Replacement of stat1002 and stat1003.Jul 6 2017, 8:47 PM

• Nuria closed subtask T152712: Replacement of stat1002 and stat1003 as Resolved.Sep 19 2017, 5:52 PM