Page MenuHomePhabricator

Odd kubernetes error
Closed, ResolvedPublic

Description

Some of my tools, including autolist2, use a shared PHP library I wrote. Part of this includes working with temporary files, to save RAM. At one point, it runs a shell command to filter a result file. Example:

jq '.items' /data/project/autolist/tmp/omniscan_XS9zcO | sed 's/[^0-9]//g' |  grep -v '^$' | sed -e 's/^/Q/' | sed -e 's/$/\t0/' | sort -u > /data/project/autolist/tmp/omniscan_cUIK9E

This runs fine under gridengine, but fails on kubernetes; there, it doesn't even create the output file.
I have restarted autolist2 webservice under gridengine, until this can be fixed.

Event Timeline

Magnus created this task.Jul 21 2016, 8:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2016, 8:33 PM

i think sed and grep aren't installed....

Ah. I guess they were on gridengine, so I just expected the same config...
Will you be installing them?

@Magnus: T140110: Packages to be installed in Toolforge Kubernetes Images (Tracking) is the tracking task for packages to be installed in Kubernetes containers. You'll need to create a subtask of that.

Ah. I guess they were on gridengine, so I just expected the same config...
Will you be installing them?

I'd like to keep the default containers just have PHP and not extra utilities, but I'll also probably make available a 'fat' container that has all these things installed and allow users to specify using that.

I'll have a look at it this weekend, maybe I can code around it in PHP. Then I could continue to use the "default" container.

@Magnus thank you :D You can also go to
https://grafana-labs-admin.wikimedia.org/dashboard/db/kubernetes-pods
(login with your wikitech username/password, will be made more open soon),
and select your tool name in the dropdown on top left - will show you
graphs of memory / CPU and network usage for the selected tool. You can use
this to check memory usage. It'll be far more accurate than what gridengine
provided, since I'm not sure if gridengine properly accounted for spawned
processes.

Thanks for the stats link, very useful!

I have re-written the code to not use jq, but I see it uses "cat", "sort", "uniq" shell commands extensively for operations. I designed the code that way because, in a long-drawn battle with toolserver and then Labs resources, use of temporary DB tables and PHP memory was essentially disallowed, so I moved to the last resource available, disk. Rewriting all of this for, I believe, the fourth time, is out of the question; I'll rather close the tools involved.

I think autolist is the last tool using this "omniscan" class (located in catscan2:public_html/omniscan.inc), and that is scheduled to be replaced by PetScan.

@Magnus ok! I could either:

  1. Install cat / sort / uniq / 'base unix shell commands' on all containers

(or)

  1. Just speed up building of the 'fat' container.

I'll let you know when either of these happen - my current thinking is to
just build the fat container. I'll let you know when either of these things
happen - no need to close anything down :)

Thank you!

Actually, I'd like to get rid of autolist ;-)
I wrote PetScan to replace several tools, autolist included. The only reason I haven't retired it is that it supports WDQ, and I deliberately don't want that in PetScan. I want to retire WDQ in favour of SPARQL, but SPARQL is not quite as capable for some edge cases yet. So my plan is to discourage WDQ use by retiring some tools that use it first. Tool retirement chain!
I could shut down autloist today if I included WDQ in PetScan, which would take a few minutes; but then I'd have to support WDQ because of PetScan. Problems, problems...

After some more difficulty in the 'omnibus' solution, I am now in the process of adding small utilities to all the language level containers. That might be enough to get autolist working on that. I'll update this ticket based on how that goes.

yuvipanda closed this task as Resolved.Dec 1 2016, 3:08 PM

Ah, autolist now just redirects to PetScan :) Either way, all the tools mentioned are now in the base images :) I've switched autolist to run on k8s now, and seems to work ok.