f37be4c0e9cdc37abea94c97c64fcf8c4f8038eb
ActivePublic
Actions

Authored by zhuyifei1999 on Jan 30 2017, 11:39 AM.

Project Tags

None

Referenced Files

	F5417444: f37be4c0e9cdc37abea94c97c64fcf8c4f8038eb
	Jan 30 2017, 11:39 AM

Subscribers

None

	commit f37be4c0e9cdc37abea94c97c64fcf8c4f8038eb
	Author: zhuyifei1999 <zhuyifei1999@gmail.com>
	Date: Mon Jan 30 19:31:32 2017 +0800

	tail optimization: remove subprocess overhead

	Subprocess tailing with lines require many read and seeks in order
	to find the position of starting line, and then it is a simple
	copy-to-stdout. The former is extremely unnecessary, since we do not
	require an accurate starting line.

	Seeking to a similar position with SEEK_END will be much faster, and
	easier to implement. Now that is is implemented within Python, we no
	longer have the overhead of process initialization and inter-process
	communication.

	The last 10000 lines has 3679657 bytes. Rounding up, we change each
	line to 400 bytes.

	----

	diff --git a/precise_tools/__init__.py b/precise_tools/__init__.py
	index abe09de..c3feeeb 100644
	--- a/precise_tools/__init__.py
	+++ b/precise_tools/__init__.py
	@@ -38,7 +38,8 @@ def tools_from_accounting(days):
	delta = datetime.timedelta(days=days)
	cutoff = int(utils.totimestamp(datetime.datetime.now() - delta))
	tools = []
	- for line in utils.tail('/data/project/.system/accounting', 45000 * days):
	+ for line in utils.lines_in_last_n_bytes('/data/project/.system/accounting',
	+ 400 * 45000 * days):
	parts = line.split(':')
	job = dict(zip(ACCOUNTING_FIELDS, parts))
	if int(job['end_time']) < cutoff:
	diff --git a/precise_tools/utils.py b/precise_tools/utils.py
	index 8b1ae65..e34b82a 100644
	--- a/precise_tools/utils.py
	+++ b/precise_tools/utils.py
	@@ -17,29 +17,23 @@
	# with this program. If not, see <http://www.gnu.org/licenses/>.

	from __future__ import division
	+
	+import os
	import datetime
	-import subprocess
	-
	-
	-def tail(filename, lines):
	- """Get last n lines from the filename as an iterator."""
	- # Inspired by http://stackoverflow.com/a/4418193/8171
	- cmd = ['tail', '-%d' % lines, filename]
	- proc = subprocess.Popen(
	- cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
	- while True:
	- line = proc.stdout.readline()
	- if line == '' and proc.poll() is not None:
	- break
	- yield line
	-
	- if proc.returncode == 0:
	- raise StopIteration
	- else:
	- raise subprocess.CalledProcessError(
	- returncode=proc.returncode,
	- command=' '.join(cmd)
	- )
	+
	+
	+def lines_in_last_n_bytes(filename, nbytes):
	+ """Get lines from last n bytes from the filename as an iterator."""
	+ with open(filename, 'r') as f:
	+ f.seek(-nbytes, os.SEEK_END)
	+
	+ # Ignore first line as it may be only part of a line
	+ f.readline()
	+
	+ # We can't simply `return f` as the returned f will be closed
	+ # Do all the IO within this function
	+ for line in f:
	+ yield line


	def totimestamp(dt, epoch=None):

Event Timeline

zhuyifei1999 created this paste.Jan 30 2017, 11:39 AM

zhuyifei1999 mentioned this in T156386: Create a wall for tools migration to trusty.Jan 30 2017, 11:51 AM

f37be4c0e9cdc37abea94c97c64fcf8c4f8038ebActivePublicActions

Event Timeline

f37be4c0e9cdc37abea94c97c64fcf8c4f8038eb
ActivePublic
Actions