Page MenuHomePhabricator

jmail is not puppetized
Closed, ResolvedPublic

Description

jmail can be used in ~/.forward to process incoming mail (cf. https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Processing_email_programatically). It is currently not puppetized. The source is on tools-mail in /usr/local/bin/jmail.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Current script:

#! /bin/bash
#
# Copyright © 2013 Marc-André Pelletier <mpelletier@wikimedia.org>
#
# Permission to use, copy, modify, and/or distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
#

cat >>/tmp/jmail.log <<EOF
$(date):
cmdline: $@
$(id)
----
EOF
email="/data/project/.system/store/mail/.deliver.$(date +%s).$$"
prog="$1"
shift
cd $HOME
exe=$(PATH="$PATH:$HOME" /usr/bin/which "$prog")

if [ ! -x "$exe" ]; then
  echo "Processing program unavailable" >&2
  exit 1
fi

trap "/bin/rm -f '$email' '$email.out'" 0
if ! cat >$email 2>/dev/null; then
  exit 1
fi
/usr/bin/qsub -N mail.$(/usr/bin/id -nu) \
        -sync y -b y -m n \
        -o "$email.out" -j y -i "$email" \
        -q mailq -l h_vmem=500M -r n \
        "$exe" "$@" >/dev/null
rv=$?
cat "$email.out"
exit $rv

Actually, there seems to be a bug in that script: The first run of cat >>/tmp/jmail.log after a reboot creates /tmp/jmail.log being owned by that particular user (mostly drtrigonbot) and subsequent invocations by other users will fail. But:

scfc@tools-mail:~$ sudo find -H /data/project -mindepth 2 -maxdepth 2 -type f -name .forward\* -exec fgrep jmail {} +
/data/project/tsreports-dev/.forward:|jmail /data/project/tsreports-dev/test.py
/data/project/csbot/.forward.test:|jmail mailtest
/data/project/drtrigonbot/.forward.subster:|jmail cat >> ~/data/subster/mail_inbox
/data/project/bd808-test/.forward.jmail:|jmail tee -a /data/project/bd808-test/jmail.spool
scfc@tools-mail:~$

so the chance is pretty slim that someone would have noticed :-).

I believe the part about email="/data/project/.system/store/mail/.deliver.$(date +%s).$$" should also be replaced by files created by mktemp in the tool's home directory. I wasn't aware of this use of /data/project/.system/store/mail (https://gerrit.wikimedia.org/r/#/c/326306/ as proof :-)) and I think per-tool storage is a better idea. Also I would only remove the files on success. There seem to be some confusion on which signals trap something 0 is executed by which shells.

It would be swell to have this rewritten in python and added to the labs/toollabs.git repo as part of the jobutils deb.

Email as an API seems like an idea fraught with peril to me. Can we remove this entirely? What's the must-have use case here?

jmail is documented on wiki: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Processing_email_programatically

I don't know if this was something carried over from Toolserver or a local invention.

(I disagree that mail APIs are worse than others; in fact IMHO the inherent queue system saves much trouble. Being able to use Phabricator or Debian Bugs asynchronously is priceless.)

It looks as if drtrigonbot is the only user/tool that uses jmail for non-testing purposes. @DrTrigon, do you actively use that?

Change 339920 had a related patch set uploaded (by Tim Landscheidt):
Package jmail

https://gerrit.wikimedia.org/r/339920

I have put the rewrite at /usr/local/bin/jmail.new and set a symbolic link from /usr/local/bin/jmail. I ran several tests and I'm confident that the patch works.

Change 340276 had a related patch set uploaded (by Tim Landscheidt; owner: Tim Landscheidt):
Package jmail

https://gerrit.wikimedia.org/r/340276

Mentioned in SAL (#wikimedia-labs) [2017-02-28T03:52:27Z] <scfc_de> Deployed jobtools and misctools 1.20/1.20~precise+1 (T158722).

I removed the now-obsolete files under /usr/local/bin and tested successfully /usr/bin/jmail with echo Test 15 | mail -s Test tools.scfc-test-can-be-deleted-anytime (as seen above, no existing ~/.forward referred to /usr/local/bin/jmail by its absolute path).