Page MenuHomePhabricator

Add python3 support for SULWatcher
Closed, ResolvedPublic

Event Timeline

Melos created this task.Feb 13 2019, 11:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2019, 11:41 AM
Melos moved this task from General to SULWatcher on the stewardbots board.Feb 13 2019, 11:48 AM
MarcoAurelio triaged this task as High priority.Oct 13 2019, 11:27 AM

Change 562848 had a related patch set uploaded (by Xqt; owner: Xqt):
[labs/tools/stewardbots@master] [PY2to3] Enable SULWatcher with Python 3

https://gerrit.wikimedia.org/r/562848

Xqt claimed this task.Jan 8 2020, 4:28 PM

Hi @Xqt - thanks for the patch!

I'm testing it and it looks like it's not working though. I keep seeing:

python3 /data/project/stewardbots/SULWatcher/SULWatcher.py
Traceback (most recent call last):
  File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 22, in <module>
    from ircbot import SingleServerIRCBot
ImportError: No module named 'ircbot'

What do you think this might be?

Thanks.

Xqt added a comment.EditedJan 8 2020, 5:43 PM

ircbot has been changed to irc.bot. Patch comming soon

Xqt added a comment.Jan 8 2020, 6:57 PM

Done. The ircbot and irclib is extremely outdated 0.6.3 (2012) or earlier. Now we have 18.0. You have to install the package first. Python 3.6 is needed for the newest release. For older Pythons the right version will be installed with pip install irc.

Done. The ircbot and irclib is extremely outdated 0.6.3 (2012) or earlier. Now we have 18.0. You have to install the package first. Python 3.6 is needed for the newest release. For older Pythons the right version will be installed with pip install irc.

Will they need to be installed for the whole Toolforge project or just doing pip/pip3 install <package_name> for tools.stewardbots work?

Thanks.

Xqt added a comment.Jan 12 2020, 8:19 AM

Done. The ircbot and irclib is extremely outdated 0.6.3 (2012) or earlier. Now we have 18.0. You have to install the package first. Python 3.6 is needed for the newest release. For older Pythons the right version will be installed with pip install irc.

Will they need to be installed for the whole Toolforge project or just doing pip/pip3 install <package_name> for tools.stewardbots work?

Thanks.

It depends from where is Python and its side package is installed. If the steward bots have their own Python environment pip install <package> will be enough. Try pip list or pip freeze to show the current installation

:-(

tools.stewardbots@tools-sgebastion-07:~$ pip freeze
-bash: pip: command not found
tools.stewardbots@tools-sgebastion-07:~$ pip3 freeze
-bash: pip3: command not found
bd808 added a subscriber: bd808.Jan 14 2020, 7:35 PM

It looks like the bot is started on the job grid using jstart -N stewardbot -mem 2G /data/project/stewardbots/venv/bin/python2.7 /data/project/stewardbots/StewardBot/StewardBot.py, so the virtual environment that you need to run pip from is /data/project/stewardbots/venv. It is all python2 right now, so if you are going to try a python3 deployment you will need a new venv. Something like python3 -mvenv $HOME/venv-py3 would make you a new one.

Thanks both. I'll see what I can do.

Xqt added a comment.EditedJan 16 2020, 5:50 AM

Sorry I am not familiar with neither venv nor tool lab. The migrated scripts may run with both, Python 2.7 and 3.4+. Both need the irc package. The package version depends on the Python version installed but is selected automatically via pip. The newest irc lib needs Python 3.6.

Xqt added a subscriber: valhallasw.Jan 16 2020, 5:58 AM

MySQLdb/ MySQL-python do not support Python 3. They were replaced by mysqlclient as far as I could read.

bd808 added a comment.Jan 17 2020, 9:56 PM

MySQLdb/ MySQL-python do not support Python 3. They were replaced by mysqlclient as far as I could read.

I would suggest PyMySQL as pure-Python MySQL client library. It is compatible with Python 2.7 and >=3.4. This is the library used by Legoktm's toolforge Python helper library.

Change 562848 merged by jenkins-bot:
[labs/tools/stewardbots@master] [PY2to3] Enable SULWatcher with Python 3

https://gerrit.wikimedia.org/r/562848

@Legoktm @Xqt I've applied the code kindly submitted by you two and I get the following error message:

sulwatcher.out
Traceback (most recent call last):
  File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 22, in <module>
    from irc.client import nm_to_n
ImportError: cannot import name 'nm_to_n'

Packages installed:

pip freeze
(venv-py3) tools.stewardbots@tools-sgebastion-07:~$ pip freeze
importlib-metadata==1.4.0
importlib-resources==1.0.2
inflect==3.0.2
irc==17.1
jaraco.classes==2.0
jaraco.collections==2.1
jaraco.functools==2.0
jaraco.itertools==4.4.2
jaraco.logging==2.0
jaraco.stream==2.0
jaraco.text==3.2.0
more-itertools==8.1.0
pkg-resources==0.0.0
PyMySQL==0.9.3
pytz==2019.3
six==1.14.0
tempora==1.14.1
zipp==1.0.0
Xqt added a comment.EditedJan 22 2020, 7:27 PM

Seems there was a greater change in these 10 years.

from irc.client import NickMask

def nm_to_n(nm):
    return NickMask(nm).nick

would be a way. I'll update it soon.

Xqt added a comment.Jan 22 2020, 7:41 PM

Btw. is it necessary to keep this script Python 2 compatible?

I'd assume not, as the whole point of the migration is that python2 shouldn't be used anymore. Python 2 should fail explicitly and quickly so we know that everything's actually been migrated properly.

Change 566591 had a related patch set uploaded (by Xqt; owner: Xqt):
[labs/tools/stewardbots@master] [bugfix] Define nm_to_n function

https://gerrit.wikimedia.org/r/566591

Xqt added a comment.Jan 22 2020, 8:34 PM

@Legoktm @Xqt I've applied the code kindly submitted by you two and I get the following error message:

sulwatcher.out
Traceback (most recent call last):
  File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 22, in <module>
    from irc.client import nm_to_n
ImportError: cannot import name 'nm_to_n'

@MarcoAurelio: thanks for reviewing. I made a new patch to solve this bug. That function was dropped and removed in 2012.

Xqt added a comment.Jan 22 2020, 8:39 PM

I'd assume not, as the whole point of the migration is that python2 shouldn't be used anymore. Python 2 should fail explicitly and quickly so we know that everything's actually been migrated properly.

The last patches are intended to work wit both releases. I'll submitt a new patch which removes all Python 2 stuff after that nd propose to test the new libraries irc/pymysql first.

Change 566601 had a related patch set uploaded (by Xqt; owner: Xqt):
[labs/tools/stewardbots@master] [PY3] Python 3 support only for SULWatcher.py

https://gerrit.wikimedia.org/r/566601

Change 566591 merged by jenkins-bot:
[labs/tools/stewardbots@master] [bugfix] Define nm_to_n function

https://gerrit.wikimedia.org/r/566591

@Xqt @Legoktm

Looks like this might be nearly done, but still got errors:

1getConfig(self, 'nickname')
2getConfig(self, 'alias')
3getConfig(self, 'password')
4getConfig(self, 'channel')
5getConfig(self, 'server')
6getConfig(self, 'wmserver')
7getConfig(self, 'rcfeed')
8buildRegex(self)
9buildWhitelist(self)
10buildRegex(self)
11buildWhitelist(self)
12Exception in thread Thread-3:
13Traceback (most recent call last):
14 File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
15 self.run()
16 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 905, in run
17 self.startbot(self.b)
18 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 908, in startbot
19 bot.start()
20 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/bot.py", line 333, in start
21 super(SingleServerIRCBot, self).start()
22 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 1196, in start
23 self.reactor.process_forever()
24 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 842, in process_forever
25 consume(infinite_call(one))
26 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/more_itertools/recipes.py", line 137, in consume
27 deque(iterator, maxlen=0)
28 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/jaraco/itertools.py", line 416, in <genexpr>
29 return (f() for _ in itertools.repeat(None))
30 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 823, in process_once
31 self.process_data(in_)
32 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 787, in process_data
33 conn.process_data()
34 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 271, in process_data
35 self._process_line(line)
36 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 304, in _process_line
37 handler(arguments, command, source, tags)
38 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 340, in _handle_message
39 self._handle_event(event)
40 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 369, in _handle_event
41 self.reactor._handle_event(self, event)
42 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 919, in _handle_event
43 result = handler.callback(connection, event)
44 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 1153, in _dispatcher
45 method(connection, event)
46 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 831, in on_pubmsg
47 a = e.arguments()[0]
48TypeError: 'list' object is not callable
49
50Exception in thread Thread-2:
51Traceback (most recent call last):
52 File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
53 self.run()
54 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 905, in run
55 self.startbot(self.b)
56 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 908, in startbot
57 bot.start()
58 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/bot.py", line 333, in start
59 super(SingleServerIRCBot, self).start()
60 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 1196, in start
61 self.reactor.process_forever()
62 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 842, in process_forever
63 consume(infinite_call(one))
64 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/more_itertools/recipes.py", line 137, in consume
65 deque(iterator, maxlen=0)
66 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/jaraco/itertools.py", line 416, in <genexpr>
67 return (f() for _ in itertools.repeat(None))
68 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 823, in process_once
69 self.process_data(in_)
70 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 787, in process_data
71 conn.process_data()
72 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 271, in process_data
73 self._process_line(line)
74 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 304, in _process_line
75 handler(arguments, command, source, tags)
76 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 330, in _handle_message
77 self._handle_event(event)
78 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 369, in _handle_event
79 self.reactor._handle_event(self, event)
80 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 919, in _handle_event
81 result = handler.callback(connection, event)
82 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 1153, in _dispatcher
83 method(connection, event)
84 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 172, in on_ctcp
85 if e.arguments()[0] == "VERSION":
86TypeError: 'list' object is not callable
87
88Exception in thread Thread-1:
89Traceback (most recent call last):
90 File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
91 self.run()
92 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 905, in run
93 self.startbot(self.b)
94 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 908, in startbot
95 bot.start()
96 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/bot.py", line 333, in start
97 super(SingleServerIRCBot, self).start()
98 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 1196, in start
99 self.reactor.process_forever()
100 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 842, in process_forever
101 consume(infinite_call(one))
102 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/more_itertools/recipes.py", line 137, in consume
103 deque(iterator, maxlen=0)
104 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/jaraco/itertools.py", line 416, in <genexpr>
105 return (f() for _ in itertools.repeat(None))
106 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 823, in process_once
107 self.process_data(in_)
108 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 787, in process_data
109 conn.process_data()
110 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 271, in process_data
111 self._process_line(line)
112 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 304, in _process_line
113 handler(arguments, command, source, tags)
114 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 330, in _handle_message
115 self._handle_event(event)
116 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 369, in _handle_event
117 self.reactor._handle_event(self, event)
118 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 919, in _handle_event
119 result = handler.callback(connection, event)
120 File "/data/project/stewardbots/venv-py3/lib/python3.5/site-packages/irc/client.py", line 1153, in _dispatcher
121 method(connection, event)
122 File "/data/project/stewardbots/SULWatcher/SULWatcher.py", line 172, in on_ctcp
123 if e.arguments()[0] == "VERSION":
124TypeError: 'list' object is not callable
125
126Identifying to services...
127Joined #cvn-unifications
128Identifying to services...
129Joined #cvn-unifications

(curiously only WatcherSUL joined the channel and not SULWatcher - we use those two bots to avoid the bots being kicked due to flood)

Probably that line of code should be e.arguments[0] == "VERSION" now?

Xqt added a comment.Jan 22 2020, 10:05 PM

Probably that line of code should be e.arguments[0] == "VERSION" now?

Yes. I found a hint in the irc history that some methods are changed to property attributes [1]
The others are to be checked too.

[1] https://python-irc.readthedocs.io/en/latest/history.html Version 5.0

Change 569355 had a related patch set uploaded (by Melos; owner: Melos):
[labs/tools/stewardbots@master] SulWatcher:fix py3 migration

https://gerrit.wikimedia.org/r/569355

Change 569355 merged by jenkins-bot:
[labs/tools/stewardbots@master] SulWatcher:fix py3 migration

https://gerrit.wikimedia.org/r/569355

MarcoAurelio added a comment.EditedFeb 3 2020, 10:41 PM

@Melos r/569355 didn't worked :-( Would it be possible to arrange some sort of live debugging on IRC one of this days to take a look at what's going on? Thanks!

It is working now after changing a database row. Let's keep this open for a while and see if it crashes (logs are sent together at sulwatcher.out) and if everything runs as expected, we can call this task resolved :-)

Change 570005 had a related patch set uploaded (by Melos; owner: Melos):
[labs/tools/stewardbots@master] SULWatcher: ignore encoding errors from irc stream

https://gerrit.wikimedia.org/r/570005

Change 570005 merged by jenkins-bot:
[labs/tools/stewardbots@master] SULWatcher: ignore encoding errors from irc stream

https://gerrit.wikimedia.org/r/570005

Change 566601 merged by jenkins-bot:
[labs/tools/stewardbots@master] [PY3] Python 3 support only for SULWatcher.py

https://gerrit.wikimedia.org/r/566601

Melos closed this task as Resolved.Feb 7 2020, 12:57 PM