Page MenuHomePhabricator

transfer.py argument parsing exception
Open, LowPublic

Description

I somewhat managed to crash it:

# transfer.py cumin1001.eqiad.wmnet:/home/jynus/:abuser_revisions.txt mwmaint1002.eqiad.wmnet:/home/jynus
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 60, in execute
    hosts = self._query_default_backend(query_string)
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 95, in _query_default_backend
    return query.execute(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 47, in execute
    self._build(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/puppetdb.py", line 304, in _build
    super()._build(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 76, in _build
    parsed = self.grammar.parseString(query_string.strip(), parseAll=True)
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1632, in parseString
    raise exc
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1626, in parseString
    se._parse( instring, loc )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3395, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1383, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3183, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected end of text (at char 21), (line:1, col:22)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 63, in execute
    hosts = super().execute(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 47, in execute
    self._build(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 111, in _build
    super()._build(query_string)
  File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 76, in _build
    parsed = self.grammar.parseString(query_string.strip(), parseAll=True)
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1632, in parseString
    raise exc
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1622, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3717, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3378, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3717, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3545, in parseImpl
    raise maxException
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3530, in parseImpl
    ret = e._parse( instring, loc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3717, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3378, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3717, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3378, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3545, in parseImpl
    raise maxException
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 3530, in parseImpl
    ret = e._parse( instring, loc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 1383, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3/dist-packages/pyparsing.py", line 2503, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected {Combine:({Combine:({D | K | P "{"}) SkipTo:("}") "}"}) | Combine:({"A" ":" W:(ABCD...)}) | {"(" Forward: ... ")"}} (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/transfer.py", line 11, in <module>
    load_entry_point('transferpy==1.0', 'console_scripts', 'transfer.py')()
  File "/usr/lib/python3/dist-packages/transferpy/transfer.py", line 265, in main
    result = t.run()
  File "/usr/lib/python3/dist-packages/transferpy/Transferer.py", line 554, in run
    self.sanity_checks()
  File "/usr/lib/python3/dist-packages/transferpy/Transferer.py", line 363, in sanity_checks
    result = self.host_exists(self.source_host)
  File "/usr/lib/python3/dist-packages/transferpy/Transferer.py", line 83, in host_exists
    result = self.run_command(host, command)
  File "/usr/lib/python3/dist-packages/transferpy/Transferer.py", line 55, in run_command
    return self.remote_executor.run(host, command)
  File "/usr/lib/python3/dist-packages/wmfmariadbpy/RemoteExecution/CuminExecution.py", line 40, in run
    hosts = query.Query(self.config).execute(host)
  File "/usr/lib/python3/dist-packages/cumin/query.py", line 68, in execute
    query=query_string, name=self.config['default_backend'], e_def=e_default, e_glob=e_global))
cumin.backends.InvalidQueryError: Unable to parse the query 'cumin1001.eqiad.wmnet/home/jynus/' neither with the default backend 'puppetdb' nor with the global grammar:
puppetdb: Expected end of text (at char 21), (line:1, col:22)
global: Expected {Combine:({Combine:({D | K | P "{"}) SkipTo:("}") "}"}) | Combine:({"A" ":" W:(ABCD...)}) | {"(" Forward: ... ")"}} (at char 0), (line:1, col:1)

Event Timeline

LSobanski triaged this task as Medium priority.Dec 21 2020, 11:47 AM
jcrespo lowered the priority of this task from Medium to Low.Dec 21 2020, 11:58 AM

This is low, while probably very easy to fix (capture exception for invalid targets), it was a user error, 0 impact normally on work.

jcrespo added subscribers: Marostegui, rafayghafoor.

Hey, @rafayghafoor,

please do not claim this task before talking to @Marostegui or me first! :-) Let us know what your plans are before working on something.

Sorry, @jcrespo, I am interested in this project as I am willing to participate in GSoC this year and would like to work on this task.

@rafayghafoor Please send first a gerrit patch (preferred) or comment here how to resolve it/ask any questions you have.

This comment was removed by Palak199.

https://wikitech.wikimedia.org/wiki/Cumin Is used as a dependency. I fact, I think wmfmariadbpy.remote_execution is a dependency, and that depends on cumin.

This mentions how to install Cumin: https://cumin.readthedocs.io/en/stable/installation.html

That should install /etc/cumin/config.yaml, and for a test install, default_backend: direct should have the least problems to be setup.

I agree this is an elaborate setup, but for this ticket I think the issue is easy enough that a test case could be enough to check it has been fixed.

This mentions how to install Cumin: https://cumin.readthedocs.io/en/stable/installation.html

I installed cumin with

pip install cumin

and confirmed later with

cumin --version

That should install /etc/cumin/config.yaml, and for a test install, default_backend: direct should have the least problems to be setup.

However this file didn't get created on its own. I tried creating on myself which I saw as a default one in the guide you mentioned.

Even with this the deb package is not getting built.

debuild -b -us -uc

Its output

 dpkg-buildpackage -us -uc -ui -b
dpkg-buildpackage: info: source package transferpy
dpkg-buildpackage: info: source version 1.0
dpkg-buildpackage: info: source distribution buster-wikimedia
dpkg-buildpackage: info: source changed by Ajumal P A <ajupazhamayil@gmail.com>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from operations-software-transferpy/debian/source/options: --extend-diff-ignore=^[^/]*[.]egg-info/ --tar-ignore=.*
dpkg-checkbuilddeps: error: Unmet build dependencies: cumin
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
dpkg-buildpackage: warning: (Use -d flag to override.)
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui -b failed

in case I pass -d flag to silent this-

dpkg-buildpackage: info: binary-only upload (no source included)
Now running lintian transferpy_1.0_amd64.changes ...
E: transferpy changes: bad-distribution-in-changes-file buster-wikimedia
W: transferpy: binary-without-manpage usr/bin/transfer.py
W: transferpy: script-with-language-extension usr/bin/transfer.py
Finished running lintian.

(this output is actually very big, but this is how it ends)

and the deb file doesn't get generated.
Please suggest

You can ignore the debian package building for this ticket- it is not necessary to work on it.

For debian package building, the debian package of cumin has to be installed. There is a repo containing it at: https://apt.wikimedia.org/wikimedia/pool/main/c/cumin/cumin_4.0.0-1_amd64.deb

This is the one that probably will autogenerate the config file.

For debian package building, the debian package of cumin has to be installed. There is a repo containing it at: https://apt.wikimedia.org/wikimedia/pool/main/c/cumin/cumin_4.0.0-1_amd64.deb

I tried this too, but no luck.
However I tried running transfer.py file with no issues (probably because cumin has issues) and think that the crash you mentioned is due to unhandled InvalidQueryError exception from cumin.backends. I went through the docs and found the description on how it can be handled with a custom class.
Let me know what you think about it and how must I proceed to fix it.

Change 674319 had a related patch set uploaded (by Palak199; owner: Palak199):
[operations/software/transferpy@master] Fix:: InvalidQueryException handling

https://gerrit.wikimedia.org/r/674319

Change 674577 had a related patch set uploaded (by Palak199; owner: Palak199):
[operations/software/transferpy@master] Modify:: The parsing function in transfer.py Add:: Whitespace between = and [ parentheses in parsing function The parsing function is modified to parse the host and path as the earlier method crashed the script.

https://gerrit.wikimedia.org/r/674577

Change 674663 had a related patch set uploaded (by Palak199; author: Palak199):
[operations/software/transferpy@master] Modify:: Modify path variable in parse function of transfer.py

https://gerrit.wikimedia.org/r/674663

Change 674663 abandoned by Jcrespo:
[operations/software/transferpy@master] Modify:: Modify path variable in parse function of transfer.py

Reason:
submitted as a new change, probably by accident

https://gerrit.wikimedia.org/r/674663

I will be checking this contribution soon as part of my job maintaining backups, and hopefully, getting it merged.