Page MenuHomePhabricator

C# MONO bot runs OK but not with jsub
Closed, ResolvedPublic

Description

The same mono based bot runs reliably (at least for a while) from the tool account but does not start from jsub. I created a minimal tool account for this (become bothasava)

tools.bothasava@tools-bastion-02:~$ mono BotHasava.exe /d 
BotHasava()
DotNetWikiBot 3.15
Copyright (c) Iaroslav Vassiliev, 2006-2016, GNU General Public License 2.0

botMessagesLang=en :-)
Logging in...
Logged in as KotzBot.
Site: ???????? (MediaWiki 1.29.0-wmf.1)
q.Count=0 ==> Maxi=99999
q2.Count=0 ==> seconds_to_wait_after_save=0
Getting category "???????:????????/???/??? ?????/???? ??????/???? ???" contents...
PageList has been filled with 1 page titles found in "????????/???/??? ?????/???? ??????/???? ???" category.
Starting go()
working on ???? ?????:????/?? ???? ?????
Page "???? ?????:????/?? ???? ?????" loaded successfully.
M=???=[[???? ?????:????/?? ???? ?????/???????]]
...

But:

tools.bothasava@tools-bastion-02:~$ jsub -l release=trusty -N hasava mono BotHasava.exe /d
Your job 139995 ("hasava") has been submitted
tools.bothasava@tools-bastion-02:~$ cat hasava.out
BotHasava()
DotNetWikiBot 3.15
Copyright (c) Iaroslav Vassiliev, 2006-2016, GNU General Public License 2.0

botMessagesLang=iv :-)
Logging in...
tools.bothasava@tools-bastion-02:~$ cat hasava.err 

Unhandled Exception:
System.TypeInitializationException: An exception was thrown by the type initializer for BotHasava ---> System.Net.WebException: The request timed out
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0 
  at DotNetWikiBot.Site.PostDataAndGetResult (System.String pageURL, System.String postData, Boolean getCookies, Boolean allowRedirect) [0x00000] in <filename unknown>:0 
  --- End of inner exception stack trace ---
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeInitializationException: An exception was thrown by the type initializer for BotHasava ---> System.Net.WebException: The request timed out
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0 
  at DotNetWikiBot.Site.PostDataAndGetResult (System.String pageURL, System.String postData, Boolean getCookies, Boolean allowRedirect) [0x00000] in <filename unknown>:0 
  --- End of inner exception stack trace ---

thanks

Event Timeline

Kotz created this task.Nov 5 2016, 5:39 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptNov 5 2016, 5:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Kotz renamed this task from C# MONO bot runs OK but not with sub to C# MONO bot runs OK but not with jsub.Nov 5 2016, 6:04 PM
Kotz added a comment.Nov 5 2016, 6:13 PM

It may be that the problem is actually with accessing https://he.wikipedia.org/w/api.php?action=query&meta=tokens&type=login&format=xml

I had some SSL problems which I thought I fixed in the account

It seems unlikely that the server has no network access, so this is likely an issue in your code or the .net mediawiki library you're using. You can test manually on the server by sshing to it (e.g. ssh tools-exec-1401). This is OK for debugging, but not as a normal way to run code.

Finally, there are some differences in the environment variables between a command run from a shell or from SGE. For example, .bashrc is not parsed by SGE, so if you have any modification there they will not be reflected. I assume .NET has a way to show current env variables -- that would be one place to get started.

Kotz added a comment.Nov 13 2016, 12:35 AM

@valhallasw thank you for commenting.

ssh -T tools-exec-1403 /usr/bin/mono BotHasava.exe works fine. I didn't find any env var justifying this problem. Certainly it's not a problem of tools-exec-1401 not having internet access but something more complex but I need a way to debug it.

Is there a way I can get the same env as jsub on the login server or in a more or less interactive shell?

thanks

Is there a way I can get the same env as jsub on the login server or in a more or less interactive shell?

You can try qlogin, but I'm not sure if the environment will be the same as in a regular jsub.

Kotz added a comment.EditedNov 13 2016, 7:32 PM

qlogin and qrsh work OK too.

I can actually do

qrsh -l release=trusty mono BotHasava.exe

and this works quite reliably. Is it acceptable to use qrsh instead of jsub this way?

Kotz added a comment.Nov 21 2016, 11:15 AM

No, this does not work reliably. Sometimes it asks for key verifications. Also I understand it would not work with cron (only jsub works there). I still need some help here. Does anyone run DotNetWikiBot bot on tools? or mono at all?

scfc added a subscriber: scfc.Nov 21 2016, 12:27 PM

@Kotz: What do you mean by "key verifications"? Please always quote error messages verbatim.

Kotz added a comment.EditedNov 21 2016, 12:57 PM

@scfc,

>become botsister qrsh -l release=trusty  pwd
Host key verification failed.

this happens only intermittently. Anyway this is not the major issue for this ticket I think. I need to be able to run by bot via jsub in order to have them scheduled by cron. the "Host key verification" happens with qrsh not jsub.

The major problem is that the same mono bot works OK from the login or dev ssh and also from qrsh and qlogin, but fails with jsub.

thanks

scfc added a comment.Nov 21 2016, 2:50 PM

qacct -j 139995 shows that maxvmem registered for that job was 457.445 MBytes. While exit_status is 1 and IIRC it would be some larger value for out-of-memory failures, maxvmem is "close" to the default memory allocation of 512 MBytes (cf. https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Allocating_additional_memory). So the error could be an artefact of the job not being able to allocate enough memory.

I would try using jsub -l release=trusty -mem 1g -N hasava mono BotHasava.exe /d to see if that makes a difference.

Kotz added a comment.Nov 21 2016, 4:00 PM

@scfc thanks this did the trick for BotHasava :-)

scfc closed this task as Resolved.Nov 22 2016, 10:00 AM
scfc assigned this task to Kotz.