Caused by: org.apache.xmlrpc.XmlRpcException: You must log in before using this part of Bugzilla.

[2014-05-15 20:49:56,495] ERROR com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor : Error while executing action status PATCH_TO_REVIEW
java.io.IOException: com.j2bugzilla.base.BugzillaException: An unknown error was encountered; fault code: 410

at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.execute(BugzillaItsFacade.java:189)
at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.performAction(BugzillaItsFacade.java:97)
at com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor.execute(ActionExecutor.java:72)
at com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor.execute(ActionExecutor.java:84)
at com.googlesource.gerrit.plugins.hooks.workflow.ActionController.onChangeEvent(ActionController.java:55)
at com.google.gerrit.common.ChangeHookRunner.fireEventForUnrestrictedListeners(ChangeHookRunner.java:619)
at com.google.gerrit.common.ChangeHookRunner.fireEvent(ChangeHookRunner.java:630)
at com.google.gerrit.common.ChangeHookRunner.doPatchsetCreatedHook(ChangeHookRunner.java:356)
at com.google.gerrit.server.change.ChangeInserter.insert(ChangeInserter.java:183)
at com.google.gerrit.server.change.CherryPickChange.createNewChange(CherryPickChange.java:244)
at com.google.gerrit.server.change.CherryPickChange.cherryPick(CherryPickChange.java:182)
at com.google.gerrit.server.change.CherryPick.apply(CherryPick.java:84)


at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)

Caused by: com.j2bugzilla.base.BugzillaException: An unknown error was encountered; fault code: 410

at com.j2bugzilla.base.XmlExceptionHandler.handleFault(XmlExceptionHandler.java:96)
at com.j2bugzilla.base.BugzillaConnector.executeMethod(BugzillaConnector.java:169)
at com.googlesource.gerrit.plugins.hooks.bz.BugzillaClient.performAction(BugzillaClient.java:128)
at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.doPerformAction(BugzillaItsFacade.java:109)
at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.access$400(BugzillaItsFacade.java:31)
at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade$3.call(BugzillaItsFacade.java:102)
at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade$3.call(BugzillaItsFacade.java:97)
at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.execute(BugzillaItsFacade.java:177)
... 56 more

Caused by: org.apache.xmlrpc.XmlRpcException: You must log in before using this part of Bugzilla.

at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197)
at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:156)
at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143)
at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69)
at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126)
at com.j2bugzilla.base.BugzillaConnector.executeMethod(BugzillaConnector.java:164)
... 62 more

12:40 < JohnLewis> Hm. Gerrit not notifying Bugzilla of patches?
12:41 < mutante> JohnLewis: do you know since when?
12:42 < JohnLewis> mutante: https://gerrit.wikimedia.org/r/#/c/133364/ was uploaded yesterday and the Bugzilla bug was not notified. That's as far as I know.
12:43 < JohnLewis> I just patched a bug and that wasn't updated so it's still an on-going issue for reference

12:50 < mutante> http://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines#Auto-linking_and_cross-referencing
12:50 < mutante> "A bot will automatically notify Bugzilla ... "

12:52 < mutante> ^d: ideas why that could have changed recently?
12:52 < ^d> godog restarting gerrit?

12:53 < mutante> i hear it broke within the last 9 hours
12:53 < mutante> ah
12:53 < andre> so the plugin in Gerrit is https://gerrit-review.googlesource.com/#/admin/projects/plugins/its-bugzilla
12:54 < andre> and qchris set it up once upon a time
12:55 < andre__> very last activity in Bugzilla was Gerrit Notification Bot (gerritadmin@wikimedia.org) 2014-05-15 11:41:52 UTC in

https://bugzilla.wikimedia.org/show_bug.cgi?id=65225#c3

13:01 < mutante> the bugzilla plugin is listed as enabled in gerrit
13:02 < mutante> hooks-bugzilla2.8.1 Enabled .. hrmm
13:02 < mutante> https://gerrit.wikimedia.org/r/plugins/hooks-bugzilla/Documentation/index.html
13:03 < mutante> https://gerrit.wikimedia.org/r/plugins/hooks-bugzilla/Documentation/config.html even
13:03 < mutante> "Bugzilla credentials and connectivity details are asked and verified during the Gerrit init."
13:03 < mutante> that?

13:12 < andre__> mutante, gerritadmin@wikimedia.org

13:13 < ^d> fwiw, gerrit is connecting.
13:13 < ^d> [2014-05-15 12:57:28,709] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaModule : Bugzilla is configured as ITS
13:13 < ^d> [2014-05-15 12:57:30,106] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade : Connected to https://bugzilla.wikimedia.org as

username=gerritadmin@wikimedia.org, userid=16761

13:13 < ^d> [2014-05-15 12:57:30,646] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade : Connected to Bugzilla at

https://bugzilla.wikimedia.org/xmlrpc.cgi, reported version is 4.4.4

13:58 < andre__> Bugzilla DID change some cookie/login related stuff between 4.4.1 and 4.4.4, but we upgraded to 4.4.4 more than one day ago

13:20 < ^d> Hmm, I reloaded the plugin and cleared the error log.
13:21 < ^d> Once again, Connected to https://bugzilla.wikimedia.org as username=gerritadmin@wikimedia.org, userid=16761
13:21 < ^d> Wonder if it'll last.

..but then it didn't

Gerrit log excerpt above from /var/lib/gerrit2/review_site/log/error_log

There were firewall changes but timing does not fit, as last activity in Bugzilla by Gerrit Notification Bot (gerritadmin@wikimedia.org) was 2014-05-15 11:41:52 UTC in bug 65225 comment 3.

<mutante> ^d: did you know about openjdk being upgraded btw?
<mutante> @ytterbium:/var/log# grep jdk /var/log/apt/history.log

<andre> plugin in Gerrit is https://gerrit-review.googlesource.com/#/admin/projects/plugins/its-bugzilla
<andre> for the records, the quick'n'dirty check if it's running again would be any patch created in Gerrit and correctly refering to a Bugzilla ID, which would create a result shown for https://bugzilla.wikimedia.org/buglist.cgi?chfield=bug_status&chfieldfrom=-1h&chfieldto=Now&chfieldvalue=PATCH_TO_REVIEW&email1=gerritadmin%40wikimedia.org&emaillongdesc1=1&emailtype1=exact&query_format=advanced (that query covers the last one hour)

yea, regarding the openjdk update, no idea if it's related but:

Start-Date: 2014-05-15 11:48:08
Commandline: apt-get install openjdk-6-jdk
Install: openjdk-6-jdk:amd64 (6b31-1.13.3-1ubuntu1~0.12.04.2)
Upgrade: icedtea-6-jre-cacao:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre-lib:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre-headless:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), icedtea-6-jre-jamvm:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2)
End-Date: 2014-05-15 11:48:13

Start-Date: 2014-05-15 11:49:39
Commandline: apt-get purge openjdk-6-jdk
Purge: openjdk-6-jdk:amd64 (6b31-1.13.3-1ubuntu1~0.12.04.2)
End-Date: 2014-05-15 11:49:39

Start-Date: 2014-05-15 11:49:57
Commandline: apt-get install openjdk-7-jdk
Install: openjdk-7-jdk:amd64 (7u25-2.3.10-1ubuntu0.12.04.2), openjdk-7-jre-lib:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, automatic)
Downgrade: openjdk-7-jre:amd64 (7u51-2.4.4-0ubuntu0.12.04.2, 7u25-2.3.10-1ubuntu0.12.04.2), openjdk-7-jre-headless:amd64 (7u51-2.4.4-0ubuntu0.12.04.2, 7u25-2.3.10-1ubuntu0.12.04.2)
Remove: icedtea-7-jre-jamvm:amd64 (7u51-2.4.4-0ubuntu0.12.04.2)
End-Date: 2014-05-15 11:50:02

Start-Date: 2014-05-15 11:51:04
Commandline: apt-get remove openjdk-7-jdk
Remove: openjdk-7-jdk:amd64 (7u25-2.3.10-1ubuntu0.12.04.2)
End-Date: 2014-05-15 11:51:04

Start-Date: 2014-05-15 11:53:14
Commandline: apt-get install openjdk-7-jre=7u55-2.4.7-1ubuntu1~0.12.04.2 openjdk-7-jdk=7u55-2.4.7-1ubuntu1~0.12.04.2 openjdk-7-jre-headless=7u55-2.4.7-1ubuntu1~0.12.04.2
Install: openjdk-7-jdk:amd64 (7u55-2.4.7-1ubuntu1~0.12.04.2)
Upgrade: openjdk-7-jre:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, 7u55-2.4.7-1ubuntu1~0.12.04.2), openjdk-7-jre-headless:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, 7u55-2.4.7-1ubuntu1~0.12.04.2)
End-Date: 2014-05-15 11:53:18

11:56 godog: installed openjdk-7-jdk on ytterbium to attempt gerrit thread dump

(In reply to Daniel Zahn from comment #5)

yea, regarding the openjdk update, no idea if it's related but:

Start-Date: 2014-05-15 11:48:08
Commandline: apt-get install openjdk-6-jdk

Assuming that time is UTC, that fits reasonably well to the last activity in Bugzilla at 11:41:52 UTC.

(In reply to Daniel Zahn from comment #2)

13:58 < andre__> Bugzilla DID change some cookie/login related stuff between
4.4.1 and 4.4.4, but we upgraded to 4.4.4 more than one day ago

To clarify what I refered to, quoting the 4.4.3 release notes:
"The User.login WebService method now also returns a token argument containing a login token which you can use in subsequent calls to authenticate. For security reasons, this method no longer generates login cookies. ( https://bugzilla.mozilla.org/show_bug.cgi?id=893195 )"

The openjdk change time fits really, really nicely. But I it should
(yes. Vague :-/ ) not affect us here, as Gerrit uses the java implementation
that's referenced in it's config file (key: container.javaHome), which is
currently

/usr/lib/jvm/java-7-openjdk-amd64/jre

I checked, and this jvm seems basically functional still.

I tried a local bugzilla-4.4.4 instance, and can reproduce the issue there.
(Bugzilla 4.2.4 works nicely)

Prodding down that line, it seems comment #8 is relevant.
(Bugzilla's 410 error is "login_required")

j2bugzilla (the third part bugzilla xmlrpc library used in hooks-bugzilla) is
still current, and has not seen adaptions for bugzilla 4.4.3.
So I'll see if j2bugzilla needs fixing for the login change of bugzilla 4.4.3.

(In reply to christian from comment #9)

I tried a local bugzilla-4.4.4 instance, and can reproduce the issue there.

Weird, because we upgraded Bugzilla to 4.4.4 already on 2014-04-22 (bug 61287 comment 5) and the notifications still worked until yesterday.

(Oops, potentially confusing last comment: We skipped the broken 4.4.3 and upgraded from 4.4.1 or 4.4.2 directly to 4.4.4)

Had a chat with qchris who asked if downgrading Bugzilla to 4.4.2 is possible (otherwise fixing would take a bit longer).

We're still wondering why the problem only arose yesterday because our 4.4.4 upgrade was three weeks ago (cached login cookie? shrug).

My guess is that reverting http://bzr.mozilla.org/bugzilla/4.4/revision/8664 (for https://bugzilla.mozilla.org/show_bug.cgi?id=893195 ) should be sufficient.

As I got to leave in 15min (will be back later tonight) I'd love to get an opinion from mutante.

Change 133589 had a related patch set uploaded by QChris:
test gerrit->bz notification

https://gerrit.wikimedia.org/r/133589

Deployed a patched j2bugzilla/hooks-bugzilla plugin that allows token passing
for bugzilla-4.4.4 instead of using sessions.

Seems to be working again (see comment 13).

Change 133732 had a related patch set uploaded by QChris:
Update hoobs-bugzilla to 5edd392d926daaa58917b1c8bb174cdb022e4c76

https://gerrit.wikimedia.org/r/133732

qchris: Thank you, thank you, thank you!

Change 133589 abandoned by QChris:
test gerrit->bz notification

Reason:
gerrit -> bugzilla notifications are working again.

https://gerrit.wikimedia.org/r/133589

Change 133732 merged by Chad:
Update hooks-bugzilla to 5edd392d926daaa58917b1c8bb174cdb022e4c76

https://gerrit.wikimedia.org/r/133732

aric wrote:

Hi guys, I would like to get this change pushed upstream to https://gerrit.googlesource.com/plugins/its-bugzilla/
I only see a binary diff in gerrit, can I get a link to the source code changed?

Thanks!

-Aric

Unzip the jar? ;)

Binary files new/com/googlesource/gerrit/plugins/hooks/bz/BugzillaClient.class and old/com/googlesource/gerrit/plugins/hooks/bz/BugzillaClient.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/bz/BugzillaItsFacade.class and old/com/googlesource/gerrit/plugins/hooks/bz/BugzillaItsFacade.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/util/IssueExtractor.class and old/com/googlesource/gerrit/plugins/hooks/util/IssueExtractor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/util/PropertyExtractor.class and old/com/googlesource/gerrit/plugins/hooks/util/PropertyExtractor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/validation/ItsValidateComment.class and old/com/googlesource/gerrit/plugins/hooks/validation/ItsValidateComment.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/action/LogEvent.class and old/com/googlesource/gerrit/plugins/hooks/workflow/action/LogEvent.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/ActionController.class and old/com/googlesource/gerrit/plugins/hooks/workflow/ActionController.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/ActionExecutor.class and old/com/googlesource/gerrit/plugins/hooks/workflow/ActionExecutor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState.class and old/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState$Transition.class and old/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState$Transition.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/RuleBase.class and old/com/googlesource/gerrit/plugins/hooks/workflow/RuleBase.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/Rule.class and old/com/googlesource/gerrit/plugins/hooks/workflow/Rule.class differ
Binary files new/com/j2bugzilla/rpc/BugzillaVersion.class and old/com/j2bugzilla/rpc/BugzillaVersion.class differ
Binary files new/com/j2bugzilla/rpc/CommentBug.class and old/com/j2bugzilla/rpc/CommentBug.class differ
Binary files new/com/j2bugzilla/rpc/GetBug.class and old/com/j2bugzilla/rpc/GetBug.class differ
Binary files new/com/j2bugzilla/rpc/GetLegalValues.class and old/com/j2bugzilla/rpc/GetLegalValues.class differ
Binary files new/com/j2bugzilla/rpc/LogIn.class and old/com/j2bugzilla/rpc/LogIn.class differ
Binary files new/com/j2bugzilla/rpc/LogOut.class and old/com/j2bugzilla/rpc/LogOut.class differ
Binary files new/com/j2bugzilla/rpc/UpdateBug.class and old/com/j2bugzilla/rpc/UpdateBug.class differ
Only in new/: hooks-bugzilla_new-01f21a43649f50302f9d4ff2cff7d7bad6a4ffb9.jar
Only in old/: hooks-bugzilla_old-bd5dce77c691d823caafd79df7a58a7a992c0687.jar
diff -r new/META-INF/MANIFEST.MF old/META-INF/MANIFEST.MF
3c3

< Implementation-Version: 2.8.1.2

Implementation-Version: 2.8.1

8c8

< Gerrit-ApiVersion: 2.8.1.2

Gerrit-ApiVersion: 2.8.1

11c11

< Build-Jdk: 1.7.0_05

Build-Jdk: 1.6.0_24

diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.properties old/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.properties
2,3c2,3
< #Fri May 16 18:01:44 CEST 2014

< version=2.8.1.2

#Mon Jan 27 13:37:07 CET 2014
version=2.8.1

diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.xml old/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.xml
24c24

< <version>2.8.1.2</version>

<version>2.8.1</version>

99c99

< <version>2.8.1</version>

<version>${project.version}</version>

104c104

< <version>2.3-SNAPSHOT</version>

<version>2.2</version>

diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.its/hooks-its/pom.properties old/META-INF/maven/com.googlesource.gerrit.plugins.its/hooks-its/pom.properties
2c2

< #Fri May 16 18:01:33 CEST 2014

#Mon Jan 27 13:35:30 CET 2014

diff -r new/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.properties old/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.properties
1,5c1,5
< #Generated by Maven
< #Fri May 16 18:01:15 CEST 2014
< version=2.3-SNAPSHOT
< groupId=com.j2bugzilla

< artifactId=j2bugzilla

#Generated by Maven
#Thu Jun 13 00:23:50 EDT 2013
version=2.2
groupId=com.j2bugzilla
artifactId=j2bugzilla

diff -r new/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.xml old/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.xml
12c12

< <version>2.3-SNAPSHOT</version>

<version>2.2</version>

114d113
< <!--
129d127
< -->
160c158

< </project>

</project>

\ No newline at end of file

ehm... ??

aric wrote:

@dan, Yeah, from the contents of the .jar I can't actually see inside the .class files to see what was changed to make the gerrit -> Buzilla posting start working. I am considering emailing qchris, but I thought it more polite to ask here first. I'd like to help make sure that the its-bugzilla plugin continues being viable going forward. Currently it's support is stuck at gerrit 2.9. Though I'm no java programmer, I'm imagining that If I can lay the task out clearly that I'll be able to enlist someone.

Created attachment 16426
Extracted jar

Attached:

^ Try that? Used http://jd.benow.ca/

(In reply to Aric Gardner from comment #19)

Hi guys, I would like to get this change pushed upstream to
https://gerrit.googlesource.com/plugins/its-bugzilla/
I only see a binary diff in gerrit, can I get a link to the source code
changed?

A patch to hooks-bugzilla is not sufficient, I had to patch j2bugzilla too.

I sent you the source files by email.

(The fix is an ugly hack. I do not like it.)

(Meanwhile (in June) j2bugzilla saw an update too. Maybe that allows for
an easier fix?)

aric wrote:

Thank you @Christian!