Page MenuHomePhabricator

gerrit->bugzilla notification broken (due to Bugzilla 4.4.3 changes in token/session passing)
Closed, ResolvedPublic


this is a test ticket for the gerrit->BZ notification bot/gerrit plugin

Version: wmf-deployment
Severity: major



Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 3:14 AM
bzimport set Reference to bz65370.

Caused by: org.apache.xmlrpc.XmlRpcException: You must log in before using this part of Bugzilla.

[2014-05-15 20:49:56,495] ERROR com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor : Error while executing action status PATCH_TO_REVIEW com.j2bugzilla.base.BugzillaException: An unknown error was encountered; fault code: 410

at com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor.execute(
at com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor.execute(
at com.googlesource.gerrit.plugins.hooks.workflow.ActionController.onChangeEvent(

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
at org.eclipse.jetty.util.thread.QueuedThreadPool$

Caused by: com.j2bugzilla.base.BugzillaException: An unknown error was encountered; fault code: 410

at com.j2bugzilla.base.XmlExceptionHandler.handleFault(
at com.j2bugzilla.base.BugzillaConnector.executeMethod(
... 56 more

Caused by: org.apache.xmlrpc.XmlRpcException: You must log in before using this part of Bugzilla.

at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(
at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(
at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(
at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(
at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(
at org.apache.xmlrpc.client.XmlRpcClient.execute(
at org.apache.xmlrpc.client.XmlRpcClient.execute(
at org.apache.xmlrpc.client.XmlRpcClient.execute(
at com.j2bugzilla.base.BugzillaConnector.executeMethod(
... 62 more

12:40 < JohnLewis> Hm. Gerrit not notifying Bugzilla of patches?
12:41 < mutante> JohnLewis: do you know since when?
12:42 < JohnLewis> mutante: was uploaded yesterday and the Bugzilla bug was not notified. That's as far as I know.
12:43 < JohnLewis> I just patched a bug and that wasn't updated so it's still an on-going issue for reference

12:50 < mutante>
12:50 < mutante> "A bot will automatically notify Bugzilla ... "

12:52 < mutante> ^d: ideas why that could have changed recently?
12:52 < ^d> godog restarting gerrit?

12:53 < mutante> i hear it broke within the last 9 hours
12:53 < mutante> ah
12:53 < andre> so the plugin in Gerrit is
12:54 < andre
> and qchris set it up once upon a time
12:55 < andre__> very last activity in Bugzilla was Gerrit Notification Bot ( 2014-05-15 11:41:52 UTC in

13:01 < mutante> the bugzilla plugin is listed as enabled in gerrit
13:02 < mutante> hooks-bugzilla2.8.1 Enabled .. hrmm
13:02 < mutante>
13:03 < mutante> even
13:03 < mutante> "Bugzilla credentials and connectivity details are asked and verified during the Gerrit init."
13:03 < mutante> that?

13:12 < andre__> mutante,

13:13 < ^d> fwiw, gerrit is connecting.
13:13 < ^d> [2014-05-15 12:57:28,709] INFO : Bugzilla is configured as ITS
13:13 < ^d> [2014-05-15 12:57:30,106] INFO : Connected to as, userid=16761

13:13 < ^d> [2014-05-15 12:57:30,646] INFO : Connected to Bugzilla at, reported version is 4.4.4

13:58 < andre__> Bugzilla DID change some cookie/login related stuff between 4.4.1 and 4.4.4, but we upgraded to 4.4.4 more than one day ago

13:20 < ^d> Hmm, I reloaded the plugin and cleared the error log.
13:21 < ^d> Once again, Connected to as, userid=16761
13:21 < ^d> Wonder if it'll last.

..but then it didn't

Gerrit log excerpt above from /var/lib/gerrit2/review_site/log/error_log

There were firewall changes but timing does not fit, as last activity in Bugzilla by Gerrit Notification Bot ( was 2014-05-15 11:41:52 UTC in bug 65225 comment 3.

<mutante> ^d: did you know about openjdk being upgraded btw?
<mutante> @ytterbium:/var/log# grep jdk /var/log/apt/history.log

<andre> plugin in Gerrit is
> for the records, the quick'n'dirty check if it's running again would be any patch created in Gerrit and correctly refering to a Bugzilla ID, which would create a result shown for (that query covers the last one hour)

yea, regarding the openjdk update, no idea if it's related but:

Start-Date: 2014-05-15 11:48:08
Commandline: apt-get install openjdk-6-jdk
Install: openjdk-6-jdk:amd64 (6b31-1.13.3-1ubuntu1~
Upgrade: icedtea-6-jre-cacao:amd64 (6b30-1.13.1-1ubuntu2~, 6b31-1.13.3-1ubuntu1~, openjdk-6-jre-lib:amd64 (6b30-1.13.1-1ubuntu2~, 6b31-1.13.3-1ubuntu1~, openjdk-6-jre-headless:amd64 (6b30-1.13.1-1ubuntu2~, 6b31-1.13.3-1ubuntu1~, icedtea-6-jre-jamvm:amd64 (6b30-1.13.1-1ubuntu2~, 6b31-1.13.3-1ubuntu1~, openjdk-6-jre:amd64 (6b30-1.13.1-1ubuntu2~, 6b31-1.13.3-1ubuntu1~
End-Date: 2014-05-15 11:48:13

Start-Date: 2014-05-15 11:49:39
Commandline: apt-get purge openjdk-6-jdk
Purge: openjdk-6-jdk:amd64 (6b31-1.13.3-1ubuntu1~
End-Date: 2014-05-15 11:49:39

Start-Date: 2014-05-15 11:49:57
Commandline: apt-get install openjdk-7-jdk
Install: openjdk-7-jdk:amd64 (7u25-2.3.10-1ubuntu0.12.04.2), openjdk-7-jre-lib:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, automatic)
Downgrade: openjdk-7-jre:amd64 (7u51-2.4.4-0ubuntu0.12.04.2, 7u25-2.3.10-1ubuntu0.12.04.2), openjdk-7-jre-headless:amd64 (7u51-2.4.4-0ubuntu0.12.04.2, 7u25-2.3.10-1ubuntu0.12.04.2)
Remove: icedtea-7-jre-jamvm:amd64 (7u51-2.4.4-0ubuntu0.12.04.2)
End-Date: 2014-05-15 11:50:02

Start-Date: 2014-05-15 11:51:04
Commandline: apt-get remove openjdk-7-jdk
Remove: openjdk-7-jdk:amd64 (7u25-2.3.10-1ubuntu0.12.04.2)
End-Date: 2014-05-15 11:51:04

Start-Date: 2014-05-15 11:53:14
Commandline: apt-get install openjdk-7-jre=7u55-2.4.7-1ubuntu1~ openjdk-7-jdk=7u55-2.4.7-1ubuntu1~ openjdk-7-jre-headless=7u55-2.4.7-1ubuntu1~
Install: openjdk-7-jdk:amd64 (7u55-2.4.7-1ubuntu1~
Upgrade: openjdk-7-jre:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, 7u55-2.4.7-1ubuntu1~, openjdk-7-jre-headless:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, 7u55-2.4.7-1ubuntu1~
End-Date: 2014-05-15 11:53:18

11:56 godog: installed openjdk-7-jdk on ytterbium to attempt gerrit thread dump

(In reply to Daniel Zahn from comment #5)

yea, regarding the openjdk update, no idea if it's related but:

Start-Date: 2014-05-15 11:48:08
Commandline: apt-get install openjdk-6-jdk

Assuming that time is UTC, that fits reasonably well to the last activity in Bugzilla at 11:41:52 UTC.

(In reply to Daniel Zahn from comment #2)

13:58 < andre__> Bugzilla DID change some cookie/login related stuff between
4.4.1 and 4.4.4, but we upgraded to 4.4.4 more than one day ago

To clarify what I refered to, quoting the 4.4.3 release notes:
"The User.login WebService method now also returns a token argument containing a login token which you can use in subsequent calls to authenticate. For security reasons, this method no longer generates login cookies. ( )"

The openjdk change time fits really, really nicely. But I it should
(yes. Vague :-/ ) not affect us here, as Gerrit uses the java implementation
that's referenced in it's config file (key: container.javaHome), which is


I checked, and this jvm seems basically functional still.

I tried a local bugzilla-4.4.4 instance, and can reproduce the issue there.
(Bugzilla 4.2.4 works nicely)

Prodding down that line, it seems comment #8 is relevant.
(Bugzilla's 410 error is "login_required")

j2bugzilla (the third part bugzilla xmlrpc library used in hooks-bugzilla) is
still current, and has not seen adaptions for bugzilla 4.4.3.
So I'll see if j2bugzilla needs fixing for the login change of bugzilla 4.4.3.

(In reply to christian from comment #9)

I tried a local bugzilla-4.4.4 instance, and can reproduce the issue there.

Weird, because we upgraded Bugzilla to 4.4.4 already on 2014-04-22 (bug 61287 comment 5) and the notifications still worked until yesterday.

(Oops, potentially confusing last comment: We skipped the broken 4.4.3 and upgraded from 4.4.1 or 4.4.2 directly to 4.4.4)

Had a chat with qchris who asked if downgrading Bugzilla to 4.4.2 is possible (otherwise fixing would take a bit longer).

We're still wondering why the problem only arose yesterday because our 4.4.4 upgrade was three weeks ago (cached login cookie? shrug).

My guess is that reverting (for ) should be sufficient.

As I got to leave in 15min (will be back later tonight) I'd love to get an opinion from mutante.

Change 133589 had a related patch set uploaded by QChris:
test gerrit->bz notification

Deployed a patched j2bugzilla/hooks-bugzilla plugin that allows token passing
for bugzilla-4.4.4 instead of using sessions.

Seems to be working again (see comment 13).

Change 133732 had a related patch set uploaded by QChris:
Update hoobs-bugzilla to 5edd392d926daaa58917b1c8bb174cdb022e4c76

qchris: Thank you, thank you, thank you!

Change 133589 abandoned by QChris:
test gerrit->bz notification

gerrit -> bugzilla notifications are working again.

Change 133732 merged by Chad:
Update hooks-bugzilla to 5edd392d926daaa58917b1c8bb174cdb022e4c76

aric wrote:

Hi guys, I would like to get this change pushed upstream to
I only see a binary diff in gerrit, can I get a link to the source code changed?



Binary files new/com/googlesource/gerrit/plugins/hooks/bz/BugzillaClient.class and old/com/googlesource/gerrit/plugins/hooks/bz/BugzillaClient.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/bz/BugzillaItsFacade.class and old/com/googlesource/gerrit/plugins/hooks/bz/BugzillaItsFacade.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/util/IssueExtractor.class and old/com/googlesource/gerrit/plugins/hooks/util/IssueExtractor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/util/PropertyExtractor.class and old/com/googlesource/gerrit/plugins/hooks/util/PropertyExtractor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/validation/ItsValidateComment.class and old/com/googlesource/gerrit/plugins/hooks/validation/ItsValidateComment.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/action/LogEvent.class and old/com/googlesource/gerrit/plugins/hooks/workflow/action/LogEvent.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/ActionController.class and old/com/googlesource/gerrit/plugins/hooks/workflow/ActionController.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/ActionExecutor.class and old/com/googlesource/gerrit/plugins/hooks/workflow/ActionExecutor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState.class and old/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState$Transition.class and old/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState$Transition.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/RuleBase.class and old/com/googlesource/gerrit/plugins/hooks/workflow/RuleBase.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/Rule.class and old/com/googlesource/gerrit/plugins/hooks/workflow/Rule.class differ
Binary files new/com/j2bugzilla/rpc/BugzillaVersion.class and old/com/j2bugzilla/rpc/BugzillaVersion.class differ
Binary files new/com/j2bugzilla/rpc/CommentBug.class and old/com/j2bugzilla/rpc/CommentBug.class differ
Binary files new/com/j2bugzilla/rpc/GetBug.class and old/com/j2bugzilla/rpc/GetBug.class differ
Binary files new/com/j2bugzilla/rpc/GetLegalValues.class and old/com/j2bugzilla/rpc/GetLegalValues.class differ
Binary files new/com/j2bugzilla/rpc/LogIn.class and old/com/j2bugzilla/rpc/LogIn.class differ
Binary files new/com/j2bugzilla/rpc/LogOut.class and old/com/j2bugzilla/rpc/LogOut.class differ
Binary files new/com/j2bugzilla/rpc/UpdateBug.class and old/com/j2bugzilla/rpc/UpdateBug.class differ
Only in new/: hooks-bugzilla_new-01f21a43649f50302f9d4ff2cff7d7bad6a4ffb9.jar
Only in old/: hooks-bugzilla_old-bd5dce77c691d823caafd79df7a58a7a992c0687.jar

< Implementation-Version:

Implementation-Version: 2.8.1


< Gerrit-ApiVersion:

Gerrit-ApiVersion: 2.8.1


< Build-Jdk: 1.7.0_05

Build-Jdk: 1.6.0_24

diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/ old/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/
< #Fri May 16 18:01:44 CEST 2014

< version=

#Mon Jan 27 13:37:07 CET 2014

diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.xml old/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.xml

< <version></version>



< <version>2.8.1</version>



< <version>2.3-SNAPSHOT</version>


diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.its/hooks-its/ old/META-INF/maven/com.googlesource.gerrit.plugins.its/hooks-its/

< #Fri May 16 18:01:33 CEST 2014

#Mon Jan 27 13:35:30 CET 2014

diff -r new/META-INF/maven/com.j2bugzilla/j2bugzilla/ old/META-INF/maven/com.j2bugzilla/j2bugzilla/
< #Generated by Maven
< #Fri May 16 18:01:15 CEST 2014
< version=2.3-SNAPSHOT
< groupId=com.j2bugzilla

< artifactId=j2bugzilla

#Generated by Maven
#Thu Jun 13 00:23:50 EDT 2013

diff -r new/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.xml old/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.xml

< <version>2.3-SNAPSHOT</version>


< <!--
< -->

< </project>


\ No newline at end of file

ehm... ??

aric wrote:

@dan, Yeah, from the contents of the .jar I can't actually see inside the .class files to see what was changed to make the gerrit -> Buzilla posting start working. I am considering emailing qchris, but I thought it more polite to ask here first. I'd like to help make sure that the its-bugzilla plugin continues being viable going forward. Currently it's support is stuck at gerrit 2.9. Though I'm no java programmer, I'm imagining that If I can lay the task out clearly that I'll be able to enlist someone.

Created attachment 16426
Extracted jar


(In reply to Aric Gardner from comment #19)

Hi guys, I would like to get this change pushed upstream to
I only see a binary diff in gerrit, can I get a link to the source code

A patch to hooks-bugzilla is not sufficient, I had to patch j2bugzilla too.

I sent you the source files by email.

(The fix is an ugly hack. I do not like it.)

(Meanwhile (in June) j2bugzilla saw an update too. Maybe that allows for
an easier fix?)