Page MenuHomePhabricator
Create Task
Maniphest T196869

In Gerrit some users are reporting problems saving there preferences
Closed, ResolvedPublic
Actions

Description

In my Gerrit preferences ( https://gerrit.wikimedia.org/r/settings/ ) I have tried to set a status with an emoji and get an internal server error. On the backend:

[HTTP-5531] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in PUT /r/accounts/self/status
java.io.IOException: Permission denied
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createNewFile(File.java:1012)
	at org.eclipse.jgit.util.FS_POSIX.createNewFile(FS_POSIX.java:367)
	at org.eclipse.jgit.internal.storage.file.LockFile.lock(LockFile.java:171)
	at org.eclipse.jgit.internal.storage.file.RefDirectoryUpdate.tryLock(RefDirectoryUpdate.java:86)
	at org.eclipse.jgit.lib.RefUpdate.updateImpl(RefUpdate.java:708)
	at org.eclipse.jgit.lib.RefUpdate.update(RefUpdate.java:582)
	at org.eclipse.jgit.lib.RefUpdate.update(RefUpdate.java:563)
	at com.google.gerrit.server.git.VersionedMetaData$1.updateRef(VersionedMetaData.java:384)
	at com.google.gerrit.server.git.VersionedMetaData$1.commitAt(VersionedMetaData.java:342)
	at com.google.gerrit.server.git.VersionedMetaData$1.commit(VersionedMetaData.java:334)
	at com.google.gerrit.server.git.VersionedMetaData.commit(VersionedMetaData.java:188)
	at com.google.gerrit.server.account.AccountConfig.commit(AccountConfig.java:172)
	at com.google.gerrit.server.account.AccountsUpdate.commit(AccountsUpdate.java:348)
	at com.google.gerrit.server.account.AccountsUpdate.commit(AccountsUpdate.java:342)
	at com.google.gerrit.server.account.AccountsUpdate.update(AccountsUpdate.java:244)
	at com.google.gerrit.server.account.AccountsUpdate.update(AccountsUpdate.java:220)
	at com.google.gerrit.server.account.PutStatus.apply(PutStatus.java:83)
	at com.google.gerrit.server.account.PutStatus.apply(PutStatus.java:70)
	at com.google.gerrit.server.account.PutStatus.apply(PutStatus.java:37)
	at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:405)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:483)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:100)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:122)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:69)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133)
	at com.ericsson.gerrit.plugins.goimport.GoImportFilter.doFilter(GoImportFilter.java:138)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:129)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:69)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:534)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

Related Objects

Event Timeline

hashar created this task.Jun 11 2018, 8:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 11 2018, 8:08 AM
hashar added a project: Release-Engineering-Team (Kanban).Jun 11 2018, 8:19 AM

Gerrit issue #3157 - Error in POST /accounts/self/preferences hints at permissions issue in All-Projects.git. And indeed looking at the repository on the server, there are a bunch of objects and references owned by root.

$ ls -la /srv/gerrit/git/All-Users.git/refs/users/24/24
-rw-r--r-- 1 root root 41 Jun  8 17:17 /srv/gerrit/git/All-Users.git/refs/users/24/24
$

Part of Gerrit 2.15 upgrade ( T177201 ) must have been made as root, specially the part that migrate some data from the database to git repositories.

Example:

June 8th 17:17:11 Remove "My Drafts" menu items

--- a/preferences.config
+++ b/preferences.config
-[my "Drafts"]
-       url = "#/q/owner:self+is:draft"

June 8th 17:17:03 Update account

--- /dev/null
+++ b/account.config
@@ -0,0 +1,3 @@
+[account]
+       fullName = Hashar
+       preferredEmail = hashar@free.fr

There are a lot of root owned objects in /srv/gerrit/git/All-Users.git/objects/ . All from around June 8th.

hashar added a comment.Jun 11 2018, 8:21 AM

So I think it is all about changing rights of root:root files under /srv/gerrit/git/All-Users.git to gerrit2:gerrit2

Legoktm subscribed.Jun 11 2018, 8:32 AM

Same as T196862: Some users are reporting problems saving there preferences maybe?

hashar triaged this task as Unbreak Now! priority.Jun 11 2018, 8:34 AM
hashar merged a task: T196862: Some users are reporting problems saving there preferences.
hashar added subscribers: Paladox, TerraCodes, Liuxinyu970226, • demon.

Yup. I have merged T196862 in there since I already have all the stacktraces. Thanks Paladox :]

hashar renamed this task from PUT "/r/accounts/self/status" to In Gerrit some users are reporting problems saving there preferences.Jun 11 2018, 8:35 AM
hashar added a project: SRE.Jun 11 2018, 10:57 AM

We need SRE to fix up permissions on cobalt.wikimedia.org

Files under /srv/gerrit/git/All-Users.git/ being owned by root:root should instead be owned by gerrit2:gerrit2.

Paladox added a comment.EditedJun 11 2018, 10:58 AM

sudo chrown gerrit2:gerrit2 /srv/gerrit/git (might as well do that to make sure all the repos are owned by gerrit2)

Dzahn subscribed.Jun 11 2018, 11:09 AM
In T196869#4271568, @hashar wrote:

We need SRE to fix up permissions on cobalt.wikimedia.org

Files under /srv/gerrit/git/All-Users.git/ being owned by root:root should instead be owned by gerrit2:gerrit2.

I checked cobalt and it looks already done meanwhile.

4.0K drwxr-xr-x 7 gerrit2 gerrit2 4.0K Jun 11 09:59 All-Users.git

Paladox added a comment.Jun 11 2018, 11:10 AM

@Dzahn though the objects inside that folder could be owned by root.

Dzahn added a comment.Jun 11 2018, 11:16 AM
In T196869#4271568, @hashar wrote:

We need SRE to fix up permissions on cobalt.wikimedia.org

Fixed.

root@cobalt:/srv/gerrit/git/All-Users.git# find . -uid 0 -exec chown gerrit2:gerrit2 {} \;
root@cobalt:/srv/gerrit/git/All-Users.git# find . -uid 0
root@cobalt:/srv/gerrit/git/All-Users.git#
hashar assigned this task to Dzahn.Jun 11 2018, 11:35 AM

Fixed mutante ! Danke.

My test case was to go to https://gerrit.wikimedia.org/r/settings/ and try to fill something in the Status field. At firstI thought it could be related to using an emoji, but that was really just a permission problem.

Dzahn closed this task as Resolved.Jun 11 2018, 12:08 PM
Liuxinyu970226 unsubscribed.Jun 11 2018, 1:26 PM
Paladox mentioned this in T196898: public key error when fetching from Wikimedia remotes (due to uppercase user name).Jun 11 2018, 1:53 PM
• Vvjjkkii renamed this task from In Gerrit some users are reporting problems saving there preferences to qabaaaaaaa.Jul 1 2018, 1:05 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed Dzahn as the assignee of this task.
• Vvjjkkii lowered the priority of this task from Unbreak Now! to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from qabaaaaaaa to In Gerrit some users are reporting problems saving there preferences.Jul 2 2018, 12:38 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to Dzahn.
CommunityTechBot raised the priority of this task from High to Unbreak Now!.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added a subscriber: Aklapper.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL