Page MenuHomePhabricator
Create Task
Maniphest T141525

Wikidata Special:EntityData not being filled with new data
Closed, ResolvedPublic
Actions

Description

https://grafana.wikimedia.org/dashboard/db/wikidata-special-entitydata

The user running the script to add the data does not have access to hive / hadoop!

addshore@stat1002:/a/analytics-wmde/src/scripts/src/wikidata$ sudo -u analytics-wmde php specialEntityData.php
2016-07-28 14:25:42 START /a/analytics-wmde/src/scripts/src/wikidata/specialEntityData.php
addshore@stat1002:/a/analytics-wmde/src/scripts/src/wikidata$ sudo -u analytics-wmde cat entitydata_errors_1.txt
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.
FAILED: SemanticException Unable to determine if hdfs://analytics-hadoop/wmf/data/wmf/webrequest is encrypted: org.apache.hadoop.security.AccessControlException: Permission denied: user=analytics-wmde, access=READ, inode="/wmf/data/wmf/webrequest":hdfs:analytics-privatedata-users:drwxr-x---
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6605)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6587)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6512)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:9122)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1608)
        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEZForPath(AuthorizationProviderProxyClientProtocol.java:926)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1343)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

Details

SubjectRepoBranchLines +/-
Create wikidata/specialentitydata_metrics coordinatoranalytics/refinerymaster+332 -0
Create WikidataSpecialEntityDataMetricsanalytics/refinery/sourcemaster+143 -0
Remove specialEntityData scriptanalytics/wmde/scriptsproduction+0 -101
Remove specialEntityData scriptanalytics/wmde/scriptsmaster+0 -101
Remove analytics-wmde from analytics cluster, improve docs around analytics cluster usersoperations/puppetproduction+16 -11
Stop running wikidata/specialEntityData.php in cronanalytics/wmde/scriptsproduction+0 -1
Stop running wikidata/specialEntityData.php in cronanalytics/wmde/scriptsmaster+0 -1
Add analytics-wmde user to role::analytics_cluster::usersoperations/puppetproduction+38 -19
Customize query in gerrit

Related Objects

Event Timeline

Addshore created this task.Jul 28 2016, 2:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2016, 2:31 PM
Addshore triaged this task as High priority.Jul 28 2016, 2:31 PM
Addshore moved this task from Incoming to Doing on the WMDE-Analytics-Engineering board.
Addshore moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-Addshore board.
gerritbot subscribed.Jul 28 2016, 2:51 PM

Change 301610 had a related patch set uploaded (by Addshore):
Add analytics-wmde user to role::analytics_cluster::users

https://gerrit.wikimedia.org/r/301610

Addshore mentioned this in rOPUPd0d5c229afff: Add analytics-wmde user to role::analytics_cluster::users.Jul 28 2016, 2:58 PM
Addshore mentioned this in rOPUP74ab1c3fe22a: Add analytics-wmde user to role::analytics_cluster::users.Jul 28 2016, 3:22 PM
Addshore mentioned this in rOPUP627c635f189c: Add analytics-wmde user to role::analytics_cluster::users.
Addshore mentioned this in rOPUP25f97389c74d: Add analytics-wmde user to role::analytics_cluster::users.Jul 28 2016, 3:28 PM
Addshore mentioned this in rOPUP103562d96194: Add analytics-wmde user to role::analytics_cluster::users.Jul 28 2016, 3:32 PM
Ottomata mentioned this in rOPUPec208464e520: Add analytics-wmde user to role::analytics_cluster::users.Jul 28 2016, 3:37 PM
Addshore mentioned this in rOPUPa4db21f16410: Add analytics-wmde user to role::analytics_cluster::users.Jul 28 2016, 3:37 PM
gerritbot added a comment.Jul 28 2016, 3:38 PM

Change 301610 merged by Ottomata:
Add analytics-wmde user to role::analytics_cluster::users

https://gerrit.wikimedia.org/r/301610

Addshore added a subscriber: Ottomata.Jul 28 2016, 4:38 PM

After discussion with @Ottomata access to the webrequest data has never been given to a system user before (of which analytics-wmde is one).

Thus the best way forward would be to convert the script to an oozie job to be run in the same way as the ArticlePlaceholder oozie job.
(This would have been in the long term plan anyway)

gerritbot added a comment.Jul 28 2016, 4:38 PM

Change 301637 had a related patch set uploaded (by Addshore):
Stop running wikidata/specialEntityData.php in cron

https://gerrit.wikimedia.org/r/301637

Addshore mentioned this in rADES6623d0d82e25: Stop running wikidata/specialEntityData.php in cron.Jul 28 2016, 4:38 PM
gerritbot added a comment.Jul 28 2016, 4:39 PM

Change 301638 had a related patch set uploaded (by Addshore):
Stop running wikidata/specialEntityData.php in cron

https://gerrit.wikimedia.org/r/301638

gerritbot added a comment.Jul 28 2016, 4:39 PM

Change 301637 merged by jenkins-bot:
Stop running wikidata/specialEntityData.php in cron

https://gerrit.wikimedia.org/r/301637

gerritbot added a comment.Jul 28 2016, 4:39 PM

Change 301638 merged by jenkins-bot:
Stop running wikidata/specialEntityData.php in cron

https://gerrit.wikimedia.org/r/301638

gerritbot added a comment.Jul 28 2016, 4:39 PM

Change 301639 had a related patch set uploaded (by Ottomata):
Remove analytics-wmde from analytics cluster, improve docs around analytics cluster users

https://gerrit.wikimedia.org/r/301639

Addshore mentioned this in rADES450f4f36ed5e: Stop running wikidata/specialEntityData.php in cron.Jul 28 2016, 4:40 PM
gerritbot added a comment.Jul 28 2016, 4:41 PM

Change 301639 merged by Ottomata:
Remove analytics-wmde from analytics cluster, improve docs around analytics cluster users

https://gerrit.wikimedia.org/r/301639

Ottomata mentioned this in rOPUP5b9d9d65c795: Remove analytics-wmde from analytics cluster, improve docs around analytics….Jul 28 2016, 4:43 PM
gerritbot added a comment.Jul 28 2016, 6:34 PM

Change 301657 had a related patch set uploaded (by Addshore):
WIP Create WikidataSpecialEntityDataMetrics

https://gerrit.wikimedia.org/r/301657

gerritbot added a comment.Jul 28 2016, 6:49 PM

Change 301661 had a related patch set uploaded (by Addshore):
WIP Create wikidata/specialentitydata_metrics coordinator

https://gerrit.wikimedia.org/r/301661

Addshore moved this task from Back Burner 🏛️ to Active 🚁 on the User-Addshore board.Jul 31 2016, 4:28 PM
gerritbot added a comment.Jul 31 2016, 5:53 PM

Change 302120 had a related patch set uploaded (by Addshore):
Remove specialEntityData script

https://gerrit.wikimedia.org/r/302120

gerritbot added a comment.Jul 31 2016, 5:53 PM

Change 302121 had a related patch set uploaded (by Addshore):
Remove specialEntityData script

https://gerrit.wikimedia.org/r/302121

Addshore mentioned this in rADESa1dec9adf0d5: Remove specialEntityData script.Jul 31 2016, 6:00 PM
Addshore mentioned this in rADESf4bc1ba70a5b: Remove specialEntityData script.
Addshore mentioned this in rADES034a839790ea: Remove specialEntityData script.
Addshore mentioned this in rADESa77664793e4c: Remove specialEntityData script.
gerritbot added a comment.Jul 31 2016, 6:16 PM

Change 302120 merged by jenkins-bot:
Remove specialEntityData script

https://gerrit.wikimedia.org/r/302120

gerritbot added a comment.Jul 31 2016, 6:16 PM

Change 302121 merged by jenkins-bot:
Remove specialEntityData script

https://gerrit.wikimedia.org/r/302121

Addshore moved this task from incoming to ready to go on the Wikidata board.Jul 31 2016, 8:18 PM
gerritbot added a comment.Aug 9 2016, 3:26 PM

Change 301657 merged by Nuria:
Create WikidataSpecialEntityDataMetrics

https://gerrit.wikimedia.org/r/301657

gerritbot added a comment.Aug 9 2016, 4:13 PM

Change 301661 merged by Nuria:
Create wikidata/specialentitydata_metrics coordinator

https://gerrit.wikimedia.org/r/301661

Addshore closed this task as Resolved.Aug 21 2016, 11:10 AM
Addshore moved this task from Active 🚁 to Closing ✔️ on the User-Addshore board.
Addshore moved this task from Doing to Done on the WMDE-Analytics-Engineering board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL