Page MenuHomePhabricator
Create Task
Maniphest T214539

Monitor dbstore1002 migration to a multi instance setup on dbstore100[3-5] - WMDE analytics scripts integrity
Closed, ResolvedPublic
Actions

Description

  • Ensure that all WMDE WD usage analytics scripts remain operational in the face of changes announced in T210478;
  • Critical: April 2019.
  • Probably no intervention will be needed (no cross-wiki joins are performed in our scripts).

Related Objects

Event Timeline

GoranSMilovanovic created this task.Jan 24 2019, 12:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2019, 12:47 AM
Reedy renamed this task from Monitor dbstore1002 migrationto a multi instance setup on dbstore100[3-5] - WMDE analytics scripts integrity to Monitor dbstore1002 migration to a multi instance setup on dbstore100[3-5] - WMDE analytics scripts integrity.Jan 24 2019, 2:25 PM
GoranSMilovanovic mentioned this in T213894: Modify analytics/wmde/scripts repo to work with the new analytics multisource db setup.Feb 7 2019, 11:13 PM
GoranSMilovanovic triaged this task as High priority.Feb 22 2019, 11:19 AM
GoranSMilovanovic moved this task from Incoming to Prioritized on the User-GoranSMilovanovic board.
GoranSMilovanovic mentioned this in T217156: Pyspark2 fails to read.csv when run with spark2-submit.Feb 27 2019, 11:13 AM
GoranSMilovanovic added a comment.EditedMar 11 2019, 7:16 PM
  • New WDCM_Sqoop.R script is currently being tested.

The following sqoop command:

/usr/bin/sqoop import --connect jdbc:mysql://s1-analytics-replica.eqiad.wmnet/enwiki:3311 --password-file /user/goransm/mysql-analytics-research-client-pw.txt --username research -m 16 --query "select * from wbc_entity_usage where \$CONDITIONS" --split-by eu_row_id --as-avrodatafile --target-dir /user/goransm/wdcmsqoop/wdcm_clients_wb_entity_usage/wiki_db=enwiki --delete-target-dir>

fails to deliver:

Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j-jcl.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j-jdk14.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j-nop.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j-simple.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/03/11 19:05:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.15.0
19/03/11 19:05:12 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
19/03/11 19:05:12 INFO tool.CodeGenTool: Beginning code generation
19/03/11 19:07:23 ERROR manager.SqlManager: Error executing statement: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2193)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2225)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2024)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:904)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
        at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
        at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
        at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
        at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:253)
        at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:336)
        at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1858)
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1657)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
        ... 33 more
19/03/11 19:07:23 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1663)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

Inspecting the issue now.

@JAllemandou @chelsyx Any ideas?

GoranSMilovanovic added subscribers: • chelsyx, JAllemandou.Mar 11 2019, 7:18 PM
GoranSMilovanovic added a subscriber: elukey.Mar 11 2019, 7:39 PM

Thanks to @JAllemandou and @elukey the sqoop problem is now resolved;

  • testing WDCM_Sqoop.R now.
GoranSMilovanovic lowered the priority of this task from High to Medium.Mar 11 2019, 9:09 PM
GoranSMilovanovic added a subscriber: • RazShuty.Mar 12 2019, 1:08 PM

@RazShuty

  • Test failed:
  • R wrappers around 'mysql-analytics' calls cannot tryCatch() the target's internal errors;
  • the script WDCM_Sqoop.R needs a simple re-factoring to avoid encountering such situations;
  • estimate: re-factor today, test in the evening/night, deploy to production March 13, 2019.
GoranSMilovanovic added a comment.Mar 12 2019, 9:00 PM
  • refactored; test again.
GoranSMilovanovic added a comment.Mar 13 2019, 9:58 AM

@RazShuty

  • test complete;
  • success;
  • next step: add logging procedures, deploy to production.
GoranSMilovanovic closed this task as Resolved.Mar 13 2019, 10:44 AM
  • deployed in production;
  • closing the ticket.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL