Page MenuHomePhabricator
Create Task
Maniphest T246560

"corhist" tool having problems running on 2020 Kubernetes cluster
Closed, ResolvedPublic
Actions

Description

In T246559: Investigate CrashLoopBackoff Pods on 2020 Kubernetes cluster I was looking into why several tools were failing after being migrated to the 2020 Kubernetes cluster. This bug report is a follow up to the things I documented in T246559#5930151.

It looks like there are some logic and/or java class version errors that need help.

$ kubectl logs -f corhist-84fddf9cbb-w2hks

2020-03-01T06:04:25.922+0000  SEVERE  org.wikidata.history.sparql.NumericValueFactory$PropertyIRI; local class incompatible: stream classdesc serialVersionUID = -7130349094387029343, local class serialVersionUID = -3463140113163263022
java.io.InvalidClassException: org.wikidata.history.sparql.NumericValueFactory$PropertyIRI; local class incompatible: stream classdesc serialVersionUID = -7130349094387029343, local class serialVersionUID = -3463140113163263022
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1940)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2097)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2222)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
        at org.wikidata.history.game.CorrectionLookup.lambda$loadRules$0(CorrectionLookup.java:41)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.wikidata.history.game.CorrectionLookup.loadRules(CorrectionLookup.java:37)
        at org.wikidata.history.game.CorrectionLookup.<init>(CorrectionLookup.java:32)
        at org.wikidata.history.game.Main.main(Main.java:28)

2020-03-01T06:04:41.011+0000  SEVERE  org.wikidata.history.corhist.mining.SimpleConstraintRule
java.lang.ClassNotFoundException: org.wikidata.history.corhist.mining.SimpleConstraintRule
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:720)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1923)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2097)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
        at org.wikidata.history.game.CorrectionLookup.lambda$loadRules$0(CorrectionLookup.java:41)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.wikidata.history.game.CorrectionLookup.loadRules(CorrectionLookup.java:37)
        at org.wikidata.history.game.CorrectionLookup.<init>(CorrectionLookup.java:32)
        at org.wikidata.history.game.Main.main(Main.java:28)

2020-03-01T06:04:41.608+0000  SEVERE  invalid type code: 00
java.io.StreamCorruptedException: invalid type code: 00
        at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2926)
        at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2960)
        at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:3032)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at java.io.ObjectInputStream$BlockDataInputStream.readInt(ObjectInputStream.java:3237)
        at java.io.ObjectInputStream.readInt(ObjectInputStream.java:1066)
        at java.util.ArrayList.readObject(ArrayList.java:786)
        at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2233)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
        at org.wikidata.history.game.CorrectionLookup.lambda$loadRules$0(CorrectionLookup.java:41)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.wikidata.history.game.CorrectionLookup.loadRules(CorrectionLookup.java:37)
        at org.wikidata.history.game.CorrectionLookup.<init>(CorrectionLookup.java:32)
        at org.wikidata.history.game.Main.main(Main.java:28)

2020-03-01T06:04:41.744+0000  SEVERE  org.wikidata.history.corhist.mining.SimpleConstraintRule
java.lang.ClassNotFoundException: org.wikidata.history.corhist.mining.SimpleConstraintRule
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:720)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1923)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2097)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
        at org.wikidata.history.game.CorrectionLookup.lambda$loadRules$0(CorrectionLookup.java:41)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.wikidata.history.game.CorrectionLookup.loadRules(CorrectionLookup.java:37)
        at org.wikidata.history.game.CorrectionLookup.<init>(CorrectionLookup.java:32)
        at org.wikidata.history.game.Main.main(Main.java:28)

2020-03-01T06:04:51.529+0000  SEVERE  invalid type code: AC
java.io.StreamCorruptedException: invalid type code: AC
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1656)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
        at org.wikidata.history.game.CorrectionLookup.lambda$loadRules$0(CorrectionLookup.java:41)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.wikidata.history.game.CorrectionLookup.loadRules(CorrectionLookup.java:37)
        at org.wikidata.history.game.CorrectionLookup.<init>(CorrectionLookup.java:32)
        at org.wikidata.history.game.Main.main(Main.java:28)

2020-03-01T06:05:51.630+0000  SEVERE  invalid type code: AC
java.io.StreamCorruptedException: invalid type code: AC
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1656)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
        at org.wikidata.history.game.CorrectionLookup.lambda$loadRules$0(CorrectionLookup.java:41)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.wikidata.history.game.CorrectionLookup.loadRules(CorrectionLookup.java:37)
        at org.wikidata.history.game.CorrectionLookup.<init>(CorrectionLookup.java:32)
        at org.wikidata.history.game.Main.main(Main.java:28)

2020-03-01T06:05:56.346+0000  SEVERE  invalid type code: 73
java.io.StreamCorruptedException: invalid type code: 73
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1810)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2097)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
        at org.wikidata.history.game.CorrectionLookup.lambda$loadRules$0(CorrectionLookup.java:41)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.wikidata.history.game.CorrectionLookup.loadRules(CorrectionLookup.java:37)
        at org.wikidata.history.game.CorrectionLookup.<init>(CorrectionLookup.java:32)
        at org.wikidata.history.game.Main.main(Main.java:28)

2020-03-01T06:06:45.818+0000  SEVERE  invalid type code: AC
java.io.StreamCorruptedException: invalid type code: AC
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1656)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
        at org.wikidata.history.game.CorrectionLookup.lambda$loadRules$0(CorrectionLookup.java:41)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.wikidata.history.game.CorrectionLookup.loadRules(CorrectionLookup.java:37)
        at org.wikidata.history.game.CorrectionLookup.<init>(CorrectionLookup.java:32)
        at org.wikidata.history.game.Main.main(Main.java:28)

2020-03-01T06:07:05.927+0000  WARNING  Invalid cookie header: "Set-Cookie: WMF-Last-Access=01-Mar-2020;Path=/;HttpOnly;secure;Expires=Thu, 02 Apr 2020 00:00:00 GMT". Invalid 'expires' attribute: Thu, 02 Apr 2020 00:00:00 GMT
2020-03-01T06:07:05.929+0000  WARNING  Invalid cookie header: "Set-Cookie: WMF-Last-Access-Global=01-Mar-2020;Path=/;Domain=.wikidata.org;HttpOnly;secure;Expires=Thu, 02 Apr 2020 00:00:00 GMT". Invalid 'expires' attribute: Thu, 02 Apr 2020 00:00:00 GMT
2020-03-01T06:09:37.725+0000  WARNING  unexpected end of stream on Connection{www.wikidata.org:443, proxy=DIRECT hostAddress=www.wikidata.org/208.80.154.224:443 cipherSuite=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 protocol=http/1.1}
java.io.IOException: unexpected end of stream on Connection{www.wikidata.org:443, proxy=DIRECT hostAddress=www.wikidata.org/208.80.154.224:443 cipherSuite=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 protocol=http/1.1}
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:205)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:135)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227)
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211)
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
        ... 19 more

Related Objects

Event Timeline

bd808 created this task.Mar 1 2020, 6:11 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 1 2020, 6:11 AM
Tpt added a comment.Mar 1 2020, 11:59 AM

Thank you for investigating. I'm going to fix this problem.

taavi closed this task as Resolved.Jul 19 2021, 7:33 AM
taavi added a subscriber: taavi.

Looks fixed:

taavi@tools-sgebastion-11:~ $ k get pod -n tool-corhist
NAME                       READY   STATUS    RESTARTS   AGE
corhist-84fddf9cbb-hzlx2   1/1     Running   1          19d
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL