Page MenuHomePhabricator

Glent generate_query_similarity_candidates fails with NPE
Open, Needs TriagePublic5 Estimated Story Points

Description

Glent seems to be broken since nov 22:

Exception in thread "main" java.lang.NullPointerException
        at org.wikimedia.search.glent.fst.SerializableFST.writeObject(SerializableFST.java:33)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1154)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
        at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$blockifyObject$4(TorrentBroadcast.scala:319)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
        at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:321)
        at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:138)
        at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:91)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:35)
        at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:77)
        at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1509)
        at org.apache.spark.api.java.JavaSparkContext.broadcast(JavaSparkContext.scala:546)
        at org.wikimedia.search.glent.fst.AllPairsLevenshtein.toLookup(AllPairsLevenshtein.java:147)
        at org.wikimedia.search.glent.fst.AllPairsLevenshtein.getForwardLookup(AllPairsLevenshtein.java:180)
        at org.wikimedia.search.glent.fst.AllPairsLevenshtein.apply(AllPairsLevenshtein.java:70)
        at org.wikimedia.search.glent.fst.AllPairsLevenshtein.apply(AllPairsLevenshtein.java:54)
        at org.wikimedia.search.glent.SimilarQueriesSuggester.generateCandidatesFromStructs(SimilarQueriesSuggester.java:126)
        at org.wikimedia.search.glent.SimilarQueriesSuggester.generateCandidates(SimilarQueriesSuggester.java:96)
        at org.wikimedia.search.glent.Method$M1RunCandidates.accept(Method.java:177)
        at org.wikimedia.search.glent.Method$M1RunCandidates.accept(Method.java:162)
        at org.wikimedia.search.glent.GlentControl.main(GlentControl.java:63)

Details

Related Changes in Gerrit:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
pfischer set the point value for this task to 5.Mon, Jan 12, 4:54 PM

Change #1229681 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[search/glent@master] Repair timestamp parsing in log loading

https://gerrit.wikimedia.org/r/1229681

Change #1229681 merged by jenkins-bot:

[search/glent@master] Repair timestamp parsing in log loading

https://gerrit.wikimedia.org/r/1229681

EBernhardson subscribed.

This is now running again. It has completed the generation stage and is currently creating fresh glent indices in opensearch. At a high level this looks like, but i don't have old enough data to verify 100%, it was caused by a change in meta.dt timestamps to now include millisecond precision, likely as part of T376026.