Is 48k transaction records all transactions committed up to that point?

how is the data being imported? is this on labs?

@GWicke, it looks like it, the loader script reports processing 51K rows, which means 51K vertices inserted/updated, so 48K is pretty close to that number (not sure why they are not equal though). The script now commits every 1000 rows, but I'm not sure how Titan counts transactions.

@aude its on my local machine but I can put it on labs if somebody tells me how :)

I imagine its that transactions are being held onto in TitanBlueprintsGraph for some reason. Doing a bit of digging.

@Manybubbles I can give you the heap dumps (or attach it here if huge files are ok here) if it helps. Also, the code I'm using is here: https://github.com/smalyshev/wikidata-gremlin/tree/titan

@Smalyshev ok, makes sense and helps to see the code :)

looks fine for just experimenting, but have you looked at the Wikidata toolkit yet?

https://github.com/Wikidata/Wikidata-Toolkit/

if we are to go with titan or whatever java thing, then i would be tempted to see what, if anything of the toolkit would be useful here.

@Smalyshev OK - I can you try adding g.rollback() to the end of any of your scripts? I _think_ Titan isn't being super careful to automatically close any outstanding transactions. It autocreates them it lots of cases.

@Smalyshev - disregard that last bit - now that I see the code it makes more sense.

@aude - Yeah - wikidata toolkit might be super useful for parsing, not sure.

Oh, another missing piece - the data is loaded now by somethine like: dataLoader.gzipFile("/Users/smalyshev/Downloads/xaax.gz").load(1000000)

So DataLoader is the starting point. Sorry, it's work in progress, a bit messy now :)
@aude will definitely check it out, thanks.

@Smalyshev - try adding mgmt.rollback right before https://github.com/smalyshev/wikidata-gremlin/blob/titan/src/main/groovy/org/wikidata/gremlin/Loader.groovy#L340 . It looks like every call to getManagementSystem starts a transaction whether or not you need one........................ RAGE!

Yes, the docs say:
https://thinkaurelius.github.io/titan/javadoc/current/com/thinkaurelius/titan/core/TitanGraph.html#getManagementSystem()

TitanManagement getManagementSystem()
Returns the management system for this graph instance. The management system provides functionality to change global configuration options, install indexes and inspect the graph schema.
The management system operates in its own transactional context which must be explicitly closed.

Which appears to mean each time getManagementSystem is called, it needs to be explicitly closed - even if no changes whatsoever was done. Adding this to the code seems to fix the GC/OOM problems.