For the first pass, the following configuration is being used for an hour long test conducted from stat1006 with config file wdqs-split-test.yml as follows.

datasets:
  - name: "split"
connections:
  - name: "baseline"
    endpoint: "https://wdqs1022.eqiad.wmnet/sparql"
  - name: "wikidata_main_graph"
    endpoint: "https://wdqs1024.eqiad.wmnet/sparql"

tasks:
  - className: "org.aksw.iguana.cc.tasks.impl.Stresstest"
    configuration:
      timeLimit: 3600000
      warmup:
        timeLimit: 30000
        workers:
          - threads: 4
            className: "SPARQLWorker"
            queriesFile: "queries_for_performance.txt"
            timeOut: 5000
      queryHandler:
        className: "DelimInstancesQueryHandler"
        configuration:
          delim: "### BENCH DELIMITER ###"
      workers:
        - threads: 4
          className: "SPARQLWorker"
          queriesFile: "queries_for_performance.txt"
          timeOut: 60000
          parameterName: "query"
          gaussianLatency: 100

metrics:
  - className: "QMPH"
  - className: "QPS"
  - className: "NoQPH"
  - className: "AvgQPS"
  - className: "NoQ"

storages:
  - className: "NTFileStorage"
    configuration:
      fileName: result.nt

queries_for_performance.txt is based on the following basic code, which says to get queries known to work against both the full graph and the main (non-scholarly) graph and returning similar results, so as to reduce garbage input and somewhat better control the parameters of the test.

scala> val wikidata = spark.read.parquet("hdfs:///user/dcausse/T352538_wdqs_graph_split_eval/wikidata_classified.parquet")
scala> val full = spark.read.parquet("hdfs:///user/dcausse/T352538_wdqs_graph_split_eval/full_classified.parquet")
scala> val joined5 = wikidata.as("w").join(full.as("f")).where("w.id = f.id and w.success = true and  w.success = f.success and w.resultSize = f.resultSize and w.reorderedHash = f.reorderedHash").select(concat(col("w.query"), lit("\n### BENCH DELIMITER ###"))).distinct
scala> joined5.repartition(1).write.option("compression", "none").text("queries_for_performance_2024_01_25.txt")
scala> :quit

$ hdfs dfs -copyToLocal hdfs://analytics-hadoop/user/dr0ptp4kt/queries_for_performance_2024_01_25.txt/part-00000-6b8caed3-3a4d-4cb2-bf74-6bbcd7af0478-c000.txt ./queries_for_performance.txt
$ /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java -jar iguana-3.3.3.jar wdqs-split-test.yml

The IGUANA build is based on https://gitlab.wikimedia.org/repos/search-platform/IGUANA/-/merge_requests/4 .

Dropping in a screenshot from Grafana from this first pass and made a copy of result.nt to result.nt.001. Re-running to see that server behavior is similar.

Now, a screenshot showing the re-run. And then a screenshot showing them side-by-side. This is just for the visual, and the data produced from IGUANA (what is in the .nt output that we can convert to a handy CSV) should be more telling.

Next up, I'll randomize the order of the queries and do it again.

Now, the screenshot from the randomized order queries. I'll run one more time to see that comparable output is achieved. Those were produced with the following. This latest output file has been moved to result.nt.003.

scala> val joined6 = wikidata.as("w").join(full.as("f")).where("w.id = f.id and w.success = true and  w.success = f.success and w.resultSize = f.resultSize and w.reorderedHash = f.reorderedHash").select(concat(col("w.query"), lit("\n### BENCH DELIMITER ###"))).distinct.sample(withReplacement=false, fraction=1.0, seed=42)
scala> joined6.count // matches same as joined5.count
scala> joined6.repartition(1).write.option("compression", "none").text("queries_for_performance_randomized_2024_01_26.txt")
scala> :quit
$ hdfs dfs -copyToLocal hdfs://analytics-hadoop/user/dr0ptp4kt/queries_for_performance_randomized_2024_01_26.txt/part-00000-131df78f-da7a-4ffc-aad4-9874342165ca-c000.txt ./queries_for_performance_randomized.txt 
$ sha1sum queries_for_performance.txt queries_for_performance_randomized.txt
$ # they're different
$ diff queries_for_performance.txt queries_for_performance_randomized.txt | wc -l
$ # they're very different
$ cp wdqs-split-test.yml wdqs-split-test-randomized.yml
$ # changed pointers to query file to be queries_for_performance_randomized.txt
$ bash start-iguana.sh wdqs-split-test-randomized.yml
$ mv result.nt result.nt.003

{F41720432}

Now a screenshot from the re-run of the randomized order queries, followed by a screenshot showing the two runs on the randomized order queries side by side.

Here were the data produced by IGUANA once piped through the CSV utility introduced in https://gitlab.wikimedia.org/repos/search-platform/IGUANA/-/merge_requests/3/diffs with a command of the following form (for the attentive reader, note that I had to rename the originally produced files to have an .nt extension to make the underlying Jena libraries not throw an exception).

/usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java -cp iguana-3.3.3.jar org.aksw.iguana.rp.analysis.TabularTransform -e result.003.nt > result.003.execution.csv

Keep in mind that a delay between was introduced in the configuration for these "stress tests" (a "stress test" here means that the execution of the queries goes continuously for the specified time interval at its concurrency and delay spec). This was to more closely model what a somewhat busy, but not completely saturated, WDQS node might experience, although we should be mindful that the server specs are a bit different between these test servers and the WDQS hosts used for serving end user WDQS production requests. When interpreting a value like avgqps, remember that this is akin to what might happen if queries were executed serially without delay if it were possible to hold JVM performance constant for such request patterns (do note that this is generally not possible to guarantee, so caveats abound; in other words it's entirely possible that avgqps could degrade in reality).

The successfullQueriesQPH metric is probably the most interesting one. It's suggestive of about a 5%-6% speed advantage for the smaller "main" graph versus a fully populated "full" graph for this query mix when conditions model a somewhat busy WDQS node (again, remember that server spec is a bit different between the SUT and production nodes so there is a caveat). During the very first run the performance difference was more pronounced - closer to 10% - but subsequent runs were closer to the 5%-6% level for wikidata_main_graph versus baseline.

Additional basic summary statistics upon the data from with per-query CSV exports (using the -q flag) against the .nt files to come. Note that in Andrea's previous analysis these sorts of statistics (as well as some tweaks to get somewhat finer precision via BigDecimal instead of Double types) were incorporated directly into the Java source of IGUANA - see https://github.com/dice-group/IGUANA/compare/main...AndreaWesterinen:IGUANA:main for changes up to June 13, 2022 against current main branch of IGUANA; n.b., to future readers you may need to re-correlate the code changes when IGUANA upstream changes. But, I opted to make fewer changes to our fork (i.e., I didn't merge Andrea's fork into our fork, even if there is some dependency similarity in the POMs) as this data can be determined in Spark summary stat calls.

We may be interested in how to take forward some of the enhancement opportunities for IGUANA upstream (Andrea's fork and David's CSV utilities in our fork, for example) should we see the need for more IGUANA work later, but then again we may not do that as our needs are narrower.

Following below are "per-query" summary stats. I actually just put this together by bringing CSV data into Google Sheets for now - all of the columns are calculated upon the "per-query" rows (but you'll see how the Mean corresponds basically with the value calculated up above). The underlying CSV data don't bear actual queries (the .nt files from which they're generated do), but rather rows of this form:

endpointLabel,task,queryId,totalTime,success,failed,timeouts,resultSize,unknownException,wrongCodes,qps,penalizedQPS
baseline,http://iguana-benchmark.eu/resource/1706221131/1/1,http://iguana-benchmark.eu/resource/1989023647/sparql0,53.592,2,0,0,1,0,0,37.319002836244216,37.319002836244216

No big surprises here. The "per-query" behavior was similar between nodes. The "main" graph skewed somewhat faster over the full range of queries with one exception: the absolute fastest singular query for "randomized 1" run was slightly faster on the "baseline" full graph...but generally, everything else skewed faster for the "main" graph otherwise.

Per-query theoretical throughput (queries per second for given query)

A run is in progress for 78K+ queries from a set of 100,000 random queries. It should be done in under 10 hours from now.

scala> val full_random = spark.read.parquet("hdfs:///user/dcausse/T352538_wdqs_graph_split_eval/full_random_classified.parquet")

scala> val wikidata_random = spark.read.parquet("hdfs:///user/dcausse/T352538_wdqs_graph_split_eval/wikidata_random_classified.parquet")

scala> full_random.count
res0: Long = 100000 

scala> wikidata_random.count
res6: Long = 100000  

scala> val joined11 = wikidata_random.as("w").join(full_random.as("f")).where("w.id = f.id and w.success = true and  w.success = f.success and w.resultSize = f.resultSize and w.reorderedHash = f.reorderedHash").select(concat(col("w.query"), lit("\n### BENCH DELIMITER ###"))).distinct.sample(withReplacement=false, fraction=1.0, seed=42)

scala> joined11.count
res0: Long = 78862

scala> joined11.repartition(1).write.option("compression", "none").text("queries_for_performance_2024_01_31.txt")

scala> :quit

$ hdfs dfs -copyToLocal hdfs://analytics-hadoop/user/dr0ptp4kt/queries_for_performance_2024_01_31.txt/part-00000-29c4e72d-800d-4148-b804-8e428ee71e9e-c000.txt ./queries_for_performance_file_renamed_randomized_2024_01_31.txt

$ bash start-iguana.sh wdqs-split-test-randomized-2024-01-31.yml

start-iguna.sh previously ran from stat1006, but this time around it's running from stat1008 in order to use more RAM for the larger query mix.

Here's the output from the latest run based upon a larger set of queries from a random sample of WDQS queries.

$ /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java -cp iguana-3.3.3.jar org.aksw.iguana.rp.analysis.TabularTransform -e result.nt > result.execution.csv
$ cut -f1,3,5,6,7,9 -d"," result.execution.csv | sed 's/,/|/g'

Here's the screen capture from Grafana.

The non-scholarly "main" wikidata_main_graph (wdqs1022) window completed more queries. During the end of its window, a larger garbage collection pass occurred on the baseline ("full") graph node (wdqs1024) - where the window had closed much earlier. This happened to coincide with a small window of temporary failing queries observed, which is aggregated across all three nodes, "main", "full", and "scholarly". Depending on how one looks at this this corresponds to maybe 1.5%-3.0% of the wikidata_main_graph's run time, so I wouldn't read too much into this.

To repeat the verbiage from the earlier runs...

Following below are "per-query" summary stats. I actually just put this together by bringing CSV data into Google Sheets for now - all of the columns are calculated upon the "per-query" rows (but you'll see how the Mean corresponds basically with the value calculated up above). The underlying CSV data don't bear actual queries (the .nt files from which they're generated do), ...

The CSV data were generated with the following command:
/usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java -cp iguana-3.3.3.jar org.aksw.iguana.rp.analysis.TabularTransform -q result.nt > result.query.csv

Although the max and 99th percentile queries were just ever so slightly faster on the baseline "full" graph, more generally things were faster on the non-scholarly "main" graph. The performance difference is obvious but not dramatic.

Here's the content of wdqs-split-test-randomized-2024-01-31.yml, comments removed for brevity. The main difference in this configuration file from the earlier presented one is five hours allowed per graph, to accommodate a larger query mix, and the updated filename pointing to the larger query mix based on the set of queries from the random sample.

datasets:
  - name: "split"
connections:
  - name: "baseline"
    endpoint: "https://wdqs1022.eqiad.wmnet/sparql"
  - name: "wikidata_main_graph"
    endpoint: "https://wdqs1024.eqiad.wmnet/sparql"

tasks:
  - className: "org.aksw.iguana.cc.tasks.impl.Stresstest"
    configuration:
      timeLimit: 18000000
      warmup:
        timeLimit: 30000
        workers:
          - threads: 4
            className: "SPARQLWorker"
            queriesFile: "queries_for_performance_file_renamed_randomized_2024_01_31.txt"
            timeOut: 5000
      queryHandler:
        className: "DelimInstancesQueryHandler"
        configuration:
          delim: "### BENCH DELIMITER ###"
      workers:
        - threads: 4
          className: "SPARQLWorker"
          queriesFile: "queries_for_performance_file_renamed_randomized_2024_01_31.txt"
          timeOut: 60000
          parameterName: "query"
          gaussianLatency: 100

metrics:
  - className: "QMPH"
  - className: "QPS"
  - className: "NoQPH"
  - className: "AvgQPS"
  - className: "NoQ"

storages:
  - className: "NTFileStorage"
    configuration:
      fileName: result.nt

@dr0ptp4kt thanks! is the difference in the number of successful queries only explained by the improvement in query time or are there some improvements in the number of queries that timeout as well?

In T355037#9508760, @dcausse wrote:

@dr0ptp4kt thanks! is the difference in the number of successful queries only explained by the improvement in query time or are there some improvements in the number of queries that timeout as well?

Good question! It appears to be related to query time.

Looking at this latest run, for example, there were no recorded timeouts according to the CSV of the IGUANA .nt (🤔).

Taking things on a head-to-head basis for identical queries between the endpoints, here's what we see for the difference in speed for wikidata_main_graph minus baseline. It's unsurprising in a way given the distribution shown in the prior Phabricator comment, but it is another way of knowing that, under this parameters of this test anyway (a controlled flow and not maximum load test), about 70% of the queries noted as successful seemed to be faster when run against the wikidata_main_graph. Note that about 16% of the queries hit wrongCodes / failed, which are discussed after the table.

About 58% of queries titled toward wikidata_main_graph, and about 25% tilted toward baseline, and 58/(58+25) is about 0.7. The stuff where the difference is negligible probably don't matter that much. Yet, there's a bit more detail to consider in IGUANA's conception here...

For the sake of completeness, and because this may be interesting to consider later on or to contextualize the QPS distributions in the prior Phabricator comment: looking at a different class of issues, let's suppose that we use wrongCodesas a proxy for things that could have gone wrong.wrongCodes and failed map to each other in the CSV, and their QPSes land as 0 for these (penalizedQPS, not included in the tables above, lands by default as 0.01666666667 for these records, but this is close enough to 0 if we wanted to look at it that way). These sorts of records thus drive down summary mean, median, and so on. As an aside, in terms of actual time (totalTime), these wrongCodes ones occupy very little time.

The wrongCodes resulted for basically the same queries across both endpoints (and each row of the CSVs is a per-query row, as we know).

The ratio of wikidata_main_graph to baseline for sum wrongCodes (1.036629326091525) is very close to the ratio up above for successfullQueries (1.036316228814752).

I summarized at https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Graph_split_IGUANA_performance . When we have a mailing list post during the next week or so, we'll want to move this to be a subpage of the target page of the post.