Change Details

see also https://github.com/WDscholia/scholia/issues/2412 **Description:** The Wikimedia Foundation is nearing the 4TB limit on its Blazegraph database, necessitating the exploration of federated queries, multiple SPARQL endpoints, and potentially different query languages due to the imminent graph split planned for Q1/2024. The current 1-minute timeout on the official Wikidata Query Service (WDQS) further compounds the issue, making efficient query management crucial. **Problem:** As Blazegraph approaches its storage limit and with a graph split under testing, our ability to handle queries efficiently is becoming strained. This situation may force the adoption of various technical adjustments like federated queries, use of different SPARQL endpoints, and more, potentially complicating the query process. **Proposed Solution:** - Conversion to Named Queries: Shift all relevant queries to a named query format with parameters. This change will make queries easier to manage and modify without altering the core application code. - blackbox style SPARQL compatible middleware Introduction: Implement a SPARQL compatible middleware layer that handles the execution of these queries. The middleware will be responsible for routing queries to the appropriate data stores and translating them as necessary, thereby abstracting the complexities from the end-users. The actual SPARQL query will be hidden from the user This middleware will act as a broker between the client requests and the backend data stores, ensuring queries are executed on the correct store and results are returned efficiently. It will support named queries with parameters, enhancing flexibility and scalability. **Alternatives Considered:** Setting up a private instance of Wikidata as described in the CEUR-WS Vol-3262 paper, though this is resource-intensive and may not be feasible for all users. **Additional Context:** Recent discussions in Search Platform Office Hours and proposals for handling named queries suggest a growing need for more sophisticated query management solutions. Examples include short URLs supported by the Wikidata Query Service and internal handling by QLever, which could be extended further by our middleware. **References:** - [https://github.com/w3c/sparql-dev/issues/57 Query Parameterization and label resolving issues on W3C] - Previous analysis of Blazegraph alternatives. - Potential for testing on platforms like https://scholia.portal.mardi4nfdi.de/. - Linked queries and their handling in other projects like Scholia and CEUR-WS. - https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split - https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/October_2023_scaling_update - https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Federation - https://phabricator.wikimedia.org/T356773 - https://www.wikidata.org/wiki/User:Sannita_(WMF)

see also https://github.com/WDscholia/scholia/issues/2412 **Description:** The Wikimedia Foundation is nearing the 4TB limit on its Blazegraph database, necessitating the exploration of federated queries, multiple SPARQL endpoints, and potentially different query languages due to the imminent graph split planned for Q1/2024. The current 1-minute timeout on the official Wikidata Query Service (WDQS) further compounds the issue, making efficient query management crucial. **Problem:** As Blazegraph approaches its storage limit and with a graph split under testing, our ability to handle queries efficiently is becoming strained. This situation may force the adoption of various technical adjustments like federated queries, use of different SPARQL endpoints, and more, potentially complicating the query process. **Proposed Solution:** - Conversion to Named Queries: Shift all relevant queries to a named query format with parameters. This change will make queries easier to manage and modify without altering the core application code. - blackbox style SPARQL compatible middleware Introduction: Implement a SPARQL compatible middleware layer that handles the execution of these queries. The middleware will be responsible for routing queries to the appropriate data stores and translating them as necessary, thereby abstracting the complexities from the end-users. The actual SPARQL query will be hidden from the user This middleware will act as a broker between the client requests and the backend data stores, ensuring queries are executed on the correct store and results are returned efficiently. It will support named queries with parameters, enhancing flexibility and scalability. **Alternatives Considered:** Setting up a private instance of Wikidata as described in the CEUR-WS Vol-3262 paper, though this is resource-intensive and may not be feasible for all users. **Additional Context:** Recent discussions in Search Platform Office Hours and proposals for handling named queries suggest a growing need for more sophisticated query management solutions. Examples include short URLs supported by the Wikidata Query Service and internal handling by QLever, which could be extended further by our middleware. **References:** - [[https://github.com/w3c/sparql-dev/issues/57 | Query Parameterization and label resolving issues on W3C ]] - Previous analysis of Blazegraph alternatives. - Potential for testing on platforms like https://scholia.portal.mardi4nfdi.de/. - Linked queries and their handling in other projects like Scholia and CEUR-WS. - https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split - https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/October_2023_scaling_update - https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Federation - https://phabricator.wikimedia.org/T356773 - https://www.wikidata.org/wiki/User:Sannita_(WMF)

see also https://github.com/WDscholia/scholia/issues/2412 **Description:** The Wikimedia Foundation is nearing the 4TB limit on its Blazegraph database, necessitating the exploration of federated queries, multiple SPARQL endpoints, and potentially different query languages due to the imminent graph split planned for Q1/2024. The current 1-minute timeout on the official Wikidata Query Service (WDQS) further compounds the issue, making efficient query management crucial. **Problem:** As Blazegraph approaches its storage limit and with a graph split under testing, our ability to handle queries efficiently is becoming strained. This situation may force the adoption of various technical adjustments like federated queries, use of different SPARQL endpoints, and more, potentially complicating the query process. **Proposed Solution:** - Conversion to Named Queries: Shift all relevant queries to a named query format with parameters. This change will make queries easier to manage and modify without altering the core application code. - blackbox style SPARQL compatible middleware Introduction: Implement a SPARQL compatible middleware layer that handles the execution of these queries. The middleware will be responsible for routing queries to the appropriate data stores and translating them as necessary, thereby abstracting the complexities from the end-users. The actual SPARQL query will be hidden from the user This middleware will act as a broker between the client requests and the backend data stores, ensuring queries are executed on the correct store and results are returned efficiently. It will support named queries with parameters, enhancing flexibility and scalability. **Alternatives Considered:** Setting up a private instance of Wikidata as described in the CEUR-WS Vol-3262 paper, though this is resource-intensive and may not be feasible for all users. **Additional Context:** Recent discussions in Search Platform Office Hours and proposals for handling named queries suggest a growing need for more sophisticated query management solutions. Examples include short URLs supported by the Wikidata Query Service and internal handling by QLever, which could be extended further by our middleware. **References:** - - [[https://github.com/w3c/sparql-dev/issues/57 | Query Parameterization and label resolving issues on W3C ]] - Previous analysis of Blazegraph alternatives. - Potential for testing on platforms like https://scholia.portal.mardi4nfdi.de/. - Linked queries and their handling in other projects like Scholia and CEUR-WS. - https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split - https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/October_2023_scaling_update - https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Federation - https://phabricator.wikimedia.org/T356773 - https://www.wikidata.org/wiki/User:Sannita_(WMF)