Making the outputs queryable by SQL is a huge win for analysis, and this data can be plugged into Superset dashboards.
Currently, the outputs are only in .ndjson and .csv files under https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/, but these can only be analyzed after import or by using eg. `jq`.
- [] Plan Hive schema. Map data types, decide on naming and namespace,
- [] Choose Hive adapter
- [] Scraper persists outputs in Hive. Start with `pipeline.ex`. Split hive persistence into a separate module.
to Hive when configured production.
- [] Document fields and give some example queries.
== Hive adapter
There seems to be no existing, specialized nor general Hive adapter for Elixir or Erlang. As the simplest workaround, we should be able to use the hive commandline tools on the analytics clients, by crafting special output files and piping into the tools.
In case we need lower-level access, realtime streaming, etc., it's possible to send commands directly to Hive. The Hive backend HiveServer2 speaks [[ https://thrift.apache.org/ | Thrift ]], an RPC abstraction and the [[ https://hexdocs.pm/thrift/ | Elixir Thrift ]] was able to read [[ https://github.com/apache/hive/blob/trunk/service/if/TCLIService.thrift | Hive Thrift definition ]] and generate an Erlang interface code POC in the [[ https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/138 | scraper hive branch ]].
To develop against hive, launch a local Docker container:
```
export HIVE_VERSION=4.2.0
docker pull apache/hive:${HIVE_VERSION}
docker run -p 10000:10000 -p 10002:10002 \
--env SERVICE_NAME=hiveserver2 \
--env SERVICE_OPTS="-Dhive.server2.logging.operation.level=VERBOSE -Dhive.server2.authentication=NOSASL" \
--name hive4 apache/hive:${HIVE_VERSION}
```
The thrift port is localhost:10000
Example Elixir to connect, open a session and run HiveQL:
```
{:ok, client} = :thrift_client_util.new(~c"localhost", 10000, :t_c_l_i_service_thrift, [])
{client, {:ok, {:TOpenSessionResp, status, protocol, session, config}}} = :thrift_client.call(client, :OpenSession, [{:TOpenSessionReq, 10, :undefined, :undefined, :undefined}])
{client, {:ok, {:TExecuteStatementResp, status, operation}}} = :thrift_client.call(client, :ExecuteStatement, [{:TExecuteStatementReq, session, ~c"create table a (b int)", :undefined, false}])
```
There are *cough* some rough edges for the moment.
== Out of scope
There are no known external consumers of the published analytics files, so no obligation to keep producing the files. Old published files could even be deleted and json code run optionally depending on configuration.