Page MenuHomePhabricator

wdqs.data-reload cookbook fails when deleting the old namespace
Closed, ResolvedPublic5 Estimated Story Points

Description

As wdqs maintainer I want to reload all categories on wdqs machines without errors.

Processing zuwiktionary...
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><meta http-equiv="Content-Type" content="text&#47;html;charset=UTF
-8"><title>blazegraph&trade; by SYSTAP</title
></head
><body<p>totalElapsed=1ms, elapsed=1ms, connFlush=0ms, batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p

><hr><p>COMMIT: totalElapsed=29ms, commitTime=1606887972763, mutationCount=197</p
></html
>DELETE NAMESPACE: namespace=categories20200416
java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=categories20200416

        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:292)
        at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doDeleteNamespace(MultiTenancyServlet.java:622)

[...]
Caused by: java.lang.RuntimeException: com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=categories20200416

        at com.bigdata.rdf.sail.BigdataSail.getUnisolatedConnectionLocksAndRunLambda(BigdataSail.java:1459)
[...]
        ... 1 more
Caused by: com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=categories20200416
[...]
        ... 12 more
2020-12-02T05:46:12+00:00 categories reload done
================
PASS |███████████████████████████████████████| 100% (1/1) [1:33:15<00:00, 5595.63s/hosts]
FAIL |                                                 |   0% (0/1) [1:33:15<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/usr/local/bin/r...tegories.sh wdqs'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Categories loaded in 1:33:46.955369
Enabling Puppet with reason "T259588: reload categories on all wdqs - ryankemper@cumin1001 - T259588" on 1 hosts: wdqs1006.eqiad.wmnet
----- OUTPUT of 'enable-puppet "T...n1001 - T259588"' -----
================
PASS |███████████████████████████████████████████| 100% (1/1) [00:02<00:00,  2.86s/hosts]
FAIL |                                                   |   0% (0/1) [00:02<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'enable-puppet "T...n1001 - T259588"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'awk '/^\s*comman...cinga/icinga.cfg' -----
/var/lib/icinga/rw/icinga.cmd
================
PASS |███████████████████████████████████████████| 100% (1/1) [00:00<00:00,  2.08hosts/s]
FAIL |                                                   |   0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'awk '/^\s*comman...cinga/icinga.cfg'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'echo -n "[160688...ga/rw/icinga.cmd' -----
================
PASS |███████████████████████████████████████████| 100% (1/1) [00:00<00:00,  1.73hosts/s]
FAIL |                                                   |   0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'echo -n "[160688...ga/rw/icinga.cmd'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (PASS) - Cookbook sre.wdqs.data-reload (exit_code=0)
done!

AC:

  • reloading categories works without errors

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Here's something interesting: On a server where I've recently run the data-reload for categories (wdqs1006 in this case), the categories journal looks like so:

-rw-rw-r-- 1 blazegraph blazegraph 21G Dec 3 07:19 categories.jnl

Yet on a wdqs instance where I have not yet run the reload (wdqs2007 for example), it looks like this:

-rw-rw-r-- 1 blazegraph blazegraph 23G Dec 3 07:32 categories.jnl

So the freshly reloaded instances have only a 21G categories.jnl, whereas the not-freshly-reloaded instances have a 23G categories.jnl - a full 2GB bigger. I'm not sure if that indicates a problem somewhere, but naively I would not expect the journal to be smaller after reloading.

dcausse renamed this task from wdqs.data-reload cookbook fails when switching to wdqs.data-reload cookbook fails when deleting the old namespace.Dec 4 2020, 4:04 PM
CBogen set the point value for this task to 5.Dec 7 2020, 6:27 PM

Change 666610 had a related patch set uploaded (by ZPapierski; owner: ZPapierski):
[operations/puppet@production] Don't drop namespace if it's already gone

https://gerrit.wikimedia.org/r/666610

Change 666610 merged by Ryan Kemper:
[operations/puppet@production] Don't drop namespace if it's already gone

https://gerrit.wikimedia.org/r/666610

Change 667054 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/puppet@production] wdqs: improve replaceNamespace log output

https://gerrit.wikimedia.org/r/667054

Change 667054 merged by Ryan Kemper:

[operations/puppet@production] wdqs: improve replaceNamespace log output

https://gerrit.wikimedia.org/r/667054