@Lydia_Pintscher Done. Please check the dashboard.

I just looked and the Form data seems to be the same as the Property data and the Lexeme data seems to be the same as the Item data. Could you double check?

I am on it.

@Lydia_Pintscher In the Wikidata namespaces documentation: https://www.wikidata.org/wiki/Help:Namespaces there are no mentions of the Lexems and Forms namespaces.

In order to have the current Graphite query for Items/Properties:

aliasSub(
    aliasSub(
        aliasByNode(
            daily.wikidata.site_stats.pages_by_namespace.{0,120}.nonredirects, 
            4),
            '120', 'Properties'),
            '0', 'Items')

where 120 = properties and 0 = items, modified to work with Lexems/Forms, I need to know the respective namespace codes (one for Lexems and one for Forms). Please advise.

https://www.wikidata.org/w/api.php?action=query&meta=siteinfo&siprop=namespaces says Lexeme is 146. Forms are not in a separate namespace but part of the Lexeme page.

@Lydia_Pintscher Modifying the existing Graphite query for Items and Properties (see T191424#4236348) to incorporate the Lexems in the following way:

aliasSub(
   aliasSub(
      aliasSub(
         aliasByNode(daily.wikidata.site_stats.pages_by_namespace.{0,120,146}.nonredirects, 4), 
      '120', 'Properties'), 
   '0', 'Items'), 
'146', 'Lexems')

keeps on retrieving the data on Items and Properties, but returns no data points for Lexems (namespace: 146, as suggested).

The logic should be clear: if 146 is just another namespace, and we know from Graphite documentation that aliasSub does nothing else but regex gsub, then nesting another aliasSub call around the existing two simply must work. But it doesn't.

@Lydia_Pintscher Who wrote the initial Graphite query? His or her usage of aliasByNode is not completely clear to me, and maybe the problem is there.

Also, it is unclear to me at this point how to fetch the Forms data from Graphite, if these data are said to be parts of the content of the respective lexeme page (and they manifestly are). Is there a separate Graphite storage for these data? Because if they are really stored only as a content of the lexeme pages, well it's sounds crazy but web-scraping will be the only way to go:)

The following is inessential for the task at hand, but also I can't find the data that this dashboard uses on https://graphite.wikimedia.org/ - @Addshore You were mentioning another Graphite storage to me once, could you please remind where do to these things go and where the respective series can be browsed? - Thanks.

@Lydia_Pintscher Modifying the existing Graphite query for Items and Properties (see T191424#4236348) to incorporate the Lexems in the following way:

aliasSub(
   aliasSub(
      aliasSub(
         aliasByNode(daily.wikidata.site_stats.pages_by_namespace.{0,120,146}.nonredirects, 4), 
      '120', 'Properties'), 
   '0', 'Items'), 
'146', 'Lexems')

keeps on retrieving the data on Items and Properties, but returns no data points for Lexems (namespace: 146, as suggested).

@Lydia_Pintscher @Addshore The mystery is temporarily resolved: namely, the tracking for lexems and forms is still not implemented, c.f.

• https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/wikidata/site_stats/pages_by_namespace.php#L20
• https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/wikidata/site_stats/sql/select_pages_by_namespace.sql#L6-L8

Someone please ping me when the data for this are in place. Thanks.

@GoranSMilovanovic fancy trying to modify the script?

It should be a 2 line addition to https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/wikidata/site_stats/sql/select_pages_by_namespace.sql#L6-L8

The namespaces to be added can be seen at https://github.com/wikimedia/mediawiki-extensions-WikibaseLexeme/blob/master/extension.json#L28-L33

@Lydia_Pintscher @Addshore https://gerrit.wikimedia.org/r/#/c/analytics/wmde/scripts/+/440901/

Q1. Where do you make this SQL calls from? It would help me to understand the workflow.
Q2. Where from (and when) do you send the data obtained from SQL to Graphite?
Q3. Would you consider switching this and similar event logging things to Big Data (HiveQL from the Data Lake)? Because sooner or later...
Q4. How will we track the forms data (see T191424#4236354).

If this is schematic - in a sense that I always need to introduce the changes into SQL code, or write new SQL code, in analytics/wmde/scripts - I can do these things from now on. It would be enough if you let me know what exactly do you need to be done on every new occasion.

In T191424#4297796, @GoranSMilovanovic wrote:

@Lydia_Pintscher @Addshore https://gerrit.wikimedia.org/r/#/c/analytics/wmde/scripts/+/440901/

Q1. Where do you make this SQL calls from? It would help me to understand the workflow.

The scripts are puppetized and run on stat1005 currently as the user "analytics-wmde"

Q2. Where from (and when) do you send the data obtained from SQL to Graphite?

The SQL @ https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/wikidata/site_stats/sql/select_pages_by_namespace.sql#L6-L8 is used in https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/wikidata/site_stats/pages_by_namespace.php which makes the query and sends the data to graphite.

Q3. Would you consider switching this and similar event logging things to Big Data (HiveQL from the Data Lake)? Because sooner or later...

Yes, this could easily be moved to the data lake and the mediawiki_history table now, although I might actually be more efficient to do it in sql? (I haven't looked into this)
This isn't really an "event logging" thing.

Q4. How will we track the forms data (see T191424#4236354).

This query can not be modified to track forms as forms are not a top level entity / have no namespace.

If this is schematic - in a sense that I always need to introduce the changes into SQL code, or write new SQL code, in analytics/wmde/scripts - I can do these things from now on. It would be enough if you let me know what exactly do you need to be done on every new occasion.

It will be easier if on occasions such as this you ask the questions and we answer them in tickets rather than always explain every step in tickets as we risk spending time explaining things that are already known.

Change 440901 had a related patch set uploaded (by Addshore; owner: GoranSMilovanovic):
[analytics/wmde/scripts@master] Add Lexeme namespace, 146, to pages by namespace script

https://gerrit.wikimedia.org/r/440901

Change 441022 had a related patch set uploaded (by Addshore; owner: GoranSMilovanovic):
[analytics/wmde/scripts@production] Add Lexeme namespace, 146, to pages by namespace script

https://gerrit.wikimedia.org/r/441022

Change 441022 merged by Addshore:
[analytics/wmde/scripts@production] Add Lexeme namespace, 146, to pages by namespace script

https://gerrit.wikimedia.org/r/441022

Change 440901 merged by Addshore:
[analytics/wmde/scripts@master] Add Lexeme namespace, 146, to pages by namespace script

https://gerrit.wikimedia.org/r/440901

@Lydia_Pintscher @Addshore

• Lexeme usage is now present at https://grafana.wikimedia.org/dashboard/db/wikidata-datamodel (note: the data has just started to roll in, so you are looking for a point in the lower right corner of the graph).
• We need to understand how to track forms. Will forms have a namespace?

• Lexems moved to the right Y-axis with properties.

In T191424#4301999, @GoranSMilovanovic wrote:

• We need to understand how to track forms. Will forms have a namespace?

Forms won't have a namespace.
The best way to count them right now would be as part of the dump scanner that runs weekly.

Maybe page properties should include counts of lemmas, forms and senses, as they do for statements and identifiers.

@GoranSMilovanovic Thanks! And yay :)

I think @Esc3300's suggestion is good. We will want to show the number of Forms and later Senses on Special:Search for example and page properties is what we're using for the number of statements and sitelinks there iirc. @Addshore What do you think?

In T191424#4303106, @Lydia_Pintscher wrote:

I think @Esc3300's suggestion is good. We will want to show the number of Forms and later Senses on Special:Search for example and page properties is what we're using for the number of statements and sitelinks there iirc. @Addshore What do you think?

We could put this on page info, although we don't really do it for anything else yet (such as statements).

In T191424#4302448, @Esc3300 wrote:

Maybe page properties should include counts of lemmas, forms and senses, as they do for statements and identifiers.

Wait a minute. Do we mean page props the table, or on page info?
I think I commented this morning before my coffee had kicked in.

Page props. We have T199611 for that.

The page property for the number of Senses and Forms of a Lexeme exists now. That hopefully makes it possible to add them to the existing graph in grafana.

FYI. This query which is pretty fast can give you total number of forms and senses but it won't stay fast in future:

MariaDB [wikidatawiki_p]> select pp_propname, SUM(pp_value) as total_number from page_props where pp_propname in ('wbl-senses', 'wbl-forms') group by pp_propname;
+-------------+--------------+
| pp_propname | total_number |
+-------------+--------------+
| wbl-forms   |        26390 |
| wbl-senses  |         3620 |
+-------------+--------------+
2 rows in set (0.12 sec)

@Ladsgroup We could totally add that to the bunch of daily scripts that we run now again the sql replicas on the analytics servers :)
But yes, one day it is going to get slow, but it's probably fine for now?

In T191424#4918150, @Addshore wrote:

But yes, one day it is going to get slow, but it's probably fine for now?

It's definitely fine for me, it's such a low-hanging fruit.

Change 487128 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[analytics/wmde/scripts@master] Add SQL to push number of forms and senses to graphite

https://gerrit.wikimedia.org/r/487128

Change 487357 had a related patch set uploaded (by Addshore; owner: Ladsgroup):
[analytics/wmde/scripts@production] Add SQL to push number of forms and senses to graphite

https://gerrit.wikimedia.org/r/487357

Change 487128 merged by jenkins-bot:
[analytics/wmde/scripts@master] Add SQL to push number of forms and senses to graphite

https://gerrit.wikimedia.org/r/487128

Change 487357 merged by jenkins-bot:
[analytics/wmde/scripts@production] Add SQL to push number of forms and senses to graphite

https://gerrit.wikimedia.org/r/487357

I don't see Forms and Senses on https://grafana.wikimedia.org/dashboard/db/wikidata-datamodel?refresh=30m&orgId=1 yet. Is there something that still needs to happen?

Yeah, This data will be injected once a day and then we can use it.

Since we had the first push last night, I added it to the graph. Does it look good to you?

Looks good to me, but I'll leave this for @Lydia_Pintscher to close!