@mforns Most of our metrics looks healthy, but there's one SQL job in particular for which data is inexplicably missing. The codemirror/sql/users_codemirror_and_wikitext.sql job is expected to output metrics into MediaWiki.CodeMirror.preferences.byPreference.*.byEnabled.true.byEditCount.$edit_count_bucket.byWiki.$wiki, but I only an empty hierarchy ending at byWiki for each preference. Running the script manually seems to work, although I could be wrong about the format of from_timestamp in the date coercion.

Please let us know if there are any job failures, and whether the intermediate output files for this job contain any data.

Please let us know if there are any job failures, and whether the intermediate output files for this job contain any data.

I have access to the logs now, and I see an error which I cannot reproduce locally:

Jun 22 09:59:13 an-launcher1002 reportupdater-codemirror[981]: 2021-06-22 09:59:13,297 - INFO - Executing "<Report key=users_codemirror_and_wikitext type=sql granularity=days lag=0 first_date=2021-04-16 start=2021-05-25 end=2021-05-26 db_key=mediawiki hql_template=None sql_template=WITH editor_preference AS ( SELECT up_user AS user_id, -- Get raw preferences, and interpret dependi... script=None explode_by={'wiki_db': 'enwiki'} max_data_points=None graphite={'path': '{_metric}.byEditCount.{edit_count_bucket}.byWiki.{wiki_db}', 'metrics': {'MediaWiki.CodeMirror.preferences.byPreference.wikitext_2010.byEnabled.true': 'users_wikitext_2010', 'MediaWiki.CodeMirror.preferences.byPreference.wikitext_2017.byEnabled.true': 'users_wikitext_2017', 'MediaWiki.CodeMirror.preferences.byPreference.CodeMirror.byEnabled.true': 'use_syntax_highlighting', 'MediaWiki.CodeMirror.preferences.byPreference.wikitext_2010_and_CodeMirror.byEnabled.true': 'users_wikitext_2010_and_codemirror', 'MediaWiki.CodeMirror.preferences.byPreference.wikitext_2017_and_CodeMirror.byEnabled.true': 'users_wikitext_2017_and_codemirror'}} results={'header': '[]', 'data': '0 rows'} group=None>"...
Jun 22 10:10:34 an-launcher1002 reportupdater-codemirror[981]: 2021-06-22 10:10:34,397 - ERROR - Report "users_codemirror_and_wikitext" could not be written because of error: 'NoneType' object has no attribute 'strftime'
Jun 22 10:10:34 an-launcher1002 reportupdater-codemirror[981]: Traceback (most recent call last):
Jun 22 10:10:34 an-launcher1002 reportupdater-codemirror[981]:   File "/srv/reportupdater/reportupdater/reportupdater/writer.py", line 45, in run
Jun 22 10:10:34 an-launcher1002 reportupdater-codemirror[981]:     self.write_results(header, updated_data, report, self.get_output_folder())
Jun 22 10:10:34 an-launcher1002 reportupdater-codemirror[981]:   File "/srv/reportupdater/reportupdater/reportupdater/writer.py", line 148, in write_results
Jun 22 10:10:34 an-launcher1002 reportupdater-codemirror[981]:     row[0] = row[0].strftime(DATE_FORMAT)
Jun 22 10:10:34 an-launcher1002 reportupdater-codemirror[981]: AttributeError: 'NoneType' object has no attribute 'strftime'

My guess is that the intermediate output files have some corrupted rows. @mforns @elukey Please remove the output cache file and fast-forward the start time to the current day (the SQL query cannot backfill historical data).

Will do today @awight, sorry for making you wait.

In T273748#7169617, @mforns wrote:

Will do today @awight, sorry for making you wait.

No wait at all! I think what happened is that your team did other work on other tasks, which got me access to the logs. Once I found the bug in the logs, I manually smoke-tested, couldn't reproduce the issue, and then failed to come back here and clearly document the things I learned.

Change 700990 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/reportupdater-queries@master] Advance start date of codemirror job

https://gerrit.wikimedia.org/r/700990

Change 700990 merged by Mforns:

[analytics/reportupdater-queries@master] Advance start date of codemirror job

https://gerrit.wikimedia.org/r/700990

@mforns Thanks for adjusting the start date. Can I ask if you were able to remove the intermediate output files? I see the same error in logs, fwiw.

@awight, I haven't yet deleted the output files, because the code change, even if merged, has not made it to production.
I thought that merging was enough for puppet agent to deploy the code, but maybe that changed?
I will continue looking into this today.

In T273748#7172339, @mforns wrote:

@awight, I haven't yet deleted the output files, because the code change, even if merged, has not made it to production.
I thought that merging was enough for puppet agent to deploy the code, but maybe that changed?
I will continue looking into this today.

Ah got it, thank you. I can wait patiently, just let me know when I should check the logs and hadoop again.

@awight, OK @elukey helped me find the problem.
The code was not reaching the production folder because of a puppet change I did last month, my bad.
He fixed it and I deleted the corrupt intermediate output report files (left cawiki.tsv because it was fine).
Next time reportupdater runs in about 30 mins, you should see new logs, and hopefully it will work.
Let me know!
Cheers

@mforns Success, I see the data landing in Graphite! Thank you for this marathon of various changes :-D

Just as a reminder to self and our team, these are the metrics that are now unlocked, and can be used to normalize the dashboard:

users_codemirror_and_wikitext:
  granularity: days
  starts: 2021-06-01
  type: sql
  execute: sql/users_codemirror_and_wikitext
  explode_by:
    wiki_db: sql/wiki_dbs.txt
  graphite:
    path: "{_metric}.byEditCount.{edit_count_bucket}.byWiki.{wiki_db}"
    metrics:
      MediaWiki.CodeMirror.preferences.byPreference.wikitext_2010.byEnabled.true: users_wikitext_2010
      MediaWiki.CodeMirror.preferences.byPreference.wikitext_2017.byEnabled.true: users_wikitext_2017
      MediaWiki.CodeMirror.preferences.byPreference.CodeMirror.byEnabled.true: use_syntax_highlighting
      MediaWiki.CodeMirror.preferences.byPreference.wikitext_2010_and_CodeMirror.byEnabled.true: users_wikitext_2010_and_codemirror
      MediaWiki.CodeMirror.preferences.byPreference.wikitext_2017_and_CodeMirror.byEnabled.true: users_wikitext_2017_and_codemirror

@awight Awesome! I'm glad it works in the end.
It's the first time I see the corrupted file problem, will keep an eye to see if it happens again.

In T273748#7175255, @mforns wrote:

@awight Awesome! I'm glad it works in the end.
It's the first time I see the corrupted file problem, will keep an eye to see if it happens again.

I think it was a result of my playing around with the date column, and at some point I had merged a query which left out the column...

I added new diagrams about code mirror enabled in preferences to the code mirror grafana board. I added normalized charts and also some with the total numbers. Please review the preferences section of the board (at the bottom) and let me know if there are questions or wishes for adaptions.