Fri, Sep 22
Great code :] The d3 part looks tough...
Works as expected and I like the overall look.
Tue, Sep 19
It's good to know that more eyes are looking at these tasks :]
I'm still working on this task since a couple days ago, and it is still broken.
One of the gerrit patches was merged, but the other one still needs more work.
So, I'm reopening the task until we can fix it.
Mon, Sep 18
Fri, Sep 15
- Reorganize wiki selector animation code
Thu, Sep 14
Wed, Sep 13
The code looks good overall, I left 2 nit-picky comments only.
But I tried to generate an error as described in the test plan (by changing the metric URL),
and both the dashboard and the detail page do not show any error message or overlay.
Not sure if I tested it right, see screenshot:
Tue, Sep 12
Maybe a visual aid that would help, is to add color filling to the area below the line (in a paler shade).
We could use that in the detail page as well, as long as there is only 1 line in the chart, no?
Works perfectly :]
There is one thing that IMO we should change, but it's just my view, let's discuss:
In the dashboard miniature, for some reason, the line chart does not have the Y-axis base to 0.
This makes the miniature exaggerate small drops or spikes, and might be confusing, especially in a chart without scales.
See pictures: This is the previous bar-chart miniature:
And this is the new line-chart miniature:
The data is the same, but the small drop in the end of the charts, is perceived quite differently in both cases.
In my opinion we should draw the line-chart with Y-axis base centered on 0.
Thanks a lot @fdans !
Probably not, the huge UPDATE batches are affecting too much disk usage..
Imagine we have purged everything except the giant tables.
Would we want to consider stopping EL and replication for like 1 day,
to be able to purge them (with initial 10-100k batch size and 1 second sleep)?
Thu, Aug 31
Looks good to me!
A small nit: The scale numbers get cut and are difficult to read when selecting 1-Month. See screenshot:
Also, there are some modified instruction lines that do not have the semi-colon at the end.
Only merging is missing.
Only merging is missing.
I checked in stat1006, and now all reports seem to be getting updated.
Will move to done.
Hi @fdans !
I was finally able to rebase the code and push.
I tested this thoroughly and I think it can be deployed.
Please, review and merge if OK.
I applied the changes that joseph suggested and also the ones to make the logging compatible with our cronjob scheme.
Also tested that in Hadoop, and everything works fine. I think this can be given a final review and be merged if OK.
I found some problem with the path to the configuration file. It was outdated since migration from stat1003 to stat1006 machines. That is why the reports weren't generating.
@Ottomata OK will merge them, to unbreak the reports.
Rebase on top of develop branch
Wed, Aug 30
I merged the queries patch.
When the puppet patch is reviewed and merged, you should see your report files here:
It will take a couple hours to compute all queries after merging though.
I see, makes total sense.
Thanks for the explanation!
While following up on EL performance regarding Popups experiment, I looked at the mysql consumer logs and I think EL is receiving around 200 events per second from the Popups schema. Is that expected? We discussed around 100 events per second in this task's description, no? This might make our calculations for space consumption fail. Here are the logs I found: I just summed the inserted events (33000) and divided by the difference of the first and last timestamps (159 seconds).
2017-08-30 18:51:27,518  (MainThread) Inserted 192 Popups_16364296 events in 0.113053 seconds 2017-08-30 18:51:28,908  (MainThread) Inserted 2215 Popups_16364296 events in 1.369190 seconds 2017-08-30 18:51:29,189  (MainThread) Inserted 593 Popups_16364296 events in 0.276136 seconds 2017-08-30 18:51:42,410  (MainThread) Inserted 199 Popups_16364296 events in 0.118885 seconds 2017-08-30 18:51:43,802  (MainThread) Inserted 2295 Popups_16364296 events in 1.375483 seconds 2017-08-30 18:51:44,045  (MainThread) Inserted 506 Popups_16364296 events in 0.238042 seconds 2017-08-30 18:52:02,979  (MainThread) Inserted 187 Popups_16364296 events in 0.116786 seconds 2017-08-30 18:52:04,551  (MainThread) Inserted 2295 Popups_16364296 events in 1.552113 seconds 2017-08-30 18:52:04,802  (MainThread) Inserted 518 Popups_16364296 events in 0.245957 seconds 2017-08-30 18:52:13,393  (MainThread) Inserted 191 Popups_16364296 events in 0.337739 seconds 2017-08-30 18:52:14,891  (MainThread) Inserted 2304 Popups_16364296 events in 1.479943 seconds 2017-08-30 18:52:15,131  (MainThread) Inserted 505 Popups_16364296 events in 0.237636 seconds 2017-08-30 18:52:28,376  (MainThread) Inserted 158 Popups_16364296 events in 0.093370 seconds 2017-08-30 18:52:28,381  (MainThread) Inserted 1 Popups_16364296 events in 0.004940 seconds 2017-08-30 18:52:29,854  (MainThread) Inserted 2349 Popups_16364296 events in 1.453029 seconds 2017-08-30 18:52:30,087  (MainThread) Inserted 492 Popups_16364296 events in 0.229931 seconds 2017-08-30 18:52:44,575  (MainThread) Inserted 169 Popups_16364296 events in 0.107121 seconds 2017-08-30 18:52:45,976  (MainThread) Inserted 2287 Popups_16364296 events in 1.384102 seconds 2017-08-30 18:52:46,242  (MainThread) Inserted 544 Popups_16364296 events in 0.260800 seconds 2017-08-30 18:53:00,650  (MainThread) Inserted 174 Popups_16364296 events in 0.105673 seconds 2017-08-30 18:53:02,240  (MainThread) Inserted 2291 Popups_16364296 events in 1.572348 seconds 2017-08-30 18:53:02,501  (MainThread) Inserted 535 Popups_16364296 events in 0.257486 seconds 2017-08-30 18:53:16,104  (MainThread) Inserted 194 Popups_16364296 events in 0.143966 seconds 2017-08-30 18:53:17,742  (MainThread) Inserted 2295 Popups_16364296 events in 1.624070 seconds 2017-08-30 18:53:18,167  (MainThread) Inserted 511 Popups_16364296 events in 0.419751 seconds 2017-08-30 18:53:33,360  (MainThread) Inserted 231 Popups_16364296 events in 0.139718 seconds 2017-08-30 18:53:34,726  (MainThread) Inserted 2228 Popups_16364296 events in 1.348730 seconds 2017-08-30 18:53:34,991  (MainThread) Inserted 541 Popups_16364296 events in 0.260142 seconds 2017-08-30 18:53:48,461  (MainThread) Inserted 179 Popups_16364296 events in 0.106170 seconds 2017-08-30 18:53:49,867  (MainThread) Inserted 2291 Popups_16364296 events in 1.384415 seconds 2017-08-30 18:53:50,121  (MainThread) Inserted 530 Popups_16364296 events in 0.251223 seconds 2017-08-30 18:54:04,423  (MainThread) Inserted 195 Popups_16364296 events in 0.119049 seconds 2017-08-30 18:54:06,002  (MainThread) Inserted 2268 Popups_16364296 events in 1.559001 seconds 2017-08-30 18:54:06,255  (MainThread) Inserted 537 Popups_16364296 events in 0.250670 seconds
Tue, Aug 29
Mon, Aug 28
Here's the diff of the new change. We'll merge it in short.
Sorry for the inactivity period after having asked you for a quick response. The purging script has been very slow the last weeks (it was still purging 2015), to the point that we stopped it and spent some time working on optimizations and manual archiving of big tables. We'll relaunch it in short.
It's deployed to production.
We should not see the same problem.
Please, reopen the task otherwise!
Fri, Aug 25
Ok, I think I found a fix!
The last patch is the one that counts, the previous ones, are commits to the mforns-dev branch, to be able to deploy to staging and test.
I won't deploy now, because it's late friday. But next Monday this will be deployed.
Aug 24 2017
The only thing that could be improved IMO is to do a consistent instruction ending: either all instructions end with semi-colon, or all without.
Dan thinks semi-colon is better (even necessary), because the language has still some ambiguities without semi-colons.
Hearing that, I would also lean towards semi-colon. Would that be OK for you?
Nevertheless, I think we can leave this semi-colon hell to other patches, because this here involves many files.
But definitely we should stick to one convention in future patches.
Aug 23 2017
We found out what the cause of the issue is:
Wikimetrics is connecting to the same labsdb host for all databases. And it creates a connection for each database it needs to query.
So whenever there is a centralauth-expanded cohort that combined spans through more than 10 wikis, this problem will happen, because the limit of connections per user is 10.
Yes, feel free to add any links or details that you find interesting to the docs!
And yes, you can add a Gerrit change-set with your queries and config to the master branch of reportupdater_queries repository.
If you want, you can add me as a reviewer, and I'll look into it.
Aug 22 2017
Most of the examples I've seen create datasets with a single measurement, meaning there are multiple SQL queries instead of one returning multiple columns. Is this what I should also be setting up in order to facilitate having Dashiki visualize it later?
Thanks @Marostegui !
Aug 21 2017
Hi! I'm looking into this problem and it looks very much like the one Dan was fixing a couple days ago in:
Wikimetrics raises the same error when trying to connect to MW databases:
User 's52261' has exceeded the 'max_user_connections' resource (current value: 10)
Nuria put the tables there, knowing that you would have an opinion,
so she already mentioned that you could change them to wherever you saw fit.
Aug 18 2017
Aug 17 2017
Aug 14 2017
Aug 11 2017
Some comments inline.
In general also, we have to decide if we go for ; at the end of instructions or not.
And also maybe decide to follow a style guide like lint (without necessarily linting the code).
Aug 10 2017
If we want to add local views to those pages in the future, we just add the router-link.
This LGTM! The router change is going to modify this exact line to format it in the new way, though. Anyway, we can merge this in case the router change doesn't make it.
I thought that we were going to discuss this though. Do you like All languages or do you prefer All projects?
The message should be something like:
Use your Wikitech LDAP login
and add link to Wikitech that explains how to get access.