Data is not being retrieved/sync'd
Closed, ResolvedPublic1 Story Points

Description

The last data point in aggregated datasets is from Sep 29. I suspect this is caused by the restart of stat1002 (they needed to do it to perform the Linux kernel upgrade).

mpopov created this task.Oct 2 2015, 6:22 PM
mpopov updated the task description. (Show Details)
mpopov raised the priority of this task from to Unbreak Now!.
mpopov assigned this task to Ironholds.
mpopov added a subscriber: mpopov.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2015, 6:22 PM

When did they restart it? The crontab entry is still there.

I'm also seeing an error in desktop.Rout. Check /home/ironholds/golden/ and look at the .Routs, see what you see.

mpopov added a comment.Oct 2 2015, 7:00 PM

desktop.Rout

Error in `[<-.data.table`(x, j = name, value = value) : 
  Cannot use := to add columns to a null data.table (no columns), currently. You can use := to add (empty) columns to a 0-row data.table (1 or more empty columns), though.
Calls: main -> $<- -> $<-.data.table -> [<-.data.table
Execution halted

failures.Rout

Fatal error: cannot open file './search/failures.R': No such file or directory

The rest look fine.

Yeah, failures.R doesn't exist any more ;p. I'll delete that file.

The rest just need backfilling then. I'm not sure why desktop would break - unless something about the restart made it draw non-existent (0,0) data.

@mpopov is this still happening?

mpopov added a comment.EditedOct 7 2015, 4:07 PM

The last data point across all the metrics is from October 3rd http://datasets.wikimedia.org/aggregate-datasets/search/

The problem is not with rsync (I checked) -- it seems the R scripts are just not being run.

Okay, this is...really weird. I'll check it out.

Change 244355 had a related patch set uploaded (by OliverKeyes):
Switch out && statements to allow for runs when not all scripts complete

https://gerrit.wikimedia.org/r/244355

Change 244355 merged by Bearloga:
Switch out && statements to allow for runs when not all scripts complete

https://gerrit.wikimedia.org/r/244355

Ironholds set Security to None.Oct 8 2015, 4:52 PM
Ironholds edited a custom field.

The breakage of the scripts _overall_ is now fixed. We still need backfilling and to work out what happened with desktop.

Deskana closed this task as Resolved.Nov 20 2015, 5:24 AM
Deskana moved this task from Done to Resolved on the Discovery-Analysis (Current work) board.
Deskana added a subscriber: Deskana.