According to https://grafana.wikimedia.org/dashboard/db/server-board?panelId=17&fullscreen&from=1440501127031&to=1448280487032&var-server=db1046&var-network=eth0 we will run out of disk space in 3-4 weeks. Once these happens, no more logs will be able to be written.
Description
Description
Related Objects
Related Objects
- Mentioned In
- T120187: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores)
T108857: Delete obsolete schemas {tick} [5 pts] - Mentioned Here
- T119144: EventLogging sees too few distinct client IPs {oryx} [8 pts]
T108856: Set up bucketization of editCount fields {tick}
Event Timeline
Comment Actions
@jcrespo: I think m4-master holds on to data for too long. Since everything is replicated to analytics-store, I think we can shorten the amount of time data lives on m4-master. Here is the range of data available on m4-master:
mysql:eventlog@m4-master.eqiad.wmnet [log]> select min(timestamp), max(timestamp) from Edit_11448630; +----------------+----------------+ | min(timestamp) | max(timestamp) | +----------------+----------------+ | 20150710141400 | 20151122172443 | +----------------+----------------+
And here it is on analytics-store:
mysql:research@analytics-store.eqiad.wmnet [log]> select min(timestamp), max(timestamp) from Edit_11448630; +----------------+----------------+ | min(timestamp) | max(timestamp) | +----------------+----------------+ | 20150316150859 | 20151122172443 | +----------------+----------------+
So as long as analytics-store isn't running out of space, I think the solution here is to shorten the window of time data lives on m4-master to maybe no more than 30 days.
The other concerns in the email was that incoming data has increased a lot recently. I couldn't prove this theory though. Event Logging has been collecting data at a pretty steady rate as far back as I can see. And the 10 largest tables have gathered the same or less data than they normally do. Some of those even stopped completely so deleting old data should help a lot there. So that does leave the question of why you're seeing more and more data. This is what I did to get table sizes:
SELECT (data_length + index_length) as total_size,
table_schema,
table_name,
data_length,
index_length
FROM information_schema.tables
ORDER BY total_size desc;Can you check that real quick and let us know if it accounts for all the space used up on that server? If not, maybe something else is creating data there besides Event Logging?
Comment Actions
This is the physical view:
root@db1046:/srv$ df -h | grep /srv /dev/mapper/tank-data 1.4T 1.3T 136G 91% /srv root@db1046:/srv$ du -h --max-depth=2 663G ./sqldata/log 119M ./sqldata/mysql 4.0K ./sqldata/test 24K ./sqldata/heartbeat 220K ./sqldata/performance_schema 1.3T ./sqldata 0 ./tmp 1.3T . root@db1046:/srv/sqldata$ ls -lha --sort=size | head total 615G -rwxrwx--x 1 mysql mysql 88G Nov 24 17:21 _log_sql_2343_1a_main_5_1_1b_B_0.tokudb -rwxrwx--x 1 mysql mysql 75G Nov 23 09:09 _log_sql_2343_3bb52_main_2476c99_1_1b_B_0.tokudb -rw-rw---- 1 mysql mysql 43G Nov 24 10:28 _log_sql_6dc5_d85c80b_main_30627a8a_2_1b.tokudb -rw-rw---- 1 mysql mysql 33G Nov 24 17:20 ibdata1 -rwxrwx--x 1 mysql mysql 29G Nov 24 17:20 _log_sql_2343_2f86c_main_16a4401_1_1b_B_0.tokudb -rw-rw---- 1 mysql mysql 28G Nov 24 17:04 _log_sql_6dc5_407_main_902ff62_2_1b.tokudb -rwxrwx--x 1 mysql mysql 21G Nov 23 20:44 _log_sql_2343_3bb52_key_ix_MobileWebClickTracking_5929948_uuid_2476c99_1_1b_B_1.tokudb -rwxrwx--x 1 mysql mysql 21G Nov 24 17:19 _log_sql_2343_69cb3_main_5b30446_1_1b_B_0.tokudb -rwxrwx--x 1 mysql mysql 16G Nov 24 17:04 _log_sql_2343_15cd1_main_1297021_1_1b_B_0.tokudb root@db1046:/srv/sqldata$ cd log root@db1046:/srv/sqldata/log$ ls -lha --sort=size | head total 663G -rw-rw---- 1 mysql mysql 121G Nov 24 17:19 Edit_13457736.ibd -rw-rw---- 1 mysql mysql 75G Nov 24 17:13 MobileWikiAppArticleSuggestions_11448426.ibd -rw-rw---- 1 mysql mysql 74G Nov 24 17:17 MobileWikiAppSearch_10641988.ibd -rw-rw---- 1 mysql mysql 51G Nov 24 17:18 MobileWikiAppArticleSuggestions_12443791.ibd -rw-rw---- 1 mysql mysql 37G Nov 24 17:20 MobileWebSectionUsage_14321266.ibd -rw-rw---- 1 mysql mysql 20G Nov 24 16:48 ImageMetricsCorsSupport_11686678.ibd -rw-rw---- 1 mysql mysql 17G Nov 24 16:48 NavigationTiming_12405818.ibd -rw-rw---- 1 mysql mysql 17G Nov 24 17:19 EchoInteraction_5782287.ibd -rw-rw---- 1 mysql mysql 16G Nov 24 17:18 ContentTranslationCTA_11616099.ibd
Space is not a huge issue by itself, it is only a problem if we are not using it efficiently (for example, as you said, if more data than needed is being stored).
The idea would be to setup a process to purge it from the master, while slaves can keep more data as they have larger disks. Is that something that you want to do controlled by eventlogging itself or exclusively at server side?
I am afraid that for it to be effective, at least some part has to be at server side, as deleting rows will not effectively reduce the disk space used, and server-side operations like defragmenting or partitioning may be needed (that we can setup automatically with puppet). This will also help speed up other maintenance operations, like schema changes, performed in the past.
I can give you a specific proposal for the master maintenance if you agree with the general idea, and once I had researched the sizes and timestamps.
Comment Actions
@jcrespo, thanks very much for the physical report. As far as team Analytics is concerned, we only need enough data on m4-master to facilitate backfilling. So, if we have a partial outage and we need to re-insert data, we need any originally inserted data from that period to be in there so we don't insert duplicates. We have no other purpose for this server, as far as we are concerned it's just used to replicate data to analytics-store. For us, having the last 30 days of data in there, on a sliding window, will work fine. We'll just create the expectation that any backfilling has to happen within 30 days of the outage. For you, you may need to keep the data around longer if you think it may take longer to fix any replication issues that come up.
If we made the replication upsert instead of insert rows, as far as we're concerned we could keep only a couple of days of data on m4-master.
Your other concern from the email remains. Do you still see an increase in the rate of new data? If you do, we should probably figure out why that's happening, separate from the above. Because there's no explanation for it from the facts I know.
Comment Actions
I see two accelerations, on the 27 sep and on the 7 nov. There could be many explanations, from long running transactions being executed there, to the schema changes done as part of T108856. I have not performed per-file monitoring prior to this date, but I can tell you exactly what is growing in a few days.
I have just one question, when and who decides when new tables are to be created within a schema? It will simplify a lot of operations if new tables were created regularly when the number of records is high, so we could drop instantly old ones on the master. It would also probably speed up the import time, too.
Comment Actions
Looking forward to finding out more.
I have just one question, when and who decides when new tables are to be created within a schema? It will simplify a lot of operations if new tables were created regularly when the number of records is high, so we could drop instantly old ones on the master. It would also probably speed up the import time, too.
This is done automatically when events with a new <schema, revision_id> come in (because otherwise they'd have no table to write to). And we can't really drop old revision_ids for schemas in an automated way (because sometimes code take a long time to stop writing to the old schemas. For example, say an Android app is writing that code, we'd have to wait for everyone to upgrade their app before we deleted the old table.
Comment Actions
And we can't really drop old revision_ids for schemas in an automated way
Actually, I wasn't suggesting that, just doing the creation of new tables regularly, even if the revision_id doesn't change, and maintain smaller tables all the time. Then, drop older tables only based on the most recent date, no matter if there are many or just one.
Alternatively, I can handle that with mysql partitioning, but it has some drawbacks.
Comment Actions
jynus@db1046:/srv$ df -h | grep /srv /dev/mapper/tank-data 1.4T 1.3T 106G 93% /srv jynus@db1046:/srv$ du -h --max-depth=2 691G ./sqldata/log 119M ./sqldata/mysql 4.0K ./sqldata/test 24K ./sqldata/heartbeat 220K ./sqldata/performance_schema 1.3T ./sqldata 0 ./tmp 1.3T . jynus@db1046:/srv$ ls -lha --sort=size sqldata/ | head total 617G -rwxrwx--x 1 mysql mysql 89G Nov 30 09:20 _log_sql_2343_1a_main_5_1_1b_B_0.tokudb -rwxrwx--x 1 mysql mysql 75G Nov 27 11:11 _log_sql_2343_3bb52_main_2476c99_1_1b_B_0.tokudb -rw-rw---- 1 mysql mysql 43G Nov 30 05:30 _log_sql_6dc5_d85c80b_main_30627a8a_2_1b.tokudb -rw-rw---- 1 mysql mysql 33G Nov 30 09:21 ibdata1 -rwxrwx--x 1 mysql mysql 30G Nov 30 09:20 _log_sql_2343_2f86c_main_16a4401_1_1b_B_0.tokudb -rw-rw---- 1 mysql mysql 28G Nov 30 09:20 _log_sql_6dc5_407_main_902ff62_2_1b.tokudb -rwxrwx--x 1 mysql mysql 21G Nov 28 17:17 _log_sql_2343_3bb52_key_ix_MobileWebClickTracking_5929948_uuid_2476c99_1_1b_B_1.tokudb -rwxrwx--x 1 mysql mysql 21G Nov 30 09:19 _log_sql_2343_69cb3_main_5b30446_1_1b_B_0.tokudb -rwxrwx--x 1 mysql mysql 16G Nov 30 08:49 _log_sql_2343_15cd1_main_1297021_1_1b_B_0.tokudb jynus@db1046:/srv$ ls -lha --sort=size sqldata/log | head total 691G -rw-rw---- 1 mysql mysql 131G Nov 30 09:22 Edit_13457736.ibd -rw-rw---- 1 mysql mysql 75G Nov 30 09:20 MobileWikiAppArticleSuggestions_11448426.ibd -rw-rw---- 1 mysql mysql 74G Nov 30 09:18 MobileWikiAppSearch_10641988.ibd -rw-rw---- 1 mysql mysql 51G Nov 30 09:17 MobileWikiAppArticleSuggestions_12443791.ibd -rw-rw---- 1 mysql mysql 48G Nov 30 09:21 MobileWebSectionUsage_14321266.ibd -rw-rw---- 1 mysql mysql 20G Nov 30 07:24 ImageMetricsCorsSupport_11686678.ibd -rw-rw---- 1 mysql mysql 17G Nov 30 09:00 NavigationTiming_12405818.ibd -rw-rw---- 1 mysql mysql 17G Nov 30 09:19 ContentTranslationCTA_11616099.ibd -rw-rw---- 1 mysql mysql 17G Nov 30 09:17 EchoInteraction_5782287.ibd
The Edit tables is the main responsible for the growth. It should be converted to TokuDB, toghether with other tables. However, that cannot be done immediatelly, I will check what are the more urgent operations.
Comment Actions
This is a list of the first record on db1046 for each table:
mysql -A -BN -h db1046 log -e "SELECT table_name FROM information_schema.columns WHERE column_name='timestamp' and table_schema='log'" | while read table; do mysql -A -BN -h db1046 log -e "SELECT concat('$table: ', timestamp) FROM \`$table\` ORDER BY timestamp ASC LIMIT 1"; done
CentralAuth_5690875: 20150710141400
CompletionSuggestions_13424343: 20150909154807
CompletionSuggestions_13630018: 20150915212307
ContentTranslationCTA_11616099: 20150710141400
ContentTranslationError_11767097: 20150710225202
ContentTranslation_11628043: 20150710141409
DeprecatedUsage_7906187: 20150710141404
DidYouMean_13316693: 20150902183717
DidYouMean_13800499: 20151007012041
EchoInteraction_5782287: 20150710141403
EchoMail_5467650: 20150710141405
Echo_7731316: 20150710141400
EditConflict_8860941: 20150710141403
Edit_11448630: 20150710141400
Edit_13457736: 20150907214239
EditorActivation_14208837: 20151028205314
ExtDistDownloads_12369387: 20150710141537
FlowReplies_10561344: 20150710142105
GatherClicks_12114785: 20150710141408
GatherFlags_11793295: 20150710183943
GeoFeatures_12518424: 20150721230904
GeoFeatures_12914994: 20150803231959
GettingStartedRedirectImpression_7355552: 20150710141405
GuidedTourButtonClick_13869649: 20150929233655
GuidedTourButtonClick_8690550: 20150710141503
GuidedTourExited_8690566: 20150710141555
GuidedTourExternalLinkActivation_8690560: 20150710141628
GuidedTourGuiderHidden_8690549: 20150710141903
GuidedTourGuiderImpression_8694395: 20150710141459
GuidedTourInternalLinkActivation_8690553: 20150711061034
HttpsSupport_11518527: 20150710141621
ImageMetricsCorsSupport_10884476: 20150911082200
ImageMetricsCorsSupport_11686678: 20150710141400
ImageMetricsLoadingTime_10078363: 20150710141404
MediaViewer_10308479: 20150804121600
MediaViewer_10606177: 20151122121607
MediaViewer_10867062: 20150710141401
MediaViewer_8572637: 20150719195049
MediaViewer_8935662: 20150712200134
MediaViewer_9989959: 20150714200132
MobileAppCategorizationAttempts_5359208: 20150711151049
MobileAppLoginAttempts_5257721: 20150710194655
MobileAppShareAttempts_5346170: 20150715051731
MobileAppTrackingChange_5369400: 20150711193113
MobileAppTrackingChange_5412592: 20150713224501
MobileAppUploadAttempts_5257716: 20150731185743
MobileAppUploadAttempts_5334329: 20150711011407
MobileBetaWatchlist_5281061: 20150714084053
MobileOptionsTracking_14003392: 20151008233747
MobileOptionsTracking_8101982: 20150710141409
MobileWebBrowse_12119641: 20150710141408
MobileWebClickTracking_5929948: 20140101000000
MobileWebDiffClickTracking_10720373: 20150710141404
MobileWebEditing_8599025: 20150710141402
MobileWebInfobox_6221064: 20150715181650
MobileWebMainMenuClickTracking_10703095: 20150730084741
MobileWebMainMenuClickTracking_11568715: 20150710141402
MobileWebSearch_12054448: 20150710141507
MobileWebSectionUsage_14321266: 20151103201401
MobileWebUIClickTracking_10742159: 20150710141402
MobileWebWatching_11761466: 20150710141452
MobileWebWatchlistClickTracking_10720361: 20150710141642
MobileWebWikiGrok_10352247: 20150717175120
MobileWikiAppAppearanceSettings_10375462: 20150710141409
MobileWikiAppAppearanceSettings_9378399: 20150710142408
MobileWikiAppArticleSuggestions_10590869: 20150710141408
MobileWikiAppArticleSuggestions_11448426: 20150710141403
MobileWikiAppArticleSuggestions_12443791: 20150710141403
MobileWikiAppBannerClickThrough_13295306: 20150825174413
MobileWikiAppCreateAccount_8240702: 20150710141453
MobileWikiAppCreateAccount_9135391: 20150710141406
MobileWikiAppDailyStats_12637385: 20150717215522
MobileWikiAppEdit_8198182: 20150714145553
MobileWikiAppEdit_8994704: 20150710150621
MobileWikiAppEdit_9003125: 20150710141405
MobileWikiAppFindInPage_14586774: 20151112230459
MobileWikiAppInstallReferrer_12601905: 20150720195759
MobileWikiAppLangSelect_12588733: 20150717221925
MobileWikiAppLinkPreview_12143205: 20150710141405
MobileWikiAppLinkPreview_14095177: 20151023195245
MobileWikiAppLogin_8234533: 20150710141411
MobileWikiAppLogin_9135390: 20150710141407
MobileWikiAppMediaGallery_10923135: 20150710141405
MobileWikiAppMediaGallery_12588701: 20150717215914
MobileWikiAppNavMenu_12732211: 20150723194452
MobileWikiAppOnboarding_9123466: 20150710141406
MobileWikiAppOperatorCode_8983918: 20150710141405
MobileWikiAppPageScroll_14591606: 20151112215610
MobileWikiAppProtectedEditAttempt_8682497: 20150710141434
MobileWikiAppSavedPages_10375480: 20150710141405
MobileWikiAppSavedPages_8909354: 20150710141413
MobileWikiAppSearch_10641988: 20150710141405
MobileWikiAppSessions_14031591: 20151023235702
MobileWikiAppSessions_9742902: 20150713054531
MobileWikiAppShareAFact_10916168: 20150720041346
MobileWikiAppShareAFact_11331974: 20150710141405
MobileWikiAppShareAFact_12588711: 20150709203036
MobileWikiAppStuffHappens_8955468: 20150710141406
MobileWikiAppTabs_12453651: 20150717220103
MobileWikiAppToCInteraction_10375484: 20150710141405
MobileWikiAppToCInteraction_11014396: 20150710141410
MobileWikiAppToCInteraction_14585319: 20151113033414
MobileWikiAppToCInteraction_8461467: 20150710141406
MobileWikiAppWidgets_11312870: 20150710141406
ModuleStorage_6978194: 20151024180205
MultimediaViewerAttribution_9758179: 20150710141408
MultimediaViewerDimensions_10014238: 20150710141406
MultimediaViewerDuration_10427980: 20150710141407
MultimediaViewerNetworkPerformance_12458951: 20150710141406
MultimediaViewerVersusPageFilePerformance_7907636: 20150711025255
NavigationTiming_10374055: 20150714072446
NavigationTiming_10785754: 20150710141844
NavigationTiming_12405818: 20150710141406
NavigationTiming_13317958: 20151104221927
NavigationTiming_13317958_: 20150902183656
NavigationTiming_13332008: 20150908171006
NavigationTiming_5832704: 20151002055749
NavigationTiming_6703470: 20150904171159
NavigationTiming_7494934: 20150803000145
NavigationTiming_8365252: 20150716153328
NewEditorEdit_6792669: 20150710141409
PageContentSaveComplete_5588433: 20150710141408
PageCreation_7481635: 20150710141409
PageDeletion_7481655: 20150710141521
PageMove_7495717: 20150710141450
PageRestoration_7758372: 20150710142944
PersonalBar_7829128: 20150710141408
Popups_11625443: 20150710141408
PrefUpdate_5563398: 20150710141409
SaveTiming_12236257: 20150710141408
Search_11670541: 20151026154648
Search_12057910: 20150710141409
Search_14361785: 20151028234955
ServerSideAccountCreation_5487345: 20150710141422
TestSearchSatisfaction2_13223897: 20150902184511
TestSearchSatisfaction2_14098806: 20151013232737
TestSearchSatisfaction2_14318467: 20151026151001
TestSearchSatisfaction_12423691: 20150702065831
UniversalLanguageSelector-tofu_7629564: 20150101034512
UniversalLanguageSelector_7327441: 20150710141409
UploadWizardErrorFlowEvent_11772725: 20150710141435
UploadWizardErrorFlowEvent_9924376: 20150728034619
UploadWizardExceptionFlowEvent_11772722: 20150710141610
UploadWizardFlowEvent_11772723: 20150710141938
UploadWizardStep_11772724: 20150710141447
UploadWizardStep_8851805: 20150720083929
UploadWizardTutorialActions_5803466: 20150710141447
UploadWizardUploadFlowEvent_11772717: 20150710141416
UploadWizardUploadFlowEvent_9651951: 20150728034238
WikimediaBlogVisit_5308166: 20150710141431
WikipediaPortal_14377354: 20151110210521
_NavigationTiming_13317958__new: 20150902183829I wouls delete half of the records on the largest tables (only on the master, not on the slaves) and convert them to TokuDB, giving us a 10x or more compression rate.
Comment Actions
At the moment it is done manually. EventLogging's approach is to try and insert incoming events into the database on the assumption that a table for the given schema / revision already exists, and to fall back to declaring the table if the insert fails because the table doesn't exist. So you can just rename a table (I've added a '_' prefix in the past) and data will automatically go into a new table. We just have to make sure that whomever (or whatever) is analyzing the data knows about the old table.
If we get agreement on T119144, we could potentially drop the clientIp column (varchar(191)) from all tables.
Comment Actions
If we get agreement on T119144, we could potentially drop the clientIp column (varchar(191)) from all tables.
Dropping columns is not an investment worth pursuing. Partitioning (at table level, or with MySQL's partitioning) is, because it simplifies greatly the management (dropping partitions is direct and does not create fragmentation) plus it usually improves the performance (only the latest rows are scanned).
In other words, it would be great if ori's method could be a logic integrated into the app, so no rename is needed, and it happens automatically.
Comment Actions
So there seem to be two threads here.
Table level partitioning seems to me to complicate replication to the slaves and complicate application logic. It doesn't seem to me that we need to do this right now, we can save plenty of space by just deleting data we absolutely don't need.
Deleting *any* data older than 30 days in *any* table is fine with us, @jcrespo, as long as these deletes don't replicate over to the slaves. If that's the case, we could run a job that did that on a daily basis. If we ever for some reason need to keep data around longer to allow back-filling, we could just pause that job. Let us know if this makes sense and I'll write and puppetize it.
Comment Actions
@Milimetric: deleting data will not solve immediately the problem, as deleting data logically doesn't mean space is freed from disk. Hence the partitioning suggestion.
"complicate replication to the slaves and complicate application logic"
Not at all, it could be done transparently to the application. But requires maintenance.
But we can talk about that later, I will try to do some magic for now without blocking the server.
Comment Actions
BTW, I found the acceleration issue: the automatic purge process was failing since some tables had been deleted.
Comment Actions
Immediate problems have been fixed and purging has been restarted, however, the long term problem persist until we can schedule some maintenance for defragmenting and converting tables to tokudb, and setting partitioning. Until then, this is a mere patch (real disk space has not been saved).
root@db1046:/srv$ df -h | grep /srv /dev/mapper/tank-data 1.6T 1.3T 320G 81% /srv root@db1046:/srv$ cd sqldata/ root@db1046:/srv/sqldata$ ls -lha --sort=size | head total 658G -rwxrwx--x 1 mysql mysql 89G Dec 1 16:51 _log_sql_2343_1a_main_5_1_1b_B_0.tokudb -rwxrwx--x 1 mysql mysql 75G Nov 27 11:11 _log_sql_2343_3bb52_main_2476c99_1_1b_B_0.tokudb -rw-rw---- 1 mysql mysql 43G Dec 1 16:46 _log_sql_6dc5_d85c80b_main_30627a8a_2_1b.tokudb -rw-rw---- 1 mysql mysql 33G Dec 1 16:53 ibdata1 -rwxrwx--x 1 mysql mysql 30G Dec 1 16:50 _log_sql_2343_2f86c_main_16a4401_1_1b_B_0.tokudb -rw-rw---- 1 mysql mysql 28G Dec 1 16:53 _log_sql_6dc5_407_main_902ff62_2_1b.tokudb -rwxrwx--x 1 mysql mysql 21G Nov 28 17:17 _log_sql_2343_3bb52_key_ix_MobileWebClickTracking_5929948_uuid_2476c99_1_1b_B_1.tokudb -rwxrwx--x 1 mysql mysql 21G Dec 1 16:52 _log_sql_2343_69cb3_main_5b30446_1_1b_B_0.tokudb -rwxrwx--x 1 mysql mysql 16G Dec 1 16:48 _log_sql_2343_15cd1_main_1297021_1_1b_B_0.tokudb root@db1046:/srv/sqldata$ cd log root@db1046:/srv/sqldata/log$ ls -lha --sort=size | head total 632G -rw-rw---- 1 mysql mysql 133G Dec 1 16:53 Edit_13457736.ibd -rw-rw---- 1 mysql mysql 75G Dec 1 16:53 MobileWikiAppArticleSuggestions_11448426.ibd -rw-rw---- 1 mysql mysql 74G Dec 1 16:53 MobileWikiAppSearch_10641988.ibd -rw-rw---- 1 mysql mysql 51G Dec 1 16:52 MobileWikiAppArticleSuggestions_12443791.ibd -rw-rw---- 1 mysql mysql 50G Dec 1 16:53 MobileWebSectionUsage_14321266.ibd -rw-rw---- 1 mysql mysql 20G Dec 1 16:49 ImageMetricsCorsSupport_11686678.ibd -rw-rw---- 1 mysql mysql 17G Dec 1 16:51 NavigationTiming_12405818.ibd -rw-rw---- 1 mysql mysql 17G Dec 1 16:51 ContentTranslationCTA_11616099.ibd -rw-rw---- 1 mysql mysql 16G Nov 30 16:44 HttpsSupport_11518527.ibd
mysql -A -BN -h db1046 log -e "SELECT table_name FROM information_schema.columns WHERE column_name='timestamp' and table_schema='log'" | while read table; do mysql -A -BN -h db1046 log -e "SELECT concat('$table: ', timestamp) FROM \`$table\` ORDER BY timestamp ASC LIMIT 1"; done
CentralAuth_5690875: 20150720102304
CompletionSuggestions_13424343: 20150909154807
CompletionSuggestions_13630018: 20150915212307
ContentTranslationCTA_11616099: 20150715154347
ContentTranslationError_11767097: 20151003074549
ContentTranslation_11628043: 20151002164900
DeprecatedUsage_7906187: 20150713043050
DidYouMean_13316693: 20150902183717
DidYouMean_13800499: 20151007012041
EchoInteraction_5782287: 20150714184800
EchoMail_5467650: 20151002164518
Echo_7731316: 20150726225402
EditConflict_8860941: 20151002164533
Edit_11448630: 20150713074457
Edit_13457736: 20150907214239
EditorActivation_14208837: 20151028205314
ExtDistDownloads_12369387: 20151002164814
GatherClicks_12114785: 20151002164538
GatherFlags_11793295: 20151003234318
GeoFeatures_12518424: 20150721230904
GeoFeatures_12914994: 20150803231959
GettingStartedRedirectImpression_7355552: 20151002164529
GuidedTourButtonClick_13869649: 20150929233655
GuidedTourButtonClick_8690550: 20151013212858
GuidedTourExited_8690566: 20151002164625
GuidedTourExternalLinkActivation_8690560: 20151002170530
GuidedTourGuiderHidden_8690549: 20151002165309
GuidedTourGuiderImpression_8694395: 20151002164520
GuidedTourInternalLinkActivation_8690553: 20151002213649
ImageMetricsCorsSupport_11686678: 20150713182018
ImageMetricsLoadingTime_10078363: 20150929121036
MediaViewer_10308479: 20151018093726
MediaViewer_10606177: 20151122121607
MediaViewer_10867062: 20150711204557
MediaViewer_8935662: 20151003213255
MobileAppCategorizationAttempts_5359208: 20151002171133
MobileAppLoginAttempts_5257721: 20151002173948
MobileAppShareAttempts_5346170: 20151003180559
MobileAppTrackingChange_5369400: 20151005054456
MobileAppTrackingChange_5412592: 20151003172650
MobileAppUploadAttempts_5257716: 20151005172608
MobileAppUploadAttempts_5334329: 20151002164705
MobileBetaWatchlist_5281061: 20151008160315
MobileOptionsTracking_14003392: 20151008233747
MobileOptionsTracking_8101982: 20150820052109
MobileWebBrowse_12119641: 20150730223629
MobileWebClickTracking_5929948: 20140101000000
MobileWebDiffClickTracking_10720373: 20151002164812
MobileWebEditing_8599025: 20150711175313
MobileWebMainMenuClickTracking_10703095: 20151010084627
MobileWebMainMenuClickTracking_11568715: 20150722110713
MobileWebSearch_12054448: 20150822050019
MobileWebSectionUsage_14321266: 20151103201401
MobileWebUIClickTracking_10742159: 20150710205732
MobileWebWatching_11761466: 20151002164935
MobileWebWatchlistClickTracking_10720361: 20151002164909
MobileWikiAppAppearanceSettings_10375462: 20150920095820
MobileWikiAppAppearanceSettings_9378399: 20151002172352
MobileWikiAppArticleSuggestions_10590869: 20150810172910
MobileWikiAppArticleSuggestions_11448426: 20150712130113
MobileWikiAppArticleSuggestions_12443791: 20150711061146
MobileWikiAppBannerClickThrough_13295306: 20150825174413
MobileWikiAppCreateAccount_8240702: 20151002165115
MobileWikiAppCreateAccount_9135391: 20150927192635
MobileWikiAppDailyStats_12637385: 20150717215522
MobileWikiAppEdit_8198182: 20151004063021
MobileWikiAppEdit_8994704: 20151003151058
MobileWikiAppEdit_9003125: 20150728050300
MobileWikiAppFindInPage_14586774: 20151112230459
MobileWikiAppInstallReferrer_12601905: 20150720195759
MobileWikiAppLangSelect_12588733: 20150717221925
MobileWikiAppLinkPreview_12143205: 20150727124824
MobileWikiAppLinkPreview_14095177: 20151023195245
MobileWikiAppLogin_8234533: 20151002165206
MobileWikiAppLogin_9135390: 20150822022659
MobileWikiAppMediaGallery_10923135: 20150712175134
MobileWikiAppMediaGallery_12588701: 20150717215914
MobileWikiAppNavMenu_12732211: 20150723194452
MobileWikiAppOnboarding_9123466: 20151002165151
MobileWikiAppOperatorCode_8983918: 20150725183025
MobileWikiAppPageScroll_14591606: 20151112215610
MobileWikiAppProtectedEditAttempt_8682497: 20151002165157
MobileWikiAppSavedPages_10375480: 20150725081519
MobileWikiAppSavedPages_8909354: 20151002165154
MobileWikiAppSearch_10641988: 20150711040559
MobileWikiAppSessions_14031591: 20151023235702
MobileWikiAppSessions_9742902: 20151002175435
MobileWikiAppShareAFact_11331974: 20150716123002
MobileWikiAppShareAFact_12588711: 20150709203036
MobileWikiAppStuffHappens_8955468: 20151002165150
MobileWikiAppTabs_12453651: 20150717220103
MobileWikiAppToCInteraction_10375484: 20150712080631
MobileWikiAppToCInteraction_11014396: 20151002000429
MobileWikiAppToCInteraction_14585319: 20151113033414
MobileWikiAppToCInteraction_8461467: 20150721015135
MobileWikiAppWidgets_11312870: 20151002165148
ModuleStorage_6978194: 20151024180205
MultimediaViewerAttribution_9758179: 20150902191457
MultimediaViewerDimensions_10014238: 20150725113524
MultimediaViewerDuration_10427980: 20151002165158
MultimediaViewerNetworkPerformance_12458951: 20150803015723
MultimediaViewerVersusPageFilePerformance_7907636: 20151003025528
NavigationTiming_10374055: 20151004102250
NavigationTiming_10785754: 20151002200249
NavigationTiming_12405818: 20150713155324
NavigationTiming_13317958: 20151104221927
NavigationTiming_13317958_: 20150902183656
NavigationTiming_13332008: 20150908171006
NavigationTiming_5832704: 20151105001650
NavigationTiming_6703470: 20151029033952
NavigationTiming_7494934: 20151102184851
NavigationTiming_8365252: 20151010083646
NewEditorEdit_6792669: 20150829224603
PageContentSaveComplete_5588433: 20150711091453
PageCreation_7481635: 20150721191457
PageDeletion_7481655: 20151002165502
PageMove_7495717: 20151002165432
PageRestoration_7758372: 20151002170324
PersonalBar_7829128: 20150716142954
Popups_11625443: 20151002165428
PrefUpdate_5563398: 20150827103500
SaveTiming_12236257: 20150715025528
Search_11670541: 20151026154648
Search_12057910: 20150721151547
Search_14361785: 20151028234955
ServerSideAccountCreation_5487345: 20151002164350
TestSearchSatisfaction2_13223897: 20150902184511
TestSearchSatisfaction2_14098806: 20151013232737
TestSearchSatisfaction2_14318467: 20151026151001
TestSearchSatisfaction_12423691: 20150702065831
UniversalLanguageSelector-tofu_7629564: 20150101034512
UniversalLanguageSelector_7327441: 20150728204640
UploadWizardErrorFlowEvent_11772725: 20151002164500
UploadWizardErrorFlowEvent_9924376: 20151007152928
UploadWizardExceptionFlowEvent_11772722: 20151002164341
UploadWizardFlowEvent_11772723: 20151002165253
UploadWizardStep_11772724: 20150909192243
UploadWizardTutorialActions_5803466: 20150906222716
UploadWizardUploadFlowEvent_11772717: 20151002164358
UploadWizardUploadFlowEvent_9651951: 20151007152121
WikimediaBlogVisit_5308166: 20151002164502
WikipediaPortal_14377354: 20151110210521Comment Actions
@jcrespo, just open up a different task and assign it to me if I can help in any way. Don't worry about tagging analytics projects, it seems everyone's confused about how we do that so I'll take care of it.