Page MenuHomePhabricator

Run pt-table-checksum on s1 (enwiki)
Closed, ResolvedPublic

Description

Run pt-table-checksum on s1 - enwiki.
The following tables do not have a PK and thus need to be excluded:

arbcom1_vote
categorylinks
click_tracking
click_tracking_user_properties
cur
edit_page_tracking
ep_users_per_course
exarchive
flaggedrevs_tracking
gather_list_flag
gather_list_item
hidden
imagelinks
interwiki
iwlinks
l10n_cache
langlinks
linkscc
localisation_file_hash
log_search
long_run_profiling
math
module_deps
msg_resource
msg_resource_links
objectcache
old_growth
oldimage
pagelinks
pagetriage_page_tags
prefstats
prefswitch_survey
profiling
querycache
querycache_info
querycachetwo
securepoll_lists
securepoll_msgs
securepoll_properties
site_identifiers
site_stats
templatelinks
text
transcache
user_former_groups
user_newtalk
user_properties
vote_log

The __wmf_checksums is placed on:

enwiki.__wmf_checksums

Details

Related Gerrit Patches:
operations/mediawiki-config : masterdb-eqiad.php: Repool db1067
operations/mediawiki-config : masterdb-codfw.php: Depool db2069
operations/mediawiki-config : masterdb-codfw.php: Depool db2062
operations/mediawiki-config : masterdb-eqiad.php: Depool db1083, fully repool db1089
operations/mediawiki-config : masterdb-eqiad.php: Slowly repool db1089, depool db1067
operations/mediawiki-config : masterdb-eqiad.php: Repool db1105:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089 and db1105:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089,db1066
operations/mediawiki-config : masterdb-eqiad.php: Depool db1067
operations/mediawiki-config : masterdb-eqiad.php: Depool db1065
operations/mediawiki-config : masterdb-eqiad.php: Repool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089, db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1105:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1066
operations/mediawiki-config : masterdb-eqiad.php: Depool db1067
operations/mediawiki-config : masterdb-eqiad.php: Repool db1067
operations/mediawiki-config : masterdb-eqiad.php: Repool db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089 and db1067
operations/mediawiki-config : masterdb-eqiad.php: Slowly repool db1080
operations/mediawiki-config : masterdb-eqiad.php: Depool db1080
operations/mediawiki-config : masterdb-eqiad.php: Repool db1073 with low weight
operations/mediawiki-config : masterdb-eqiad.php: Depool db1073
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1067
operations/mediawiki-config : masterdb-eqiad.php: Depool db1065
operations/mediawiki-config : masterdb-eqiad.php: Repool db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1105:3311
operations/mediawiki-config : masterdb-eqiad.php: Repool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Repool db1066, depool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Repool db1067, depool db1067
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089 and db1067
operations/mediawiki-config : masterdb-eqiad.php: Depool db1065
operations/mediawiki-config : masterdb-eqiad.php: Depool db1105:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php Depool db1066
operations/mediawiki-config : masterdb-eqiad.php: Depool db1067
operations/mediawiki-config : masterdb-eqiad.php: Depool db1065, pool db1067 for vslow
operations/mediawiki-config : masterdb-eqiad.php: Depool db1105:3311
operations/mediawiki-config : masterdb-eqiad.php: Repool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Repool db1066, depool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1066 and db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089 and db1067
operations/mediawiki-config : masterdb-eqiad.php: Depool db1065
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1105:3311,db1105:3312
operations/mediawiki-config : masterdb-eqiad.php: db1067,db1066,db1099:331{1,8}
operations/mediawiki-config : masterdb-eqiad.php: Depool db1099 s1 and s8
operations/mediawiki-config : masterdb-eqiad.php: Depool db1067,db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1067, db1089

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2018-02-09T07:50:57Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1080 - T162807 (duration: 01m 11s)

Mentioned in SAL (#wikimedia-operations) [2018-02-09T09:06:00Z] <marostegui> Fix data drifts on db1067 - T162807

Mentioned in SAL (#wikimedia-operations) [2018-02-09T09:37:56Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1080 - T162807 (duration: 01m 11s)

Mentioned in SAL (#wikimedia-operations) [2018-02-09T10:02:35Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1080 - T162807 (duration: 01m 12s)

Mentioned in SAL (#wikimedia-operations) [2018-02-09T10:29:43Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully repool db1080 - T162807 (duration: 01m 11s)

Change 409307 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089 and db1067

https://gerrit.wikimedia.org/r/409307

Change 409307 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089 and db1067

https://gerrit.wikimedia.org/r/409307

Mentioned in SAL (#wikimedia-operations) [2018-02-09T10:50:18Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089 and db1067 for data checksumming - T162807 (duration: 01m 11s)

Mentioned in SAL (#wikimedia-operations) [2018-02-09T10:51:21Z] <marostegui> Stop replication in sync on db1067 and db1089 - T162807

Change 409325 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1089

https://gerrit.wikimedia.org/r/409325

Change 409325 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1089

https://gerrit.wikimedia.org/r/409325

Mentioned in SAL (#wikimedia-operations) [2018-02-09T13:33:27Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1089 - T162807 (duration: 01m 12s)

Change 409366 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1067

https://gerrit.wikimedia.org/r/409366

Change 409366 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1067

https://gerrit.wikimedia.org/r/409366

Mentioned in SAL (#wikimedia-operations) [2018-02-09T16:38:29Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1067 - T162807 (duration: 01m 12s)

Change 409815 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/409815

Change 409815 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/409815

Mentioned in SAL (#wikimedia-operations) [2018-02-12T09:05:28Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089 - T162807 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2018-02-12T09:05:41Z] <marostegui> Stop replication in sync on db1089 and db2048 - T162807

Change 409824 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1067

https://gerrit.wikimedia.org/r/409824

Change 409824 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1067

https://gerrit.wikimedia.org/r/409824

Mentioned in SAL (#wikimedia-operations) [2018-02-12T09:46:24Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1067 - T162807 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2018-02-12T09:46:38Z] <marostegui> Stop replication in sync on db1089 and db1067 - T162807

Mentioned in SAL (#wikimedia-operations) [2018-02-12T09:57:38Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1067 - T162807 (duration: 00m 55s)

Change 409828 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/409828

Change 409828 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/409828

Mentioned in SAL (#wikimedia-operations) [2018-02-12T10:06:29Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1066 - T162807 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2018-02-12T10:07:02Z] <marostegui> Stop replication in sync on db1089 and db1066 - T162807

Mentioned in SAL (#wikimedia-operations) [2018-02-12T10:36:47Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1066 - T162807 (duration: 00m 55s)

Change 409880 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/409880

Change 409880 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/409880

Mentioned in SAL (#wikimedia-operations) [2018-02-12T13:42:56Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 - T162807 (duration: 01m 06s)

Mentioned in SAL (#wikimedia-operations) [2018-02-12T15:28:49Z] <marostegui> Stop replication in sync on db1089 and db1105:3311 - T162807

Mentioned in SAL (#wikimedia-operations) [2018-02-12T15:35:25Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 - T162807 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2018-02-12T15:43:31Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1089 - T162807 (duration: 00m 55s)

I have finished the text table.
Next: oldimage

Change 410121 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089, db1099:3311

https://gerrit.wikimedia.org/r/410121

Change 410121 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089, db1099:3311

https://gerrit.wikimedia.org/r/410121

Mentioned in SAL (#wikimedia-operations) [2018-02-13T08:51:55Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089, db1099 - T162807 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2018-02-13T08:52:10Z] <marostegui> Stop replication in sync on db1089 and db1099:3311 - T162807

Change 410123 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1099:3311

https://gerrit.wikimedia.org/r/410123

Change 410123 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1099:3311

https://gerrit.wikimedia.org/r/410123

Mentioned in SAL (#wikimedia-operations) [2018-02-13T09:01:25Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 - T162807 (duration: 00m 55s)

Change 410126 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1065

https://gerrit.wikimedia.org/r/410126

Change 410126 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1065

https://gerrit.wikimedia.org/r/410126

Mentioned in SAL (#wikimedia-operations) [2018-02-13T09:20:29Z] <marostegui> Stop replication in sync on db1089 and db1065 - T162807

Mentioned in SAL (#wikimedia-operations) [2018-02-13T09:20:44Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1065 - T162807 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2018-02-13T09:29:08Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1065 - T162807 (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2018-02-13T09:30:10Z] <marostegui> Stop replication in sync on db1089 and dbstore1002 - T162807

Change 410391 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1067

https://gerrit.wikimedia.org/r/410391

Change 410391 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1067

https://gerrit.wikimedia.org/r/410391

Mentioned in SAL (#wikimedia-operations) [2018-02-14T06:44:00Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1067 - T162807 (duration: 01m 12s)

Mentioned in SAL (#wikimedia-operations) [2018-02-14T06:44:13Z] <marostegui> Stop replication in sync db1089 and db1067 - T162807

Mentioned in SAL (#wikimedia-operations) [2018-02-14T10:42:43Z] <marostegui> Stop replication in sync on db1089 and db1067 - T162807

Mentioned in SAL (#wikimedia-operations) [2018-02-14T16:41:58Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1067 - T162807 (duration: 01m 12s)

Mentioned in SAL (#wikimedia-operations) [2018-02-14T16:58:59Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1089 - T162807 (duration: 01m 09s)

Change 410660 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089,db1066

https://gerrit.wikimedia.org/r/410660

Change 410660 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089,db1066

https://gerrit.wikimedia.org/r/410660

Mentioned in SAL (#wikimedia-operations) [2018-02-15T07:54:57Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089 and db1066 - T162807 (duration: 01m 12s)

Mentioned in SAL (#wikimedia-operations) [2018-02-15T07:55:55Z] <marostegui> Stop replication in sync on db1089 and db1066 - T162807

Mentioned in SAL (#wikimedia-operations) [2018-02-16T06:46:24Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089 and db1067 - T162807 (duration: 00m 59s)

Change 411194 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/411194

Change 411194 merged by Marostegui:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/411194

Mentioned in SAL (#wikimedia-operations) [2018-02-16T09:56:26Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 - T162807 (duration: 00m 56s)

I have finished the oldimage table.
Next: revision (which is in a pretty good state, so I don't expect it to take long)

Mentioned in SAL (#wikimedia-operations) [2018-02-16T12:23:35Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 - T162807 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2018-02-16T13:11:56Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1089 and db1067 - T162807 (duration: 00m 55s)

Change 412622 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089 and db1105:3311

https://gerrit.wikimedia.org/r/412622

Change 412622 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089 and db1105:3311

https://gerrit.wikimedia.org/r/412622

Mentioned in SAL (#wikimedia-operations) [2018-02-19T06:40:40Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089 and db1105 - T162807 (duration: 00m 56s)

Change 412655 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1105:3311

https://gerrit.wikimedia.org/r/412655

Change 412655 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1105:3311

https://gerrit.wikimedia.org/r/412655

Mentioned in SAL (#wikimedia-operations) [2018-02-19T09:18:30Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1105 - T162807 (duration: 00m 55s)

Revision table is done.
Next: watchlist (which is in a pretty good state, so it shouldn't take long)

watchlist table is done.
Next: ores_classification

Change 413329 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1089, depool db1067

https://gerrit.wikimedia.org/r/413329

Change 413329 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1089, depool db1067

https://gerrit.wikimedia.org/r/413329

Mentioned in SAL (#wikimedia-operations) [2018-02-22T10:00:53Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1089 with low traffic and depool db1067 - T162807 (duration: 01m 12s)

Change 413678 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/413678

Change 413678 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083, fully repool db1089

https://gerrit.wikimedia.org/r/413678

Mentioned in SAL (#wikimedia-operations) [2018-02-23T07:58:05Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1083, fully repool db1089 - T162807 (duration: 01m 12s)

Change 415235 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2062

https://gerrit.wikimedia.org/r/415235

Change 415235 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2062

https://gerrit.wikimedia.org/r/415235

Mentioned in SAL (#wikimedia-operations) [2018-02-28T07:51:35Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2062 - T162807 (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2018-02-28T08:23:22Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2062 - T162807 (duration: 00m 57s)

Change 415241 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2069

https://gerrit.wikimedia.org/r/415241

Change 415241 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2069

https://gerrit.wikimedia.org/r/415241

Mentioned in SAL (#wikimedia-operations) [2018-02-28T08:33:40Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2069 - T162807 (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2018-02-28T08:55:17Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2069 - T162807 (duration: 00m 57s)

Change 415808 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1067

https://gerrit.wikimedia.org/r/415808

Change 415808 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1067

https://gerrit.wikimedia.org/r/415808

Mentioned in SAL (#wikimedia-operations) [2018-03-02T07:45:38Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1067 - T162807 (duration: 00m 57s)

ores_classification table is done.
Next (and last, pending another full recheck): user

Marostegui closed this task as Resolved.Mar 13 2018, 9:00 AM

user table is done.

So I consider this ticket resolved. _All_ the tables have been checked by using:

  • leftovers from pt-table-checksum
  • By running mydumper across all the "two" batches of servers we had (I explain that below). To check for all the tables differences.
  • compare.py (I will talk about this below too).

The following tables had differences and have been fixed.

archive
change_tag
geo_tags
oldimage
ores_classification
tag_summary
text
user_newtalk
watchlist
revision
user

It took quite long as some tables were pretty big and there were quite lots of different rows on some of them.
During all the checks (combining compare.py and the left overs from pt-table-checksum) I noticed that there were "two versions" of some rows. The oldest hosts (db1067, db1066) had different information from the rest of the hosts in eqiad and codfw hosts.
That was confirmed by doing a full mydumper across those and the rest of hosts in eqiad.

That was also confirmed by compare.py and its new features of being able to compare multiple hosts across them. Those were coded by Jaime lately and they have been proved to be _super_ useful and simplified these checks a lot. Being able to run all of them at once and in a very fast way.
This ticket would have taken a lot longer with those new options done by Jaime.

All the tables were fixed and I did a re-check using compare.py across all the hosts again, comparing them against each other for the tables that previously showed differences, and they are now all the same.

There is a special case of a table which is pending to fix, just on the master, which is user_newtalk. Again, just on the master (T186503).
Also a follow up - not part of this ticket/goal - is to schedule periodic table checks.
This should be done slowly and for a couple of tables to start with probably. Using the parallel features of compare.py this is pretty easy to set up and schedule.

I would start by tables like change_tag or tag_summary which have had issues historically (because of the lack of PKs), and are now fixed across all the shards.

So all in all, it is hard to say we have fixed 100% of the issues, but I believe we are in a pretty pretty good state now on s1.