User Details
- User Since
- Aug 2 2021, 1:52 PM (18 w, 2 d)
- Availability
- Available
- LDAP User
- MVernon
- MediaWiki User
- MVernon (WMF) [ Global Accounts ]
Yesterday
> AFAIK rewrite in puppet is now the canonical place for this work, i.e. SwiftMedia is no longer
Tue, Dec 7
OK, I know what the problem is (at least at one level). Our swift front-ends use a bit of middleware wmf.rewrite which is shipped by us from puppet; that calls import monotonic. But there's nothing to say that python-monotonic (or python3-monotonic) should be installed from swift front-ends as far as I can see.
Thu, Dec 2
For S3, you need three things - access key, secret key, endpoint.
Tue, Nov 30
Thu, Nov 25
The problem is that /lib/systemd/system/mariadb.service lacks the changes from https://gerrit.wikimedia.org/r/c/operations/software/+/715926
Tue, Nov 23
Account is created; I gather the usual approach is to instruct puppet to write a configuration file with the relevant details in it (taken from profile::thanos::swift::accounts_keys ), similar to how objstore.yaml is written by modules/thanos/manifests/compact.pp or the lookups in modules/profile/manifests/docker_registry_ha/registry.pp.
https://review.opendev.org/c/openstack/swift/+/818881 is this patch sent upstream.
Wed, Nov 17
xfs_repair found a number of problems with the filesystem, and more medium errors were reported by the kernel:
Nov 17 14:24:40 ms-be2059 kernel: [21720811.039143] sd 0:2:17:0: [sdr] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Nov 17 14:24:40 ms-be2059 kernel: [21720811.039147] sd 0:2:17:0: [sdr] tag#2 Sense Key : Medium Error [current] Nov 17 14:24:40 ms-be2059 kernel: [21720811.039150] sd 0:2:17:0: [sdr] tag#2 Add. Sense: No additional sense information Nov 17 14:24:40 ms-be2059 kernel: [21720811.039153] sd 0:2:17:0: [sdr] tag#2 CDB: Read(16) 88 00 00 00 00 00 0b 57 42 f0 00 00 00 18 00 00 Nov 17 14:24:40 ms-be2059 kernel: [21720811.039155] blk_update_request: I/O error, dev sdr, sector 190268144 Nov 17 14:49:37 ms-be2059 kernel: [21722304.380409] sd 0:2:17:0: [sdr] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Nov 17 14:49:37 ms-be2059 kernel: [21722304.380413] sd 0:2:17:0: [sdr] tag#3 Sense Key : Medium Error [current] Nov 17 14:49:37 ms-be2059 kernel: [21722304.380415] sd 0:2:17:0: [sdr] tag#3 Add. Sense: No additional sense information Nov 17 14:49:37 ms-be2059 kernel: [21722304.380418] sd 0:2:17:0: [sdr] tag#3 CDB: Read(16) 88 00 00 00 00 00 0b 57 42 f8 00 00 00 08 00 00 Nov 17 14:49:37 ms-be2059 kernel: [21722304.380420] blk_update_request: I/O error, dev sdr, sector 190268152 Nov 17 15:36:21 ms-be2059 kernel: [21725102.144703] sd 0:2:17:0: [sdr] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Nov 17 15:36:21 ms-be2059 kernel: [21725102.144717] sd 0:2:17:0: [sdr] tag#0 Sense Key : Medium Error [current] Nov 17 15:36:21 ms-be2059 kernel: [21725102.144720] sd 0:2:17:0: [sdr] tag#0 Add. Sense: No additional sense information Nov 17 15:36:21 ms-be2059 kernel: [21725102.144724] sd 0:2:17:0: [sdr] tag#0 CDB: Read(16) 88 00 00 00 00 00 0b 57 43 00 00 00 00 08 00 00 Nov 17 15:36:21 ms-be2059 kernel: [21725102.144727] blk_update_request: I/O error, dev sdr, sector 190268160
I hope this is sufficient to get this drive replaced?
@Papaul I'm trying to xfs_repair the filesystem, which is a lengthy process, but I'm seeing medium errors in the kernel log again:
Nov 17 13:43:29 ms-be2059 kernel: [21718345.521149] sd 0:2:17:0: [sdr] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Nov 17 13:43:29 ms-be2059 kernel: [21718345.521163] sd 0:2:17:0: [sdr] tag#1 Sense Key : Medium Error [current] Nov 17 13:43:29 ms-be2059 kernel: [21718345.521166] sd 0:2:17:0: [sdr] tag#1 Add. Sense: No additional sense information Nov 17 13:43:29 ms-be2059 kernel: [21718345.521171] sd 0:2:17:0: [sdr] tag#1 CDB: Read(16) 88 00 00 00 00 00 0b 57 42 f8 00 00 00 08 00 00 Nov 17 13:43:29 ms-be2059 kernel: [21718345.521175] blk_update_request: I/O error, dev sdr, sector 190268152
That's the same sector as previously, which really does make me think there's a hardware fault here. Is that not enough to convince Dell?
[I'll update when xfs_repair completes]
I don't think so, no - the frontends will not route requests to down servers (at least in theory!); we'll be more vulnerable to failures elsewhere, I think we have to live with that.
Thu, Nov 11
Here's the SupportAssistCollection output.
Wed, Nov 10
Nov 8 2021
Hi,
It's true that the main Swift cluster doesn't support S3, but the Thanos cluster does, and that has enough capacity at least for your proof-of-concept needs.
Nov 3 2021
Sorry for the delay in getting back to you. I have a couple of questions about your request, if I may:
- Are you OK with using the S3 protocol (rather than the Swift protocol)?
- Do you have an idea of how much performance/bandwidth you need (read and write)?
- What sort of timescale do you need this storage on?
Oct 26 2021
Oct 25 2021
@Papaul thanks :)
[subscribing so I get a ping once we know if there's an available spare or not]
Oct 21 2021
Stick the patch here just in case...
diff --git a/swift/cli/recon.py b/swift/cli/recon.py index cd0952875..304a75a90 100644 --- a/swift/cli/recon.py +++ b/swift/cli/recon.py @@ -895,6 +895,7 @@ class SwiftRecon(object): percents = {} top_percents = [(None, 0)] * top low_percents = [(None, 100)] * lowest + hosts_checked = [] recon = Scout("diskusage", self.verbose, self.suppress_errors, self.timeout) print("[%s] Checking disk usage now" % self._ptime()) @@ -902,6 +903,10 @@ class SwiftRecon(object): recon.scout, hosts): if status == 200: hostusage = [] + host = urlparse(url).netloc.split(':')[0] + if host in hosts_checked: + continue + hosts_checked.append(host) for entry in response: if not isinstance(entry['mounted'], bool): print("-> %s/%s: Error: %s" % (url, entry['device'],
Oct 20 2021
Yes, it actually fired on 2021-10-09 (and resolved when I restarted the offending unit) -
<jinxer-wm> (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1119:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org <jinxer-wm> (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db1119:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated -https://alerts.wikimedia.org
This was in #wikimedia-data-persistence channel.
Oct 18 2021
Full weight restored, so closing this (again ;-) )
Oct 12 2021
I think we have an outline of how to make this work, so I'll take ownership of this item.
Oct 11 2021
@Papaul system was stable over the weekend, so I'll take this ticket and start restoring this system to the Swift rings. Thanks!
In case it's useful for future comparison, this is what lsof says about the exporter process now (ie during normal behviour)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME prometheu 8593 prometheus cwd DIR 8,1 4096 2 / prometheu 8593 prometheus rtd DIR 8,1 4096 2 / prometheu 8593 prometheus txt REG 8,1 9681936 1982817 /usr/bin/prometheus-mysqld-exporter prometheu 8593 prometheus mem REG 8,1 1824496 1971450 /usr/lib/x86_64-linux-gnu/libc-2.28.so prometheu 8593 prometheus mem REG 8,1 146968 1971464 /usr/lib/x86_64-linux-gnu/libpthread-2.28.so prometheu 8593 prometheus mem REG 8,1 165632 1971446 /usr/lib/x86_64-linux-gnu/ld-2.28.so prometheu 8593 prometheus 0r CHR 1,3 0t0 6 /dev/null prometheu 8593 prometheus 1u unix 0xffff968af4e47400 0t0 719869330 type=STREAM prometheu 8593 prometheus 2u unix 0xffff968af4e47400 0t0 719869330 type=STREAM prometheu 8593 prometheus 3u IPv6 719782869 0t0 TCP *:9104 (LISTEN) prometheu 8593 prometheus 4u a_inode 0,13 0 10390 [eventpoll] prometheu 8593 prometheus 5u IPv6 719811990 0t0 TCP db1119.eqiad.wmnet:9104->prometheus1003.eqiad.wmnet:36304 (ESTABLISHED) prometheu 8593 prometheus 6u IPv6 719801884 0t0 TCP db1119.eqiad.wmnet:9104->prometheus1004.eqiad.wmnet:52084 (ESTABLISHED)
journalctl output is below - you can see the restart I did on 9 October. I presume the failure on 5 April is the "pme started before mysqld" error that we resolved with the work on T289488 ; so this does look like a different issue. At least we know the alerting works ;-)
-- Logs begin at Mon 2021-04-05 11:12:08 UTC, end at Mon 2021-10-11 10:09:09 UTC. -- Apr 05 11:12:15 db1119 systemd[1]: Started Prometheus exporter for MySQL server. Apr 05 11:12:15 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:15Z" level=info msg="Starting mysqld_exporter (version=0.11.0+ds, branch=debian/sid, revision=0.11.0+ds-1+b20)" source="mysqld_exporter.go:206" Apr 05 11:12:15 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:15Z" level=info msg="Build context (go=go1.11.5, user=pkg-go-maintainers@lists.alioth.debian.org, date=20190311-02:06:43)" source="mysqld_exporter.go:207" Apr 05 11:12:15 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:15Z" level=info msg="Enabled scrapers:" source="mysqld_exporter.go:218" Apr 05 11:12:15 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:15Z" level=info msg=" --collect.global_status" source="mysqld_exporter.go:222" Apr 05 11:12:15 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:15Z" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:222" Apr 05 11:12:15 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:15Z" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:222" Apr 05 11:12:15 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:15Z" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:222" Apr 05 11:12:15 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:15Z" level=info msg="Listening on :9104" source="mysqld_exporter.go:232" Apr 05 11:12:36 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:36Z" level=error msg="Error pinging mysqld: dial unix /run/mysqld/mysqld.sock: connect: no such file or directory" source="exporter.go:119" Apr 05 11:12:36 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:12:36Z" level=error msg="Error pinging mysqld: dial unix /run/mysqld/mysqld.sock: connect: no such file or directory" source="exporter.go:119" Apr 05 11:13:36 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:13:36Z" level=error msg="Error pinging mysqld: dial unix /run/mysqld/mysqld.sock: connect: no such file or directory" source="exporter.go:119" Apr 05 11:13:36 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:13:36Z" level=error msg="Error pinging mysqld: dial unix /run/mysqld/mysqld.sock: connect: no such file or directory" source="exporter.go:119" Apr 05 11:14:36 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:14:36Z" level=error msg="Error pinging mysqld: dial unix /run/mysqld/mysqld.sock: connect: no such file or directory" source="exporter.go:119" Apr 05 11:14:36 db1119 prometheus-mysqld-exporter[692]: time="2021-04-05T11:14:36Z" level=error msg="Error pinging mysqld: dial unix /run/mysqld/mysqld.sock: connect: no such file or directory" source="exporter.go:119" Apr 05 11:17:15 db1119 systemd[1]: Stopping Prometheus exporter for MySQL server... Apr 05 11:17:15 db1119 systemd[1]: prometheus-mysqld-exporter.service: Main process exited, code=killed, status=15/TERM Apr 05 11:17:15 db1119 systemd[1]: prometheus-mysqld-exporter.service: Succeeded. Apr 05 11:17:15 db1119 systemd[1]: Stopped Prometheus exporter for MySQL server. Apr 05 11:17:15 db1119 systemd[1]: Started Prometheus exporter for MySQL server. Apr 05 11:17:15 db1119 prometheus-mysqld-exporter[3610]: time="2021-04-05T11:17:15Z" level=info msg="Starting mysqld_exporter (version=0.11.0+ds, branch=debian/sid, revision=0.11.0+ds-1+b20)" source="mysqld_exporter.go:206" Apr 05 11:17:15 db1119 prometheus-mysqld-exporter[3610]: time="2021-04-05T11:17:15Z" level=info msg="Build context (go=go1.11.5, user=pkg-go-maintainers@lists.alioth.debian.org, date=20190311-02:06:43)" source="mysqld_exporter.go:207" Apr 05 11:17:15 db1119 prometheus-mysqld-exporter[3610]: time="2021-04-05T11:17:15Z" level=info msg="Enabled scrapers:" source="mysqld_exporter.go:218" Apr 05 11:17:15 db1119 prometheus-mysqld-exporter[3610]: time="2021-04-05T11:17:15Z" level=info msg=" --collect.global_status" source="mysqld_exporter.go:222" Apr 05 11:17:15 db1119 prometheus-mysqld-exporter[3610]: time="2021-04-05T11:17:15Z" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:222" Apr 05 11:17:15 db1119 prometheus-mysqld-exporter[3610]: time="2021-04-05T11:17:15Z" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:222" Apr 05 11:17:15 db1119 prometheus-mysqld-exporter[3610]: time="2021-04-05T11:17:15Z" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:222" Apr 05 11:17:15 db1119 prometheus-mysqld-exporter[3610]: time="2021-04-05T11:17:15Z" level=info msg="Listening on :9104" source="mysqld_exporter.go:232" Oct 01 05:22:00 db1119 systemd[1]: Stopping Prometheus exporter for MySQL server... Oct 01 05:22:00 db1119 systemd[1]: prometheus-mysqld-exporter.service: Main process exited, code=killed, status=15/TERM Oct 01 05:22:00 db1119 systemd[1]: prometheus-mysqld-exporter.service: Succeeded. Oct 01 05:22:00 db1119 systemd[1]: Stopped Prometheus exporter for MySQL server. Oct 01 05:23:59 db1119 systemd[1]: Started Prometheus exporter for MySQL server. Oct 01 05:23:59 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-01T05:23:59Z" level=info msg="Starting mysqld_exporter (version=0.11.0+ds, branch=debian/sid, revision=0.11.0+ds-1+b20)" source="mysqld_exporter.go:206" Oct 01 05:23:59 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-01T05:23:59Z" level=info msg="Build context (go=go1.11.5, user=pkg-go-maintainers@lists.alioth.debian.org, date=20190311-02:06:43)" source="mysqld_exporter.go:207" Oct 01 05:23:59 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-01T05:23:59Z" level=info msg="Enabled scrapers:" source="mysqld_exporter.go:218" Oct 01 05:23:59 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-01T05:23:59Z" level=info msg=" --collect.global_status" source="mysqld_exporter.go:222" Oct 01 05:23:59 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-01T05:23:59Z" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:222" Oct 01 05:23:59 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-01T05:23:59Z" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:222" Oct 01 05:23:59 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-01T05:23:59Z" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:222" Oct 01 05:23:59 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-01T05:23:59Z" level=info msg="Listening on :9104" source="mysqld_exporter.go:232" Oct 09 09:12:02 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-09T09:12:02Z" level=error msg="Error pinging mysqld: Error 1226: User 'prometheus' has exceeded the 'max_user_connections' resource (current value: 5)" source="exporter.go:119" Oct 09 09:12:02 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-09T09:12:02Z" level=error msg="Error pinging mysqld: Error 1226: User 'prometheus' has exceeded the 'max_user_connections' resource (current value: 5)" source="exporter.go:119" Oct 09 09:12:02 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-09T09:12:02Z" level=error msg="Error pinging mysqld: Error 1226: User 'prometheus' has exceeded the 'max_user_connections' resource (current value: 5)" source="exporter.go:119" Oct 09 09:12:02 db1119 prometheus-mysqld-exporter[25582]: time="2021-10-09T09:12:02Z" level=error msg="Error pinging mysqld: Error 1226: User 'prometheus' has exceeded the 'max_user_connections' resource (current value: 5)" source="exporter.go:119" Oct 09 10:11:43 db1119 systemd[1]: Stopping Prometheus exporter for MySQL server... Oct 09 10:11:43 db1119 systemd[1]: prometheus-mysqld-exporter.service: Main process exited, code=killed, status=15/TERM Oct 09 10:11:43 db1119 systemd[1]: prometheus-mysqld-exporter.service: Succeeded. Oct 09 10:11:43 db1119 systemd[1]: Stopped Prometheus exporter for MySQL server. Oct 09 10:11:43 db1119 systemd[1]: Started Prometheus exporter for MySQL server. Oct 09 10:11:43 db1119 prometheus-mysqld-exporter[8593]: time="2021-10-09T10:11:43Z" level=info msg="Starting mysqld_exporter (version=0.11.0+ds, branch=debian/sid, revision=0.11.0+ds-1+b20)" source="mysqld_exporter.go:206" Oct 09 10:11:43 db1119 prometheus-mysqld-exporter[8593]: time="2021-10-09T10:11:43Z" level=info msg="Build context (go=go1.11.5, user=pkg-go-maintainers@lists.alioth.debian.org, date=20190311-02:06:43)" source="mysqld_exporter.go:207" Oct 09 10:11:43 db1119 prometheus-mysqld-exporter[8593]: time="2021-10-09T10:11:43Z" level=info msg="Enabled scrapers:" source="mysqld_exporter.go:218" Oct 09 10:11:43 db1119 prometheus-mysqld-exporter[8593]: time="2021-10-09T10:11:43Z" level=info msg=" --collect.global_status" source="mysqld_exporter.go:222" Oct 09 10:11:43 db1119 prometheus-mysqld-exporter[8593]: time="2021-10-09T10:11:43Z" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:222" Oct 09 10:11:43 db1119 prometheus-mysqld-exporter[8593]: time="2021-10-09T10:11:43Z" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:222" Oct 09 10:11:43 db1119 prometheus-mysqld-exporter[8593]: time="2021-10-09T10:11:43Z" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:222" Oct 09 10:11:43 db1119 prometheus-mysqld-exporter[8593]: time="2021-10-09T10:11:43Z" level=info msg="Listening on :9104" source="mysqld_exporter.go:232"
Oct 7 2021
Hi @Papaul We reimaged this host today to try and bring it back into service. After about half an hour of uptime it dropped off the network, and from the management console it looks like the network hardware has failed?
Oct 5 2021
The two changes just merged (to be auto-deployed) should be necessary and sufficient to resolve this.
Sep 23 2021
AFAICT from https://wikitech.wikimedia.org/wiki/Alertmanager alerts are (now?) meant to go into operations/alerts rather than into puppet directly. So I think we want a data-persistence routing for alerts, and then a suitable alert defined in operations/alerts (I've checked and mysql_exporter_last_scrape_error is in prometheus)
Sep 21 2021
[I was pointed at this task from IRC, I'm new in data persistence team, used to do quite a bit of Ceph at the Sanger]
Sep 15 2021
Marking this as resolved - as we deploy 10.4.21-2 everywhere, the fix will get rolled out.
I've also checked the stop/start/restart behaviour, which is as expected.
Also, that on reboot PME isn't started, but when you start mariadb, it does then get started for you.
Sep 13 2021
Hi @Papaul this system seems to have had a hardware fault(s), and is (just) still within its warranty, could you get the hardware checked out, please? Thanks :)
On reboot, the disks came back, but many of the filesystems are unhappy:
mvernon@ms-be2045:~$ sudo dmesg | grep 'Shutting down filesystem'
[ 18.244602] XFS (sda3): Corruption of in-memory data detected. Shutting down filesystem
[ 18.724649] XFS (sdf1): Corruption of in-memory data detected. Shutting down filesystem
[ 19.448076] XFS (sdg1): Corruption of in-memory data detected. Shutting down filesystem
[ 20.610420] XFS (sdj1): I/O Error Detected. Shutting down filesystem
[ 20.745769] XFS (sdn1): I/O Error Detected. Shutting down filesystem
[ 20.938081] XFS (sdi1): I/O Error Detected. Shutting down filesystem
[ 23.719222] XFS (sdh1): I/O Error Detected. Shutting down filesystem
[ 24.802161] XFS (sde1): I/O Error Detected. Shutting down filesystem
[ 30.091057] XFS (sdm1): I/O Error Detected. Shutting down filesystem
[ 31.761276] XFS (sdc1): I/O Error Detected. Shutting down filesystem
Sep 7 2021
This host is ready for DC-Ops to decommission
This host is ready for DC-Ops to decommission
This host is ready for DC-Ops to decommission
This host is ready for DC-Ops to decommission
This host is ready for DC-Ops to decommission
This host is ready for DC-Ops to decommission
This host is ready for DC-Ops to decommission
Sep 6 2021
I think this could be achieved by setting a grafana alert on the MySQL Aggregated dashboard? But I don't really know much about how alerts are set up at WMF, or when one should use a Grafana alerts vs an icinga one, or...
I think we concluded that the mariadb.target idea isn't all that useful (since mostly folk don't stop and start >1 instance at once I think @Kormat said).
Aug 24 2021
Aug 23 2021
I've split the systemd bits into a separate task - T289488
- ensure_packages('prometheus-mysqld-exporter', {'notify' => "Exec['systemctl try-restart prometheus-mysqld-exporter']"})
+ ensure_packages('prometheus-mysqld-exporter', {'notify' => Exec['systemctl try-restart prometheus-mysqld-exporter']})Aug 17 2021
Aug 9 2021
Another thought here - if we want the exporter to be automatically restarted if mysqld is restarted, then we should be able to get systemd to do this for us.