Page MenuHomePhabricator
Create Task
Maniphest T126757

test prometheus mysqld-exporter
Closed, ResolvedPublic
Actions

Description

I've played with https://github.com/prometheus/mysqld_exporter in labs to monitor some labsdb boxes, e.g. https://prometheus.wmflabs.org/grafana/dashboard/db/mysql-labsdb-cluster reads from https://prometheus.wmflabs.org prometheus server which asks several mysqld_exporter endpoints, one per labsdb box.

tracking here what's missing:

  • package mysqld_exporter into prometheus-mysqld-exporter package
  • add puppetization for mysqld_exporter
  • pick a sample of db hosts to monitor
  • read only mysql user access to sample db hosts
  • enable all relevant flags from https://github.com/prometheus/mysqld_exporter#collector-flags to configuration (Note: some flags will have to be handled separately due to privacy concerns)
  • add monitored db hosts to prometheus config

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
dbproxy: add prometheus node monitoringoperations/puppetproduction+1 -0
prometheus: Remove db1075 from the s3 slaves; it was duplicatedoperations/puppetproduction+0 -1
prometheus mysqld exporter: add all pending database instancesoperations/puppetproduction+95 -0
prometheus mysqld exporter: add a bunch of selected slaves from coreoperations/puppetproduction+63 -0
es2001-4: add node exporter to this standalones hostsoperations/puppetproduction+2 -1
prometheus mysqld exporter: disable labsdb1005 because "precise"operations/puppetproduction+5 -5
prometheus mysqld exporter: Add dbstore-eqiad hostsoperations/puppetproduction+6 -0
prometheus: add misc eqiad hosts to mysqld exporteroperations/puppetproduction+54 -0
prometheus exporter: avoid still existing precise hostsoperations/puppetproduction+6 -2
prometheus: generate mysql targets from mw configoperations/puppetproduction+107 -1
prometheus: add mysql mediawiki production db discoveryoperations/puppetproduction+33 -1
prometheus: Add parsercaches on eqiad (and fix the ones on codfw)operations/puppetproduction+19 -1
prometheus: add labsdb eqiad hosts to monitoringoperations/puppetproduction+23 -0
Labsdb: include labs salt groups and prometheus monitoring for dbsoperations/puppetproduction+18 -0
prometheus: Test mysqld-exporter on s6 slaves to check load impactoperations/puppetproduction+13 -0
mariadb: install node/mysql exporters in eqiad toooperations/puppetproduction+2 -5
Fix puppet issues generating empty files for mysql configurationoperations/puppetproduction+5 -20
Puppetize static configuration for prometheus-mysqld-exporteroperations/puppetproduction+244 -6
Quote ARGS parameters for trusty compatibilityoperations/puppetproduction+10 -20
Fix mysqld exporter prometheus config not working in trustyoperations/puppetproduction+11 -3
Configuration changes regarding mysql exporter for prometheusoperations/puppetproduction+11 -0
mariadb: collect prometheus stats in codfwoperations/puppetproduction+1 -1
mariadb: add mysql/node prometheus metrics for db2034operations/puppetproduction+1 -1
prometheus: add mysql job configurationoperations/puppetproduction+40 -1
Add prometheus's mysql-exporter to all jessie's production codfw dbsoperations/puppetproduction+120 -21
Add /var/lib/prometheus to prometheus clientsoperations/puppetproduction+11 -1
Add prometheus mysql exporter to db2069operations/puppetproduction+10 -1
prometheus: add mysqld exporteroperations/puppetproduction+76 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jcrespo mentioned this in rOPUPef9a4da0bdaa: Add prometheus's mysql-exporter to all jessie's production databases.Aug 3 2016, 11:12 AM
jcrespo mentioned this in rOPUPfa096d258f4f: Add prometheus's mysql-exporter to all jessie's core databases.
jcrespo mentioned this in rOPUPfd8f714e8060: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 3 2016, 2:05 PM
jcrespo mentioned this in rOPUP8863dbc65803: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 3 2016, 2:26 PM
jcrespo mentioned this in rOPUP217a65ece497: Add prometheus's mysql-exporter to all jessie's production dbs.
jcrespo mentioned this in rOPUPe9105b5195b1: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 3 2016, 2:58 PM
jcrespo mentioned this in rOPUPaf675f169ae2: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 3 2016, 3:03 PM
jcrespo mentioned this in rOPUP0bda42d97e27: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 3 2016, 4:37 PM
jcrespo mentioned this in rOPUPa4aefea8f90e: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 3 2016, 4:42 PM
jcrespo mentioned this in rOPUPccc415e40e13: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 4 2016, 9:04 AM
jcrespo mentioned this in rOPUP5d6da6f4a6de: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 4 2016, 9:21 AM
jcrespo mentioned this in rOPUP058a8874a53f: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 4 2016, 9:26 AM
jcrespo mentioned this in rOPUP16e7e9844e20: Add prometheus's mysql-exporter to all jessie's production dbs.
jcrespo mentioned this in rOPUPf104127cfa83: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 4 2016, 9:37 AM
jcrespo mentioned this in rOPUPfbdb9bd59004: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 4 2016, 9:47 AM
jcrespo mentioned this in rOPUP01883938dda8: Add prometheus's mysql-exporter to all jessie's production dbs.Aug 4 2016, 9:52 AM
gerritbot added a comment.Aug 4 2016, 10:00 AM

Change 302680 merged by Jcrespo:
Add prometheus's mysql-exporter to all jessie's production codfw dbs

https://gerrit.wikimedia.org/r/302680

jcrespo mentioned this in rOPUPf9772ded5a99: Add prometheus's mysql-exporter to all jessie's production codfw dbs.Aug 4 2016, 10:03 AM
jcrespo mentioned this in rOPUP0d6955a09d0a: Add prometheus's mysql-exporter to all jessie's production codfw dbs.
gerritbot added a comment.Aug 4 2016, 11:13 AM

Change 302904 had a related patch set uploaded (by Filippo Giunchedi):
prometheus: add mysql job configuration

https://gerrit.wikimedia.org/r/302904

fgiunchedi mentioned this in rOPUPb65fd39a9d7b: prometheus: add mysql job configuration.Aug 4 2016, 11:19 AM
fgiunchedi mentioned this in rOPUP618f967a30d6: prometheus: add mysql job configuration.Aug 4 2016, 11:25 AM
fgiunchedi mentioned this in rOPUP9055c891b900: prometheus: add mysql job configuration.
fgiunchedi mentioned this in rOPUP3931a1905577: prometheus: add mysql job configuration.
gerritbot added a comment.Aug 4 2016, 11:27 AM

Change 302904 merged by Filippo Giunchedi:
prometheus: add mysql job configuration

https://gerrit.wikimedia.org/r/302904

jcrespo mentioned this in T119619: Decide storage backend for performance schema monitoring stats.Aug 6 2016, 9:03 AM
gerritbot added a comment.Aug 22 2016, 9:54 AM

Change 305972 had a related patch set uploaded (by Filippo Giunchedi):
mariadb: add mysql/node prometheus metrics for db2034

https://gerrit.wikimedia.org/r/305972

fgiunchedi mentioned this in rOPUP51d042718125: mariadb: add mysql/node prometheus metrics for db2034.Aug 22 2016, 10:00 AM
gerritbot added a comment.Aug 22 2016, 10:05 AM

Change 305972 merged by Filippo Giunchedi:
mariadb: add mysql/node prometheus metrics for db2034

https://gerrit.wikimedia.org/r/305972

jcrespo mentioned this in rOPUP5c07786f250b: mariadb: add mysql/node prometheus metrics for db2034.Aug 22 2016, 10:10 AM
gerritbot added a comment.Aug 22 2016, 10:19 AM

Change 305976 had a related patch set uploaded (by Filippo Giunchedi):
mariadb: collect prometheus stats in codfw

https://gerrit.wikimedia.org/r/305976

fgiunchedi mentioned this in rOPUP6d0260c36140: mariadb: collect prometheus stats in codfw.Aug 22 2016, 10:24 AM
gerritbot added a comment.Aug 22 2016, 10:27 AM

Change 305976 merged by Filippo Giunchedi:
mariadb: collect prometheus stats in codfw

https://gerrit.wikimedia.org/r/305976

fgiunchedi mentioned this in rOPUP184a777e7617: mariadb: collect prometheus stats in codfw.Aug 22 2016, 10:29 AM
gerritbot added a comment.Aug 23 2016, 10:40 AM

Change 306174 had a related patch set uploaded (by Filippo Giunchedi):
mariadb: install node/mysql exporters in eqiad too

https://gerrit.wikimedia.org/r/306174

fgiunchedi mentioned this in rOPUP3d16a2f35e46: mariadb: install node/mysql exporters in eqiad too.Aug 23 2016, 10:43 AM
jcrespo added a comment.Aug 23 2016, 11:31 AM

So, some impressions I got after using grafana as a frontend for prometheus:

  • We are missing key metrics: SHOW ENGINE INNODB STATUS, SHOW PROCESSLIST and latency options. Latency is more complex issue, I will talk about it separatelly.

    "-collect.global_status -collect.global_variables -collect.info_schema.innodb_metrics -collect.info_schema.processlist -collect.info_schema.processlist.min_time X -collect.slave_status" I do not know that X should be (0?). innodb_metrics and processlist should solve the above problem

    Let's test this before adding the deployment to eqiad for performance implications. Once we have that, we will have covered all current tendril graphing options.
  • I do not think we will have proper support for multi-source replication. This is not a blocker for core or for the goal, but we definitely need support for that. We should try dbstore2* hosts to see what happens
  • pt-heartbeat: we need support for it- currently, mediawiki uses this rather than Seconds_behind_master(!). It is more reliable (if replication is not running, SHOW SLAVE STATUS returns null) and easier to handle with multi-source. Also, less prone to getting blocked.
  • table statistics: currently, we gather table statistics for all available tables; as prometheus mysql user only has access to mysql and heartbeat tables, it only provides those. This is good, because runing those on s3 hosts will be almost impossible (dozens of thousands of tables), bad because it doesn't run on the interesting tables. These metrics are interesting, but maybe they should run on a single server, or on all servers, every 1 day or so. They are very static, but we do not care about the number of rows every minute.
  • latency metrics: this is the most problematic part- there is some performance_Schema support, but we talked about not exposing queries publicly, as they could contain private data. There are easier alternatives as extra metrics, but they require plugins installed on mysql that (in most cases) are not very taxing, but the whole idea of performance_schema is not to have those plugins. I do not know how to go with this, but we definitely need an "average latency" and a "latency of queries by group" measurement.
jcrespo added a comment.Aug 23 2016, 11:33 AM

I have created a sample dashboard (per server), similar to that of tendril:
https://grafana-admin.wikimedia.org/dashboard/db/mysql?var-server=db2019

Performance seems to be very good.

I will create next a dashboard with aggregated values to push it to the limits.

gerritbot added a comment.Aug 24 2016, 9:38 AM

Change 306404 had a related patch set uploaded (by Jcrespo):
Configuration changes regarding firewall and mysql for prometheus

https://gerrit.wikimedia.org/r/306404

jcrespo mentioned this in rOPUP3e3504c8c0b3: Configuration changes regarding firewall and mysql for prometheus.Aug 24 2016, 9:44 AM
jcrespo mentioned this in rOPUP439767c438a8: Configuration changes regarding firewall and mysql for prometheus.Aug 24 2016, 9:49 AM
jcrespo mentioned this in rOPUPed536dc930de: Configuration changes regarding firewall and mysql for prometheus.Aug 24 2016, 11:20 AM
jcrespo mentioned this in rOPUP17a82ac518a1: Configuration changes regarding mysql exporter for prometheus.Aug 24 2016, 11:25 AM
jcrespo mentioned this in rOPUP20b098611a43: Configuration changes regarding mysql exporter for prometheus.
gerritbot added a comment.Aug 24 2016, 4:05 PM

Change 306404 merged by Jcrespo:
Configuration changes regarding mysql exporter for prometheus

https://gerrit.wikimedia.org/r/306404

jcrespo mentioned this in rOPUPb36263d6ca01: Configuration changes regarding mysql exporter for prometheus.Aug 24 2016, 4:06 PM
gerritbot added a comment.Aug 24 2016, 5:24 PM

Change 306470 had a related patch set uploaded (by Jcrespo):
Fix mysqld exporter prometheus config not working in trusty

https://gerrit.wikimedia.org/r/306470

gerritbot added a comment.Aug 24 2016, 5:30 PM

Change 306470 merged by Jcrespo:
Fix mysqld exporter prometheus config not working in trusty

https://gerrit.wikimedia.org/r/306470

jcrespo mentioned this in rOPUP39bffc9c2c4f: Fix mysqld exporter prometheus config not working in trusty.Aug 24 2016, 5:31 PM
jcrespo mentioned this in rOPUP6156996f4e94: Fix mysqld exporter prometheus config not working in trusty.
jcrespo mentioned this in rOPUP988e56beceb2: Fix mysqld exporter prometheus config not working in trusty.
jcrespo added a comment.Aug 24 2016, 5:38 PM

So the changes on the defaults do not work on trusty, I will revert that to:
"ARGS="

I cannot find there log of why it fails, so I cannot debug, but probably the syntaxs is different.
The above patch at least will keep it working until a more permanent solution is figured out.

jcrespo added a comment.Aug 24 2016, 5:47 PM

BTW, interesting behavior seeing graphs with gaps: https://grafana.wikimedia.org/dashboard/db/mysql?from=1472055221970&to=1472060298229&var-server=db2029

jcrespo added a comment.Aug 25 2016, 8:02 AM

Another thing I cannot have easily in grafana is the ability to create dashboards; that together with the fact that handling text is not very friendly, may require some alternative solutions for a dashboard. The idea is having a table for quick reference in case of ongoing issues with:

hostname, ip, mysql version, memory, uptime, QPS, average latency, replication state (running/stopped) and replication lag

On click, it would lead me to the detailed view (https://grafana-admin.wikimedia.org/dashboard/db/mysql?var-server=$hostname)

gerritbot added a comment.Aug 25 2016, 10:10 AM

Change 306645 had a related patch set uploaded (by Jcrespo):
Quote ARGS parameters for trusty compatibility

https://gerrit.wikimedia.org/r/306645

jcrespo mentioned this in rOPUP8ae820dd0c41: Quote ARGS parameters for trusty compatibility.Aug 25 2016, 10:17 AM
gerritbot added a comment.Aug 25 2016, 10:59 AM

Change 306645 merged by Jcrespo:
Quote ARGS parameters for trusty compatibility

https://gerrit.wikimedia.org/r/306645

fgiunchedi mentioned this in T143896: MySQL metrics monitoring.Aug 25 2016, 2:15 PM
gerritbot added a comment.Aug 25 2016, 3:17 PM

Change 306675 had a related patch set uploaded (by Jcrespo):
Puppetize static configuration for prometheus-mysqld-exporter

https://gerrit.wikimedia.org/r/306675

jcrespo mentioned this in rOPUPe7af88cecccd: Puppetize static configuration for prometheus-mysqld-exporter.Aug 25 2016, 3:21 PM
jcrespo mentioned this in rOPUP990623e8a23f: Puppetize static configuration for prometheus-mysqld-exporter.
jcrespo mentioned this in rOPUPe0f51a583d75: Puppetize static configuration for prometheus-mysqld-exporter.Aug 25 2016, 3:27 PM
jcrespo mentioned this in rOPUP5f1573258e13: Puppetize static configuration for prometheus-mysqld-exporter.Aug 25 2016, 4:57 PM
gerritbot added a comment.Aug 26 2016, 10:08 AM

Change 306675 merged by Jcrespo:
Puppetize static configuration for prometheus-mysqld-exporter

https://gerrit.wikimedia.org/r/306675

gerritbot added a comment.Aug 26 2016, 11:31 AM

Change 306906 had a related patch set uploaded (by Jcrespo):
Fix puppet issues generating empty files for mysql configuration

https://gerrit.wikimedia.org/r/306906

gerritbot added a comment.Aug 26 2016, 11:33 AM

Change 306906 merged by Jcrespo:
Fix puppet issues generating empty files for mysql configuration

https://gerrit.wikimedia.org/r/306906

gerritbot added a comment.Aug 26 2016, 1:17 PM

Change 306174 merged by Filippo Giunchedi:
mariadb: install node/mysql exporters in eqiad too

https://gerrit.wikimedia.org/r/306174

gerritbot added a comment.Aug 26 2016, 2:18 PM

Change 306928 had a related patch set uploaded (by Jcrespo):
prometheus: Test mysqld-exporter on s6 slaves to check load impact

https://gerrit.wikimedia.org/r/306928

gerritbot added a comment.Aug 26 2016, 2:20 PM

Change 306928 merged by Jcrespo:
prometheus: Test mysqld-exporter on s6 slaves to check load impact

https://gerrit.wikimedia.org/r/306928

gerritbot added a comment.Aug 26 2016, 3:09 PM

Change 306936 had a related patch set uploaded (by Jcrespo):
Labsdb: include labs salt groups and prometheus monitoring for dbs

https://gerrit.wikimedia.org/r/306936

gerritbot added a comment.Aug 26 2016, 3:14 PM

Change 306937 had a related patch set uploaded (by Jcrespo):
dbproxy: add prometheus node monitoring

https://gerrit.wikimedia.org/r/306937

gerritbot added a comment.Aug 26 2016, 3:23 PM

Change 306939 had a related patch set uploaded (by Jcrespo):
es2001-4: add node exporter to this standalones hosts

https://gerrit.wikimedia.org/r/306939

gerritbot added a comment.Aug 29 2016, 8:50 AM

Change 306936 merged by Jcrespo:
Labsdb: include labs salt groups and prometheus monitoring for dbs

https://gerrit.wikimedia.org/r/306936

gerritbot added a comment.Aug 29 2016, 9:46 AM

Change 307249 had a related patch set uploaded (by Jcrespo):
prometheus: add labsdb eqiad hosts to monitoring

https://gerrit.wikimedia.org/r/307249

gerritbot added a comment.Aug 29 2016, 9:48 AM

Change 307250 had a related patch set uploaded (by Jcrespo):
prometheus: Add parsercaches on eqiad (and fix the ones on codfw)

https://gerrit.wikimedia.org/r/307250

gerritbot added a comment.Aug 29 2016, 9:56 AM

Change 307249 merged by Jcrespo:
prometheus: add labsdb eqiad hosts to monitoring

https://gerrit.wikimedia.org/r/307249

gerritbot added a comment.Aug 29 2016, 9:57 AM

Change 307250 merged by Jcrespo:
prometheus: Add parsercaches on eqiad (and fix the ones on codfw)

https://gerrit.wikimedia.org/r/307250

gerritbot added a comment.Aug 29 2016, 10:10 AM

Change 307254 had a related patch set uploaded (by Jcrespo):
prometheus: add misc eqiad hosts to mysqld exporter

https://gerrit.wikimedia.org/r/307254

gerritbot added a comment.Aug 29 2016, 1:05 PM

Change 296596 abandoned by Filippo Giunchedi:
prometheus: add mysql mediawiki production db discovery

Reason:
we'll be doing discovery through puppet data, not mw

https://gerrit.wikimedia.org/r/296596

gerritbot added a comment.Aug 29 2016, 1:05 PM

Change 296595 abandoned by Filippo Giunchedi:
prometheus: generate mysql targets from mw config

Reason:
we'll be doing discovery through puppet data, not mw

https://gerrit.wikimedia.org/r/296595

gerritbot added a comment.Aug 29 2016, 1:12 PM

Change 307285 had a related patch set uploaded (by Jcrespo):
Workaround still existing, but irrelevant, precise hosts

https://gerrit.wikimedia.org/r/307285

gerritbot added a comment.Aug 29 2016, 2:12 PM

Change 307285 merged by Jcrespo:
prometheus exporter: avoid still existing precise hosts

https://gerrit.wikimedia.org/r/307285

gerritbot added a comment.Aug 29 2016, 2:34 PM

Change 307293 had a related patch set uploaded (by Jcrespo):
prometheus mysqld exporter: Add dbstore-eqiad hosts

https://gerrit.wikimedia.org/r/307293

jcrespo added a comment.Aug 29 2016, 2:42 PM

@fgiunchedi So there are things that do not work: many labsdb hosts use a different configuration than production for the location of its socket and other exotic options, like datadir. For me that is a bug, and it makes many things complex, but they wanted it like that, and they told me are they would maintain themselves the labs/db roles (that is why they are separate from the other roles), so talk to them to solve that.

The other thing was a known issue: db1069 has many instances running at the same time. I mentioned this was not a hard requirement for the goal, but we want to support that at some point; as probably we will increase the number of instances like that at some point in the future. The probable way to go is to monitor each instance on a separate port.

jcrespo updated the task description. (Show Details)Aug 29 2016, 2:43 PM
gerritbot added a comment.Aug 29 2016, 2:48 PM

Change 307298 had a related patch set uploaded (by Jcrespo):
prometheus mysqld exporter: disable labsdb1005 because "precise"

https://gerrit.wikimedia.org/r/307298

gerritbot added a comment.Aug 29 2016, 2:52 PM

Change 307254 merged by Jcrespo:
prometheus: add misc eqiad hosts to mysqld exporter

https://gerrit.wikimedia.org/r/307254

gerritbot added a comment.Aug 29 2016, 2:55 PM

Change 307293 merged by Jcrespo:
prometheus mysqld exporter: Add dbstore-eqiad hosts

https://gerrit.wikimedia.org/r/307293

gerritbot added a comment.Aug 29 2016, 2:58 PM

Change 307298 merged by Jcrespo:
prometheus mysqld exporter: disable labsdb1005 because "precise"

https://gerrit.wikimedia.org/r/307298

gerritbot added a comment.Aug 29 2016, 6:16 PM

Change 306939 merged by Jcrespo:
es2001-4: add node exporter to this standalones hosts

https://gerrit.wikimedia.org/r/306939

gerritbot added a comment.Aug 30 2016, 8:24 AM

Change 307479 had a related patch set uploaded (by Jcrespo):
prometheus mysqld exporter: add a bunch of selected slaves from core

https://gerrit.wikimedia.org/r/307479

gerritbot added a comment.Aug 30 2016, 8:31 AM

Change 307479 merged by Jcrespo:
prometheus mysqld exporter: add a bunch of selected slaves from core

https://gerrit.wikimedia.org/r/307479

gerritbot added a comment.Aug 30 2016, 1:06 PM

Change 307503 had a related patch set uploaded (by Jcrespo):
prometheus mysqld exporter: add all pending database instances

https://gerrit.wikimedia.org/r/307503

gerritbot added a comment.Aug 30 2016, 2:36 PM

Change 307503 merged by Jcrespo:
prometheus mysqld exporter: add all pending database instances

https://gerrit.wikimedia.org/r/307503

gerritbot added a comment.Sep 8 2016, 5:52 AM

Change 309241 had a related patch set uploaded (by Jcrespo):
prometheus: Remove db1075 from the s3 slaves; it was duplicated

https://gerrit.wikimedia.org/r/309241

gerritbot added a comment.Sep 8 2016, 7:52 AM

Change 309241 merged by Jcrespo:
prometheus: Remove db1075 from the s3 slaves; it was duplicated

https://gerrit.wikimedia.org/r/309241

fgiunchedi closed this task as Resolved.Sep 13 2016, 9:45 AM

Closing this, followups are tracked at T145072: Generate instance list of active database hosts to be monitored from prometheus and T143896: MySQL metrics monitoring

gerritbot added a comment.May 15 2020, 9:30 AM

Change 306937 abandoned by Jcrespo:
dbproxy: add prometheus node monitoring

Reason:
Already included on standard

https://gerrit.wikimedia.org/r/306937

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits