Page MenuHomePhabricator
Create Task
Maniphest T206454

Setup Kafka cluster, producers and consumers for logging pipeline
Closed, ResolvedPublic
Actions

Description

This task tracks setting up a Kafka cluster dedicated to logging purposes. Given the considerations in T205873: Investigate Kafka main cluster usage for logging pipeline said cluster will initially coexist on Logstash hosts now and on dedicated hardware in the future.

Outline of steps:

  • Puppetization for Kafka brokers, in a dedicated profile
  • Setup TLS certificates for the brokers
  • Deploy Kafka brokers to logstash elasticsearch data hosts
  • Decide on the granularity of topics (one single topic? one per cluster?) and the number of partitions
  • Setup Logstash input for Kafka logging (i.e. consumers) with TLS
  • Setup rsyslog output for Kafka logging (i.e. producers) with TLS and make it opt-in

Details

SubjectRepoBranchLines +/-
rsyslog: expand output lookup table docsoperations/puppetproduction+8 -2
profile: introduce jmx_exporter_port to logstash::collectoroperations/puppetproduction+2 -2
kafka_shipper: update syslog json templateoperations/puppetproduction+4 -2
kafka_shipper: use mmrm1stspace to remove leading space in msg fieldoperations/puppetproduction+7 -0
profile::logstash::collector: set kafka shipper input codec to jsonoperations/puppetproduction+1 -0
logstash::input::kafka: add codec paramoperations/puppetproduction+3 -0
profile::rsyslog::kafka_shipper: use eqiad logging kafka brokersoperations/puppetproduction+4 -0
logstash: set rsyslog-shipper input type to syslogoperations/puppetproduction+1 -0
create rsyslog::ship_logfile - simplified logstash shipper via kafkaoperations/puppetproduction+42 -0
logstash: add rsyslog-shipper kafka input configoperations/puppetproduction+22 -1
logstash: add rsyslog-shipper kafka input configoperations/puppetproduction+20 -0
logstash::input::kafka: add topics_pattern supportoperations/puppetproduction+6 -0
logstash::input::kafka: add topics_pattern supportoperations/puppetproduction+15 -10
logstash::input::kafka add support for SSL/TLS optionsoperations/puppetproduction+24 -11
kafka_shipper: pin librdkafka1 to stretch-backports on stretchoperations/puppetproduction+9 -1
profile::rsyslog::kafka_shipper: generalized log pipeline/elk shipperoperations/puppetproduction+112 -0
role::logstash::collector: migrate to profile::logstash::collectoroperations/puppetproduction+340 -317
prometheus: add Burrow metrics for kafka-loggingoperations/puppetproduction+7 -0
hieradata: change burrow port for kafka loggingoperations/puppetproduction+2 -2
logstash: apply role::kafka::logging to logstash es data nodesoperations/puppetproduction+32 -0
logstash: set logging kafka package version to 1.1.0-1operations/puppetproduction+1 -0
aptrepo: add thirdparty/confluent component for jessieoperations/puppetproduction+1 -1
confluent::kafka::common support thirdparty/confluent on jessieoperations/puppetproduction+1 -1
site: enable logging Kafka on Logstash nodesoperations/puppetproduction+37 -0
confluent::kafka::common: force provider => 'systemd' for servicesoperations/puppetproduction+7 -3
site: enable logging Kafka on Logstash nodesoperations/puppetproduction+37 -0
New Kafka cluster logging-eqiadoperations/puppetproduction+40 -5
logstash: move to /srv/elasticsearchoperations/puppetproduction+7 -7
logstash: add ipv6 to elasticsearchoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fgiunchedi triaged this task as Medium priority.Oct 8 2018, 10:06 AM
fgiunchedi mentioned this in T205873: Investigate Kafka main cluster usage for logging pipeline.
gerritbot subscribed.Oct 8 2018, 1:36 PM

Change 465164 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: add ipv6 to elasticsearch

https://gerrit.wikimedia.org/r/465164

gerritbot added a project: Patch-For-Review.Oct 8 2018, 1:36 PM

Change 465165 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: move to /srv/elasticsearch

https://gerrit.wikimedia.org/r/465165

gerritbot added a comment.Oct 8 2018, 1:36 PM

Change 465166 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] WIP: new Kafka cluster logging-main

https://gerrit.wikimedia.org/r/465166

gerritbot added a comment.Oct 8 2018, 1:36 PM

Change 465167 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] site: enable logging Kafka on Logstash nodes

https://gerrit.wikimedia.org/r/465167

gerritbot added a comment.Oct 10 2018, 8:50 AM

Change 465164 abandoned by Filippo Giunchedi:
logstash: add ipv6 to elasticsearch

Reason:
See latest comment

https://gerrit.wikimedia.org/r/465164

fgiunchedi moved this task from Backlog to In Dev/Progress on the Wikimedia-Logstash board.Oct 15 2018, 2:25 PM
Stashbot subscribed.Oct 16 2018, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-16T13:55:17Z] <godog> depool logstash1007 to change elasticsearch data dir - T206454

gerritbot added a comment.Oct 16 2018, 1:56 PM

Change 465165 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: move to /srv/elasticsearch

https://gerrit.wikimedia.org/r/465165

Stashbot added a comment.Oct 16 2018, 2:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-16T14:06:20Z] <godog> depool in turn logstash1008 and logstash1009 to change elasticsearch data dir - T206454

Stashbot added a comment.Oct 16 2018, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-16T14:28:04Z] <godog> roll-restart elasticsearch on logstash100[456] to change elasticsearch data dir - T206454

gerritbot added a comment.Oct 17 2018, 6:01 PM

Change 465166 merged by Herron:
[operations/puppet@production] New Kafka cluster logging-eqiad

https://gerrit.wikimedia.org/r/465166

Stashbot added a comment.Oct 18 2018, 4:57 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T16:57:48Z] <herron> enabling kafka on logstash elasticsearch cluster T206454

gerritbot added a comment.Oct 18 2018, 4:59 PM

Change 465167 merged by Herron:
[operations/puppet@production] site: enable logging Kafka on Logstash nodes

https://gerrit.wikimedia.org/r/465167

Stashbot added a comment.Oct 18 2018, 5:19 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:19:23Z] <herron> aborted enabling kafka on logstash elasticsearch cluster due to puppet errors. reverted change T206454

herron subscribed.Oct 18 2018, 5:24 PM

Sadly puppet threw these errors after merging https://gerrit.wikimedia.org/r/465167 so reverted with https://gerrit.wikimedia.org/r/468362/

Error: /Stage[main]/Confluent::Kafka::Common/Service[confluent-kafka]/enable: change from false to mask failed: Could not set 'mask' on enable: undefined method `mask' for Service[confluent-kafka](provider=debian):Puppet::Type::Service::ProviderDebian at /etc/puppet/modules/confluent/manifests/kafka/common.pp:80

Error: /Stage[main]/Confluent::Kafka::Common/Service[confluent-kafka-connect]/enable: change from false to mask failed: Could not set 'mask' on enable: undefined method `mask' for Service[confluent-kafka-connect](provider=debian):Puppet::Type::Service::ProviderDebian at /etc/puppet/modules/confluent/manifests/kafka/common.pp:80

Error: /Stage[main]/Confluent::Kafka::Common/Service[confluent-zookeeper]/enable: change from false to mask failed: Could not set 'mask' on enable: undefined method `mask' for Service[confluent-zookeeper](provider=debian):Puppet::Type::Service::ProviderDebian at /etc/puppet/modules/confluent/manifests/kafka/common.pp:80

Wonder if this has anything to do with Jessie vs. Stretch? FWIW the logstash elasticsearch hosts are still running Jessie.

gerritbot added a comment.Oct 19 2018, 1:38 AM

Change 468498 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] confluent::kafka::common: dont use enable => 'mask' on jessie hosts

https://gerrit.wikimedia.org/r/468498

fgiunchedi updated the task description. (Show Details)Oct 22 2018, 10:52 AM
gerritbot added a comment.Oct 22 2018, 1:22 PM

Change 468498 merged by Herron:
[operations/puppet@production] confluent::kafka::common: force provider => 'systemd' for services

https://gerrit.wikimedia.org/r/468498

gerritbot added a comment.Oct 22 2018, 2:02 PM

Change 468983 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] site: enable logging Kafka on Logstash nodes

https://gerrit.wikimedia.org/r/468983

gerritbot added a comment.Oct 22 2018, 2:08 PM

Change 468983 merged by Herron:
[operations/puppet@production] site: enable logging Kafka on Logstash nodes

https://gerrit.wikimedia.org/r/468983

herron added a comment.Oct 22 2018, 4:04 PM

There were three issues observed during todays aborted deploy of logging kafka (https://gerrit.wikimedia.org/r/468498)

  • Need a proper import of confluent-kafka-2.11 1.1.0-1 for Jessie.
  • Need to address ERROR [KafkaServer id=1004] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: SSL key store is specified, but key store password is not specified.
  • Need to modify puppet structure to ensure that kafka logging role is not applied to logstash frontend hosts.

Patch has been reverted pending solutions to these.

gerritbot added a comment.Oct 22 2018, 9:06 PM

Change 469122 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] confluent::kafka::common support thirdparty/confluent on jessie

https://gerrit.wikimedia.org/r/469122

gerritbot added a comment.Oct 22 2018, 9:06 PM

Change 469123 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] aptrepo: add thirdparty/confluent component for jessie

https://gerrit.wikimedia.org/r/469123

gerritbot added a comment.Oct 22 2018, 9:06 PM

Change 469124 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: set logging kafka package version to 1.1.0-1

https://gerrit.wikimedia.org/r/469124

Stashbot added a comment.Oct 23 2018, 2:47 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-23T14:47:50Z] <herron> added confluent-kafka-2.11 1.1.0-1 package to jessie-wikimedia/thirdparty T206454

gerritbot added a comment.Oct 23 2018, 2:50 PM

Change 469122 abandoned by Herron:
confluent::kafka::common support thirdparty/confluent on jessie

Reason:
going with existing thirdparty component and relying on differences in package names (-2.11 vs -2.11.7) for separation between kafka and logstash clusters

https://gerrit.wikimedia.org/r/469122

gerritbot added a comment.Oct 23 2018, 2:51 PM

Change 469123 abandoned by Herron:
aptrepo: add thirdparty/confluent component for jessie

Reason:
ok, will use existing thirdparty component instead

https://gerrit.wikimedia.org/r/469123

gerritbot added a comment.Oct 23 2018, 2:52 PM

Change 469124 abandoned by Herron:
logstash: set logging kafka package version to 1.1.0-1

https://gerrit.wikimedia.org/r/469124

gerritbot added a comment.Oct 23 2018, 6:33 PM

Change 469246 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: create es/kafka combined role and assign to es data nodes

https://gerrit.wikimedia.org/r/469246

herron added a comment.Oct 23 2018, 8:21 PM
In T206454#4686063, @herron wrote:
  • Need a proper import of confluent-kafka-2.11 1.1.0-1 for Jessie.

This is done

  • Need to address ERROR [KafkaServer id=1004] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: SSL key store is specified, but key store password is not specified.

afaict to address this we'll set the password via profile::kafka::broker::ssl_password in the appropriate hieradata yaml on puppetmaster:/srv/private. However the actual location should depend on the outcome of the following item.

  • Need to modify puppet structure to ensure that kafka logging role is not applied to logstash frontend hosts.

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/469246/ is an approach to handle this and looks good with the perspective of pcc, but needs review/discussion. Please see comments in gerrit.

gerritbot added a comment.Oct 24 2018, 2:31 PM

Change 469246 merged by Herron:
[operations/puppet@production] logstash: apply role::kafka::logging to logstash es data nodes

https://gerrit.wikimedia.org/r/469246

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Oct 24 2018, 3:09 PM
herron added a comment.Oct 24 2018, 3:09 PM

Kafka service is now running on the logstash elasticsearch data hosts, and related icinga service checks are green.

herron updated the task description. (Show Details)Oct 24 2018, 3:15 PM
gerritbot added a comment.Oct 25 2018, 1:25 PM

Change 469612 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: change burrow port for kafka logging

https://gerrit.wikimedia.org/r/469612

gerritbot added a comment.Oct 25 2018, 1:25 PM

Change 469613 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add Burrow metrics for kafka-logging

https://gerrit.wikimedia.org/r/469613

gerritbot added a comment.Oct 25 2018, 1:36 PM

Change 469612 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: change burrow port for kafka logging

https://gerrit.wikimedia.org/r/469612

gerritbot added a comment.Oct 25 2018, 1:37 PM

Change 469613 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add Burrow metrics for kafka-logging

https://gerrit.wikimedia.org/r/469613

gerritbot added a comment.Oct 26 2018, 5:35 PM

Change 469945 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] create rsyslog::ship_logfile - simplified logstash shipper via kafka

https://gerrit.wikimedia.org/r/469945

gerritbot added a comment.Oct 29 2018, 7:00 PM

Change 470452 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] role::logstash::collector: migrate to profile::logstash::collector

https://gerrit.wikimedia.org/r/470452

gerritbot added a comment.Oct 29 2018, 7:24 PM

Change 470454 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add generic kafka input config

https://gerrit.wikimedia.org/r/470454

• mobrovac added projects: Platform Team Legacy (Watching / External), Services (watching).Oct 30 2018, 4:45 PM
gerritbot added a comment.Oct 31 2018, 2:32 PM

Change 470452 merged by Herron:
[operations/puppet@production] role::logstash::collector: migrate to profile::logstash::collector

https://gerrit.wikimedia.org/r/470452

gerritbot added a comment.Nov 1 2018, 8:58 PM

Change 471114 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] WIP: profile::rsyslog::logstash_shipper: generalized logstash shipper

https://gerrit.wikimedia.org/r/471114

gerritbot added a comment.Nov 7 2018, 7:31 PM

Change 471114 merged by Herron:
[operations/puppet@production] profile::rsyslog::kafka_shipper: generalized log pipeline/elk shipper

https://gerrit.wikimedia.org/r/471114

herron updated the task description. (Show Details)Nov 9 2018, 6:07 PM
gerritbot added a comment.Nov 9 2018, 6:08 PM

Change 472694 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: pin librdkafka1 to stretch-backports on stretch

https://gerrit.wikimedia.org/r/472694

gerritbot added a comment.Nov 12 2018, 4:32 PM

Change 472694 abandoned by Herron:
kafka_shipper: pin librdkafka1 to stretch-backports on stretch

Reason:
ok, fair enough! I've rebuilt librdkafka from stretch-backports as librdkafka_0.11.6-1~bpo9 1 wikimedia1. Will follow up outside gerrit.

https://gerrit.wikimedia.org/r/472694

gerritbot added a comment.Nov 12 2018, 10:28 PM

Change 473137 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] WIP: logstash::input::kafka add support for SSL/TLS options

https://gerrit.wikimedia.org/r/473137

gerritbot added a comment.Nov 12 2018, 10:28 PM

Change 473138 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] WIP: logstash::input::kafka: add topics_prefix support

https://gerrit.wikimedia.org/r/473138

herron closed subtask T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia as Resolved.Nov 13 2018, 8:11 PM
gerritbot added a comment.Nov 14 2018, 6:05 PM

Change 473137 merged by Herron:
[operations/puppet@production] logstash::input::kafka add support for SSL/TLS options

https://gerrit.wikimedia.org/r/473137

gerritbot added a comment.Nov 14 2018, 7:34 PM

Change 473138 abandoned by Herron:
logstash::input::kafka: add topics_pattern support

https://gerrit.wikimedia.org/r/473138

gerritbot added a comment.Nov 14 2018, 7:35 PM

Change 473588 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash::input::kafka: add topics_pattern support

https://gerrit.wikimedia.org/r/473588

gerritbot added a comment.Nov 14 2018, 7:43 PM

Change 473588 merged by Herron:
[operations/puppet@production] logstash::input::kafka: add topics_pattern support

https://gerrit.wikimedia.org/r/473588

gerritbot added a comment.Nov 14 2018, 8:06 PM

Change 470454 merged by Herron:
[operations/puppet@production] logstash: add rsyslog-shipper kafka input config

https://gerrit.wikimedia.org/r/470454

gerritbot added a comment.Nov 14 2018, 8:28 PM

Change 473607 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add rsyslog-shipper kafka input config

https://gerrit.wikimedia.org/r/473607

gerritbot added a comment.Nov 14 2018, 9:04 PM

Change 473607 merged by Herron:
[operations/puppet@production] logstash: add rsyslog-shipper kafka input config

https://gerrit.wikimedia.org/r/473607

gerritbot added a comment.Nov 15 2018, 2:55 PM

Change 469945 abandoned by Herron:
create rsyslog::ship_logfile - simplified logstash shipper via kafka

Reason:
abandoning in favor of Id751995020c4d505aa917ad13a2f66f5b668078c

https://gerrit.wikimedia.org/r/469945

gerritbot added a comment.Nov 15 2018, 5:43 PM

Change 473800 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: set rsyslog-shipper input type to syslog

https://gerrit.wikimedia.org/r/473800

gerritbot added a comment.Nov 15 2018, 5:48 PM

Change 473800 merged by Herron:
[operations/puppet@production] logstash: set rsyslog-shipper input type to syslog

https://gerrit.wikimedia.org/r/473800

gerritbot added a comment.Nov 16 2018, 2:47 AM

Change 473998 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] profile::rsyslog::kafka_shipper: use eqiad logging kafka brokers

https://gerrit.wikimedia.org/r/473998

gerritbot added a comment.Nov 16 2018, 2:51 AM

Change 473998 merged by Herron:
[operations/puppet@production] profile::rsyslog::kafka_shipper: use eqiad logging kafka brokers

https://gerrit.wikimedia.org/r/473998

gerritbot added a comment.Nov 16 2018, 3:43 AM

Change 474021 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash::input::kafka: add codec param

https://gerrit.wikimedia.org/r/474021

gerritbot added a comment.Nov 16 2018, 3:55 AM

Change 474021 merged by Herron:
[operations/puppet@production] logstash::input::kafka: add codec param

https://gerrit.wikimedia.org/r/474021

gerritbot added a comment.Nov 16 2018, 4:19 AM

Change 474026 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] profile::logstash::collector: set kafka shipper input codec to json

https://gerrit.wikimedia.org/r/474026

gerritbot added a comment.Nov 16 2018, 4:22 AM

Change 474026 merged by Herron:
[operations/puppet@production] profile::logstash::collector: set kafka shipper input codec to json

https://gerrit.wikimedia.org/r/474026

gerritbot added a comment.Nov 16 2018, 7:35 PM

Change 474317 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: use mmrm1stspace to remove leading space in msg field

https://gerrit.wikimedia.org/r/474317

gerritbot added a comment.Nov 16 2018, 7:40 PM

Change 474319 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: update syslog json template

https://gerrit.wikimedia.org/r/474319

herron updated the task description. (Show Details)Nov 16 2018, 7:45 PM
gerritbot added a comment.Nov 17 2018, 1:06 AM

Change 474317 merged by Herron:
[operations/puppet@production] kafka_shipper: use mmrm1stspace to remove leading space in msg field

https://gerrit.wikimedia.org/r/474317

gerritbot added a comment.Nov 17 2018, 1:12 AM

Change 474319 merged by Herron:
[operations/puppet@production] kafka_shipper: update syslog json template

https://gerrit.wikimedia.org/r/474319

gerritbot added a comment.Nov 19 2018, 1:01 PM

Change 474683 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: introduce jmx_exporter_port to logstash::collector

https://gerrit.wikimedia.org/r/474683

gerritbot added a comment.Nov 19 2018, 1:16 PM

Change 474683 merged by Filippo Giunchedi:
[operations/puppet@production] profile: introduce jmx_exporter_port to logstash::collector

https://gerrit.wikimedia.org/r/474683

herron closed this task as Resolved.Nov 19 2018, 3:19 PM
herron claimed this task.

Logs have been successfully shipped via the new pipeline comprised of rsyslog -> kafka -> logstash.

Transitioning this pipeline setup task to resolved and moving on to T205852 (onboarding producers to new pipeline)

fgiunchedi added a project: observability.Aug 19 2019, 2:31 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 19 2019, 3:15 PM
gerritbot added a comment.Sep 6 2021, 9:40 AM

Change 717311 had a related patch set uploaded (by Ema; author: Ema):

[operations/puppet@production] rsyslog: expand output lookup table docs

https://gerrit.wikimedia.org/r/717311

gerritbot added a project: Patch-For-Review.Sep 6 2021, 9:40 AM
gerritbot added a comment.Sep 6 2021, 9:43 AM

Change 717311 merged by Ema:

[operations/puppet@production] rsyslog: expand output lookup table docs

https://gerrit.wikimedia.org/r/717311

Maintenance_bot added a project: SRE Observability.Sep 6 2021, 9:45 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL