Page MenuHomePhabricator
Create Task
Maniphest T265963

Replace production deployment servers and update them to Buster
Closed, ResolvedPublic
Actions

Description

deploy1001 and deploy2001 are going to be replaced with new hardware as part of regular hardware refresh due to age

  • deploy1002 setup T265653
  • deploy2002 setup T264633

Since we are migrating all servers to Buster, it makes sense to directly install Buster.

  • install buster on deploy1002
  • install buster on deploy2002
  • apply deployment server role on deploy1002
  • temp remove deploy1002 from scap dsh groups to avoid errors for deployers
  • fix: E: Package 'mysql-client' has no installation candidate (/Profile::Mediawiki::Deployment::Server/File[/usr/local/sbin/fix-staging-perms]: Dependency Package[mysql-client] has failures: true)
  • fix scap bootstrap issue / run scap deploy --init (worked around with rsync/puppet, scap-sync-master and scap pull)

(/usr/bin/scap deploy --init fails when running on non-active servers)

  • apply deployment server role on deploy2002
  • re-add both servers to scap DSH groups
  • sync repo data over from old servers to new servers
  • schedule and announce switchover date
  • actually switch active server in puppet, check the other servers have the "warning MOTD" to tell devs what is and what isn't the right server

Details

SubjectRepoBranchLines +/-
mariadb: update grants for deployment servers to prod-m5, drop from clouddboperations/puppetproduction+2 -5
add deploy1002/deploy2002 to scap firewalloperations/homer/publicmaster+8 -0
hiera/scap: switch deployment server to deploy1002operations/puppetproduction+8 -7
switch deployment CNAME from deploy1001 to deploy1002operations/dnsmaster+2 -2
deployment: allow syncing home dirs to other dpeloyment serversoperations/puppetproduction+6 -0
deployment_server: add the php restart commands here as welloperations/puppetproduction+2 -1
scap: switch codfw deployment server and scap master to deploy2002operations/puppetproduction+3 -3
remove deploy1001.eqiad.wmnetoperations/dnsmaster+0 -4
scap: add deploy1002 and deploy2002 to mediawiki hostsoperations/puppetproduction+2 -0
add deploy1002 and deploy2002 to deployment_hosts for firewallsoperations/puppetproduction+4 -0
add fake mcrouter certs for deploy2002labs/privatemaster+0 -0
deployment::server: buster support, use default-mysql-client packageoperations/puppetproduction+1 -1
site: add deploy2002 and unify deployment server role regexoperations/puppetproduction+1 -6
scap: temp remove deploy1002 from dsh group (don't deploy mw to it)operations/puppetproduction+0 -1
switch deploy1002/deploy2002 to use buster installeroperations/puppetproduction+0 -2
scap/dsh: add deploy1002 to mediawiki_installation hostsoperations/puppetproduction+1 -0
site: add deployment_server role on deploy1002operations/puppetproduction+1 -5
add fake mcrouter certs for deploy1002.eqiad.wmnetlabs/privatemaster+0 -0
tcpircbot: add deploy1002 to allowed hostsoperations/puppetproduction+3 -0
cumin: remove hardcoded hostname from commentsoperations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedJdforrester-WMFT313563 Bump lcobucci/jwt & league/uri-components for php8
 ResolvedJdforrester-WMFT313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJdforrester-WMFT264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
 ResolvedLadsgroupT270740 Drop hacky support for doctrine/dbal class renaming
 InvalidNoneT303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
 ResolvedReedyT251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
 OpenNoneT166010 The Great Namespaceization Effort
 Resolved• tstarlingT277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
 ResolvedJdforrester-WMFT261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedPRODUCTION ERRORLegoktmT293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
 Resolved• tstarlingT297667 mysqli/mysqlnd memory leak
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedKrinkleT248191 Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest)
 ResolvedhasharT278203 Migrate all CI jobs from stretch to buster or later and drop stretch testing support
 ResolvedJdforrester-WMFT252432 Drop MediaWiki testing in stretch and instead test only in buster
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedDzahnT265963 Replace production deployment servers and update them to Buster
Unknown Object (Task)
 ResolvedPapaulT266363 (Need By: TBD) rack/setup/install deploy2002
 Resolved• CmjohnsonT265653 (Need By: TBD) setup/install deploy1002
 ResolvedRequest• CmjohnsonT275831 decommission deploy1001.eqiad.wmnet
 ResolvedRequestPapaulT275832 decommission deploy2001.codfw.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Oct 22 2020, 9:56 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-22T21:56:09Z] <mutante> deploy1002 - scap pull and added to mediawiki-installation "dsh" group - will be part of scap trains but just like any appserver (T265963)

Stashbot added a comment.Oct 22 2020, 10:03 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-22T22:03:12Z] <mutante> deploy1002 - armed keyholder, all deployment keys loaded T265963

• Cmjohnson closed subtask T265653: (Need By: TBD) setup/install deploy1002 as Resolved.Oct 30 2020, 4:21 PM
RhinosF1 subscribed.Oct 30 2020, 4:26 PM
Dzahn moved this task from Incoming 🐫 to API Gateway 🥌 on the serviceops board.Nov 3 2020, 2:16 AM
Dzahn mentioned this in Unknown Object (Task).Nov 3 2020, 2:53 AM
Papaul closed subtask Unknown Object (Task) as Resolved.Nov 9 2020, 4:06 PM
MoritzMuehlenhoff subscribed.Nov 26 2020, 12:44 PM

Let's directly install deploy1002 with Buster? This allows to adapt Puppet manifests/packages for Buster and run tests and then in January the production deployment server can simply be switched over?

jijiki added a parent task: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).Nov 26 2020, 1:39 PM
jijiki updated the task description. (Show Details)Nov 26 2020, 1:57 PM
jijiki subscribed.
In T265963#6651261, @MoritzMuehlenhoff wrote:

Let's directly install deploy1002 with Buster? This allows to adapt Puppet manifests/packages for Buster and run tests and then in January the production deployment server can simply be switched over?

Yes! I have updated the description to reflect that. If we install Buster now, we are sparing ourselves the drama of upgrading later.

jijiki renamed this task from replace production deployment servers to Replace production deployment servers and update them to Buster.Nov 26 2020, 3:14 PM
gerritbot added a comment.Nov 30 2020, 8:12 PM

Change 644319 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] switch deploy1002/deploy2002 to use buster installer

https://gerrit.wikimedia.org/r/644319

gerritbot added a comment.Nov 30 2020, 8:14 PM

Change 644319 merged by Dzahn:
[operations/puppet@production] switch deploy1002/deploy2002 to use buster installer

https://gerrit.wikimedia.org/r/644319

gerritbot added a comment.Nov 30 2020, 8:25 PM

Change 644320 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: temp remove deploy1002 from dsh group (don't deploy mw to it)

https://gerrit.wikimedia.org/r/644320

gerritbot added a comment.Nov 30 2020, 8:27 PM

Change 644320 merged by Dzahn:
[operations/puppet@production] scap: temp remove deploy1002 from dsh group (don't deploy mw to it)

https://gerrit.wikimedia.org/r/644320

ops-monitoring-bot added a comment.Nov 30 2020, 8:28 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

deploy1002.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011302027_dzahn_8991_deploy1002_eqiad_wmnet.log.

Stashbot added a comment.Nov 30 2020, 8:28 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-30T20:28:51Z] <mutante> reimaging deploy1002 with buster - not the active deployment server, deploy1001 still is (T265963)

ops-monitoring-bot added a comment.Nov 30 2020, 8:42 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

deploy2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011302042_dzahn_12260_deploy2002_codfw_wmnet.log.

Stashbot added a comment.Nov 30 2020, 8:42 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-30T20:42:53Z] <mutante> reimaging deploy2002 with buster (not active, deploy1001/2001 are) T265963

ops-monitoring-bot added a comment.Nov 30 2020, 9:18 PM

Completed auto-reimage of hosts:

['deploy2002.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Nov 30 2020, 9:43 PM

Change 644333 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add deploy2002 and unify deployment server role regex

https://gerrit.wikimedia.org/r/644333

Dzahn updated the task description. (Show Details)Nov 30 2020, 10:33 PM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Nov 30 2020, 10:36 PM
gerritbot added a comment.Nov 30 2020, 10:48 PM

Change 644350 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] deployment::server: buster support, use mariadb-client, not mysql-client

https://gerrit.wikimedia.org/r/644350

ops-monitoring-bot added a comment.Nov 30 2020, 11:08 PM

Completed auto-reimage of hosts:

['deploy1002.eqiad.wmnet']

Of which those FAILED:

['deploy1002.eqiad.wmnet']
gerritbot added a comment.Dec 1 2020, 8:01 PM

Change 644350 merged by Dzahn:
[operations/puppet@production] deployment::server: buster support, use default-mysql-client package

https://gerrit.wikimedia.org/r/644350

Dzahn updated the task description. (Show Details)Dec 1 2020, 8:11 PM
Dzahn added a comment.Dec 1 2020, 8:16 PM

@jijiki The issue with "mysql-client" not existing on buster has been fixed by using default-mysql-client. This was noop on stretch deployment servers and removed a puppet error and further dependencies for httpbb tests on the new buster servers.

The next issue is this:

Error: Execution of '/usr/bin/scap deploy --init' returned 70: 20:10:30 deploy failed: <LockFailedError> Failed to acquire lock "/var/lock/scap-global-lock"; owner is "root"; reason is "Not the active deployment server, use deploy1001.eqiad.wmnet"

puppet tries to run "scap deploy --init" (many times actually, once for each repo but they all fail because scap is locked when it is not on the active deployment server.

So either this just needs to wait until we have an actual maintenance window where we switch and then someone does the scap init dance... ( I think releng used to do this last time) or we need to allow running init on non-active servers but making sure we never mess with actual deployments of course. The horror scenario would be that it syncs to appservers before having the current code from the previous server.

Dzahn updated the task description. (Show Details)Dec 1 2020, 8:17 PM
gerritbot added a comment.Dec 1 2020, 8:26 PM

Change 644333 merged by Dzahn:
[operations/puppet@production] site: add deploy2002 and unify deployment server role regex

https://gerrit.wikimedia.org/r/644333

Dzahn added a subscriber: hashar.EditedDec 1 2020, 8:59 PM

As @hashar pointed out there is also T257317 which is describing exactly this issue and I can simply quote Jaime: "scap syncronization (all methods) should be disabled because of the lock, but probably --init should be allowed".

and T257319

gerritbot added a comment.Dec 1 2020, 9:06 PM

Change 644616 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake mcrouter certs for deploy2002

https://gerrit.wikimedia.org/r/644616

gerritbot added a comment.Dec 1 2020, 9:07 PM

Change 644616 merged by Dzahn:
[labs/private@master] add fake mcrouter certs for deploy2002

https://gerrit.wikimedia.org/r/644616

Dzahn mentioned this in rLPRI046bfa633626: add fake mcrouter certs for deploy2002.Dec 1 2020, 9:09 PM
Dzahn mentioned this in T257317: scap deploy --init on deployment server fails on first puppet run.Dec 1 2020, 9:12 PM
ops-monitoring-bot added a comment.Dec 1 2020, 9:17 PM

Icinga downtime for 40 days, 0:00:00 set by dzahn@cumin1001 on 1 host(s) and their services with reason: new_install

deploy2002.codfw.wmnet
Stashbot added a comment.Dec 1 2020, 9:18 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-01T21:18:51Z] <mutante> applied deployment_server role on deploy2002, added mcrouter cert, initial puppet run pulls mediawiki-config and other repos, downtimed in Icinga for 40 days (T265963)

Dzahn mentioned this in T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).Dec 22 2020, 7:47 PM
Stashbot added a comment.Dec 22 2020, 9:31 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-22T21:31:15Z] <mutante> deploy1002/deploy2002 - apt-get remove --purge php-readline and let puppet reinstall it (7.2 vs 7.3 after gerrit 651158) T265963

Dzahn added a comment.Dec 22 2020, 9:35 PM

php-readline 2:7.2+69+0~20190215163918.14+stretch~1.gbpfa617b+wmf1 is installed on the new servers now after deploying https://gerrit.wikimedia.org/r/651158 and letting puppet reinstall it

gerritbot added a comment.Jan 26 2021, 4:34 PM

Change 635079 merged by Dzahn:
[operations/puppet@production] add deploy1002 and deploy2002 to deployment_hosts for firewalls

https://gerrit.wikimedia.org/r/635079

gerritbot added a comment.Jan 26 2021, 5:06 PM

Change 658643 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: add deploy2002 to mediawiki installation hosts

https://gerrit.wikimedia.org/r/658643

thcipriani edited projects, added Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)); removed Release-Engineering-Team.Jan 26 2021, 5:31 PM
thcipriani added a project: Release-Engineering-Team (Deployment services).
thcipriani moved this task from INBOX to Maintenance on the Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)) board.
gerritbot added a comment.Jan 27 2021, 6:17 PM

Change 658643 merged by Dzahn:
[operations/puppet@production] scap: add deploy1002 and deploy2002 to mediawiki hosts

https://gerrit.wikimedia.org/r/658643

Dzahn mentioned this in T275826: L10n cache files building up on backup deploy hosts.Feb 25 2021, 11:49 PM
Dzahn added subtasks: T275826: L10n cache files building up on backup deploy hosts, T257317: scap deploy --init on deployment server fails on first puppet run.Feb 25 2021, 11:51 PM
Stashbot added a comment.Feb 25 2021, 11:55 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-25T23:55:13Z] <mutante> deploy1002, deploy2002 - scap-master-sync deploy1001.eqiad.wmnet (T265963)

gerritbot added a comment.Feb 25 2021, 11:59 PM

Change 635114 abandoned by Dzahn:
[operations/dns@master] remove deploy1001.eqiad.wmnet

Reason:
DNS is now automated

https://gerrit.wikimedia.org/r/635114

Dzahn raised the priority of this task from Medium to High.Feb 26 2021, 12:12 AM
gerritbot added a comment.Feb 26 2021, 12:39 AM

Change 667043 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: switch codfw deployment server and scap master to deploy2002

https://gerrit.wikimedia.org/r/667043

Stashbot added a comment.Feb 26 2021, 8:29 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-26T20:29:48Z] <mutante> deploy2001 - /srv/mediawiki-staging sudo find . -name *.cdb delete - deleted 190 GB of old cdb files (T275826 T265963)

gerritbot added a comment.Feb 26 2021, 9:05 PM

Change 667043 merged by Dzahn:
[operations/puppet@production] scap: switch codfw deployment server and scap master to deploy2002

https://gerrit.wikimedia.org/r/667043

gerritbot added a comment.Feb 26 2021, 9:57 PM

Change 667277 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] deployment_server: add the php restart commands here as well

https://gerrit.wikimedia.org/r/667277

gerritbot added a comment.Feb 26 2021, 10:04 PM

Change 667277 abandoned by Dzahn:
[operations/puppet@production] deployment_server: add the php restart commands here as well

Reason:
requires a lot of other things from mediawiki::php

https://gerrit.wikimedia.org/r/667277

gerritbot added a comment.Feb 26 2021, 10:13 PM

Change 667278 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] deployment: allow syncing home dirs to other dpeloyment servers

https://gerrit.wikimedia.org/r/667278

gerritbot added a comment.Feb 26 2021, 10:24 PM

Change 667278 merged by Dzahn:
[operations/puppet@production] deployment: allow syncing home dirs to other dpeloyment servers

https://gerrit.wikimedia.org/r/667278

Dzahn added a comment.Feb 27 2021, 1:21 AM

switching eqiad scheduled for Monday, March 1st:

https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_01

announced on ops and wikitech-l

Dzahn updated the task description. (Show Details)Feb 27 2021, 1:22 AM
gerritbot added a comment.Mar 1 2021, 8:24 PM

Change 635113 merged by Dzahn:
[operations/dns@master] switch deployment CNAME from deploy1001 to deploy1002

https://gerrit.wikimedia.org/r/635113

gerritbot added a comment.Mar 1 2021, 9:03 PM

Change 635105 merged by Dzahn:
[operations/puppet@production] hiera/scap: switch deployment server to deploy1002

https://gerrit.wikimedia.org/r/635105

Stashbot added a comment.Mar 1 2021, 9:05 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-01T21:05:19Z] <mutante> re-enabling puppet on deploy1001 - running puppet on deploy*, switching eqiad scap master and deployment_server globally (T265963)

Stashbot added a comment.Mar 1 2021, 9:08 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-01T21:08:55Z] <mutante> [mwdebug1001:~] $ /usr/local/lib/nagios/plugins/check_mw_versions --deployhost deploy1002.eqiad.wmnet - OKAY: wikiversions in sync (T265963)

Stashbot added a comment.Mar 1 2021, 9:38 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-01T21:38:16Z] <mutante> cumin 'mw*' 'grep master_rsync /etc/scap.cfg' showed all mw servers are now using deploy1002 (T265963)

gerritbot added a comment.Mar 1 2021, 10:11 PM

Change 667718 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/homer/public@master] add deploy1002/deploy2002 to scap firewall

https://gerrit.wikimedia.org/r/667718

gerritbot added a comment.Mar 1 2021, 10:23 PM

Change 667718 merged by Dzahn:
[operations/homer/public@master] add deploy1002/deploy2002 to scap firewall

https://gerrit.wikimedia.org/r/667718

Dzahn removed a project: Patch-For-Review.Mar 1 2021, 10:40 PM
Dzahn updated the task description. (Show Details)
Dzahn mentioned this in rOHPU3a15f0f2246e: add deploy1002/deploy2002 to scap firewall.Mar 1 2021, 10:45 PM
Dzahn closed this task as Resolved.Mar 2 2021, 1:12 AM
Dzahn added a subtask: T275831: decommission deploy1001.eqiad.wmnet.Mar 2 2021, 11:42 PM
Dzahn added a subtask: T275832: decommission deploy2001.codfw.wmnet.Mar 5 2021, 8:44 PM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Mar 5 2021, 9:20 PM

Change 668785 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mariadb: update grants for deployment servers to clouddb and prod-m5

https://gerrit.wikimedia.org/r/668785

gerritbot added a project: Patch-For-Review.Mar 5 2021, 9:20 PM
Dzahn added a subscriber: Andrew.Mar 8 2021, 9:39 PM

@Andrew Could you please deploy a change for clouddb grants? see comment at the bottom of https://gerrit.wikimedia.org/r/c/operations/puppet/+/668785 Thanks!

Papaul closed subtask T275832: decommission deploy2001.codfw.wmnet as Resolved.Mar 9 2021, 6:26 PM
• Cmjohnson closed subtask T275831: decommission deploy1001.eqiad.wmnet as Resolved.Mar 11 2021, 3:32 PM
hashar unsubscribed.Mar 11 2021, 3:42 PM
gerritbot added a comment.Mar 11 2021, 5:35 PM

Change 668785 abandoned by Dzahn:
[operations/puppet@production] mariadb: update grants for deployment servers to clouddb and prod-m5

Reason:
comments above

https://gerrit.wikimedia.org/r/668785

gerritbot added a comment.Mar 11 2021, 5:35 PM

Change 668785 restored by Dzahn:
[operations/puppet@production] mariadb: update grants for deployment servers to clouddb and prod-m5

https://gerrit.wikimedia.org/r/668785

gerritbot added a comment.Mar 15 2021, 9:49 AM

Change 668785 merged by Kormat:
[operations/puppet@production] mariadb: update grants for deployment servers to prod-m5, drop from clouddb

https://gerrit.wikimedia.org/r/668785

Izno removed a subtask: T275826: L10n cache files building up on backup deploy hosts.Jun 1 2021, 11:12 PM
Izno removed a subtask: T257317: scap deploy --init on deployment server fails on first puppet run.
hashar added a subtask: T257317: scap deploy --init on deployment server fails on first puppet run.Jun 7 2021, 2:01 PM
Dzahn merged a task: T242480: package requirements for upgrading deployment_servers to buster.Oct 22 2021, 6:14 PM
Dzahn added subscribers: taavi, Jdforrester-WMF, Paladox.
Jdforrester-WMF removed a subtask: T257317: scap deploy --init on deployment server fails on first puppet run.Nov 30 2021, 8:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits