Page MenuHomePhabricator

dynamicproxy data contains domains ending in '.' which can not be deleted via Horizon or cli tools
Closed, ResolvedPublic

Description

Discovered while working on T217992: Unable to delete "tiles.wmflabs.org" proxy entry via horizon.

These proxies are affected:

sqlite> select name, domain from route join project on route.project_id = project.id where domain like '%.';
maps|wma.wmflabs.org.
maps|0.wma.wmflabs.org.
maps|1.wma.wmflabs.org.
maps|2.wma.wmflabs.org.
maps|3.wma.wmflabs.org.
maps|4.wma.wmflabs.org.
maps|5.wma.wmflabs.org.
maps|6.wma.wmflabs.org.
maps|7.wma.wmflabs.org.
maps|label.wma.wmflabs.org.
dumps|dumps-stats.wmflabs.org.
tools|tools-checker-via-proxy.wmflabs.org.
mwv-apt|mwv-apt.wmflabs.org.
mobile|android-builds.wmflabs.org.
globaleducation|wikiedubackups.wmflabs.org.
visualeditor|togetherjs-hub.wmflabs.org.
visualeditor|togetherjs.wmflabs.org.
wikilabels|labels-staging.wmflabs.org.
design|design-research-methods.wmflabs.org.
discourse|discourse.wmflabs.org.
deployment-prep|mathoid-beta.wmflabs.org.
deployment-prep|restbase-beta.wmflabs.org.
deployment-prep|citoid-beta.wmflabs.org.
deployment-prep|graphoid-beta.wmflabs.org.
deployment-prep|sentry-beta.wmflabs.org.
dumps|bugs.wmflabs.org.
dumps|dumps-3.wmflabs.org.
dumps|dumps-1.wmflabs.org.
dumps|dumps-2.wmflabs.org.
extdist|extdist-01.wmflabs.org.
extdist|extdist.wmflabs.org.
math|mathbackend.wmflabs.org.
math|math-ru.wmflabs.org.
account-creation-assistance|accounts-oauth.wmflabs.org.
account-creation-assistance|accounts-dev.wmflabs.org.
account-creation-assistance|accounts.wmflabs.org.
services|appservice.wmflabs.org.
queryrapi|queryr.wmflabs.org.
wikimetrics|metrics.wmflabs.org.
wikimetrics|metrics-staging.wmflabs.org.
phragile|phragile.wmflabs.org.
otrs|otrs-oneclickspam.wmflabs.org.
dashiki|edit-analysis-test.wmflabs.org.
dashiki|language-reportcard-test.wmflabs.org.
dashiki|vital-signs-test.wmflabs.org.
dashiki|browser-reports-test.wmflabs.org.
dashiki|vital-signs.wmflabs.org.
dashiki|edit-analysis.wmflabs.org.
dashiki|language-reportcard.wmflabs.org.
dashiki|browser-reports.wmflabs.org.

Event Timeline

bd808 created this task.Mar 12 2019, 12:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2019, 12:32 AM

The 'fix' for this should be to clean the data in the database and redis cache to remove the trailing ..

Mentioned in SAL (#wikimedia-cloud) [2019-03-12T01:13:42Z] <bd808> Deleted dangling backend records in /etc/dynamicproxy-api/data.db (T218064)

Mentioned in SAL (#wikimedia-cloud) [2019-03-12T03:00:07Z] <bd808> Fixed domains with trailing dot (T218064)

bd808 claimed this task.Mar 12 2019, 3:04 AM
bd808 triaged this task as Medium priority.
fix-proxy-domains.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# Copyright (c) 2019 Bryan Davis and Wikimedia Foundation. All Rights Reserved.
from __future__ import print_function

import argparse

import redis
import sqlite3


def main():
    parser = argparse.ArgumentParser(description='Fix proxy domains')
    parser.add_argument(
        '--do-it', dest='do_it', action='store_true',
        help='Update database and redis')
    args = parser.parse_args()

    db = sqlite3.connect('/etc/dynamicproxy-api/data.db')
    cur = db.cursor()
    r = redis.Redis()

    cur.execute("""
    SELECT
      p.name as project,
      r.domain as domain,
      r.id as route_id,
      b.url as url,
      b.id as backend_id
    FROM
      project p
      join route r on p.id = r.project_id
      join backend b on r.id = b.route_id
    ORDER BY project, domain, url
    """)
    for project, domain, route_id, url, backend_id in cur.fetchall():
        new_domain = domain.rstrip('.')

        if new_domain != domain:
            print(project, ':', domain, '->', new_domain)
            if args.do_it:
                cur.execute(
                    "UPDATE route SET domain = :domain WHERE id = :id",
                    {'domain': new_domain, 'id': route_id}
                )
                pipe = r.pipeline()
                pipe.delete('frontend:' + domain)
                key = 'frontend:' + new_domain
                pipe.delete(key).sadd(key, url).execute()
    if args.do_it:
        db.commit()
    db.close()


main()
$ sudo ./fix-proxy-domains.py --do-it
account-creation-assistance : accounts-dev.wmflabs.org. -> accounts-dev.wmflabs.org
account-creation-assistance : accounts-oauth.wmflabs.org. -> accounts-oauth.wmflabs.org
account-creation-assistance : accounts.wmflabs.org. -> accounts.wmflabs.org
dashiki : browser-reports-test.wmflabs.org. -> browser-reports-test.wmflabs.org
dashiki : browser-reports.wmflabs.org. -> browser-reports.wmflabs.org
dashiki : edit-analysis-test.wmflabs.org. -> edit-analysis-test.wmflabs.org
dashiki : edit-analysis.wmflabs.org. -> edit-analysis.wmflabs.org
dashiki : language-reportcard-test.wmflabs.org. -> language-reportcard-test.wmflabs.org
dashiki : language-reportcard.wmflabs.org. -> language-reportcard.wmflabs.org
dashiki : vital-signs-test.wmflabs.org. -> vital-signs-test.wmflabs.org
dashiki : vital-signs.wmflabs.org. -> vital-signs.wmflabs.org
deployment-prep : citoid-beta.wmflabs.org. -> citoid-beta.wmflabs.org
deployment-prep : graphoid-beta.wmflabs.org. -> graphoid-beta.wmflabs.org
deployment-prep : mathoid-beta.wmflabs.org. -> mathoid-beta.wmflabs.org
deployment-prep : restbase-beta.wmflabs.org. -> restbase-beta.wmflabs.org
deployment-prep : sentry-beta.wmflabs.org. -> sentry-beta.wmflabs.org
design : design-research-methods.wmflabs.org. -> design-research-methods.wmflabs.org
discourse : discourse.wmflabs.org. -> discourse.wmflabs.org
dumps : bugs.wmflabs.org. -> bugs.wmflabs.org
dumps : dumps-1.wmflabs.org. -> dumps-1.wmflabs.org
dumps : dumps-2.wmflabs.org. -> dumps-2.wmflabs.org
dumps : dumps-3.wmflabs.org. -> dumps-3.wmflabs.org
dumps : dumps-stats.wmflabs.org. -> dumps-stats.wmflabs.org
extdist : extdist-01.wmflabs.org. -> extdist-01.wmflabs.org
extdist : extdist.wmflabs.org. -> extdist.wmflabs.org
globaleducation : wikiedubackups.wmflabs.org. -> wikiedubackups.wmflabs.org
maps : 0.wma.wmflabs.org. -> 0.wma.wmflabs.org
maps : 1.wma.wmflabs.org. -> 1.wma.wmflabs.org
maps : 2.wma.wmflabs.org. -> 2.wma.wmflabs.org
maps : 3.wma.wmflabs.org. -> 3.wma.wmflabs.org
maps : 4.wma.wmflabs.org. -> 4.wma.wmflabs.org
maps : 5.wma.wmflabs.org. -> 5.wma.wmflabs.org
maps : 6.wma.wmflabs.org. -> 6.wma.wmflabs.org
maps : 7.wma.wmflabs.org. -> 7.wma.wmflabs.org
maps : label.wma.wmflabs.org. -> label.wma.wmflabs.org
maps : wma.wmflabs.org. -> wma.wmflabs.org
math : math-ru.wmflabs.org. -> math-ru.wmflabs.org
math : mathbackend.wmflabs.org. -> mathbackend.wmflabs.org
mobile : android-builds.wmflabs.org. -> android-builds.wmflabs.org
mwv-apt : mwv-apt.wmflabs.org. -> mwv-apt.wmflabs.org
otrs : otrs-oneclickspam.wmflabs.org. -> otrs-oneclickspam.wmflabs.org
phragile : phragile.wmflabs.org. -> phragile.wmflabs.org
queryrapi : queryr.wmflabs.org. -> queryr.wmflabs.org
services : appservice.wmflabs.org. -> appservice.wmflabs.org
tools : tools-checker-via-proxy.wmflabs.org. -> tools-checker-via-proxy.wmflabs.org
visualeditor : togetherjs-hub.wmflabs.org. -> togetherjs-hub.wmflabs.org
visualeditor : togetherjs.wmflabs.org. -> togetherjs.wmflabs.org
wikilabels : labels-staging.wmflabs.org. -> labels-staging.wmflabs.org
wikimetrics : metrics-staging.wmflabs.org. -> metrics-staging.wmflabs.org
wikimetrics : metrics.wmflabs.org. -> metrics.wmflabs.org
bd808 closed this task as Resolved.Mar 12 2019, 3:04 AM