Page MenuHomePhabricator

Flink Page View: Create K8s resources
Closed, ResolvedPublic

Description

For the Flink Page View application we need new resources in K8s. All the resources are required in the dse cluster.

  • Namespace
  • Users
  • S3 secrets
  • S3 bucket

I'm not sure what will be the right namespace name for this, maybe webrequest-page-view? This is a Flink application reading from webrequest_frontend_text and writing in a new topic called webrequest.page_view, although the topic and stream name could be changed if believe another name is better.

Task is done if:

  • Namespace webrequest-page-view is created
  • Users for the new namespace are created
  • Secrets for S3 are created an included in the namespace
  • A new S3 bucket is created

Event Timeline

Change #1287906 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/puppet@production] stream: webrequest-page-view

https://gerrit.wikimedia.org/r/1287906

Change #1287906 merged by Brouberol:

[operations/puppet@production] stream: webrequest-page-view

https://gerrit.wikimedia.org/r/1287906

Change #1289894 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] stream: webrequest-page-view

https://gerrit.wikimedia.org/r/1289894

I created the s3 user, added the keypair into the webrequest-page-view-next secrets and created the bucket

brouberol@cephosd1001:~$ sudo radosgw-admin user create --uid=webrequest-page-view-next --display-name="webrequest-page-view-next"
{
    "user_id": "webrequest-page-view-next",
    "display_name": "webrequest-page-view-next",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "webrequest-page-view-next",
            "access_key": "[REDACTED]",
            "secret_key": "[REDACTED]"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no mb s3://webrequest-page-view-next.dse-k8s-eqiad
Bucket 's3://webrequest-page-view-next.dse-k8s-eqiad/' created

Change #1290010 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad: define the webrequest-page-view-next namespace

https://gerrit.wikimedia.org/r/1290010

Change #1290010 merged by Brouberol:

[operations/deployment-charts@master] dse-k8s-eqiad: define the webrequest-page-view-next namespace

https://gerrit.wikimedia.org/r/1290010

Change #1289894 merged by jenkins-bot:

[operations/deployment-charts@master] stream: webrequest-page-view

https://gerrit.wikimedia.org/r/1289894

Change #1290715 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] flink: webrequest-page-view

https://gerrit.wikimedia.org/r/1290715

Change #1290715 merged by jenkins-bot:

[operations/deployment-charts@master] flink: webrequest-page-view

https://gerrit.wikimedia.org/r/1290715