I have downgraded to version 0.12.1 and it looks like things are better.

I will run the reindexing job now.

The reindexing job did not fare any better under this version. I think that the greater number of elements in the UI comes from the fact that the code in version 0.12.1 queries the database more and version 0.13.3 queries the ES indices more.

We think that what's happened is that the topic names used by the mae-consumer and mce-consumer processes have changed over the different versions and we have not kept up with the changes in our custom helm charts.
@brouberol noticed a large backlog of events in the kafka-jumbo cluster for the MetadataChangeLog_Version_v1 topic.

If we look at the values.yaml file for the upstream charts, we see a list of topics:

topics:
  metadata_change_event_name: "MetadataChangeEvent_v4"
  failed_metadata_change_event_name: "FailedMetadataChangeEvent_v4"
  metadata_audit_event_name: "MetadataAuditEvent_v4"
  datahub_usage_event_name: "DataHubUsageEvent_v1"
  metadata_change_proposal_topic_name: "MetadataChangeProposal_v1"
  failed_metadata_change_proposal_topic_name: "FailedMetadataChangeProposal_v1"
  metadata_change_log_versioned_topic_name: "MetadataChangeLog_Versioned_v1"
  metadata_change_log_timeseries_topic_name: "MetadataChangeLog_Timeseries_v1"
  platform_event_topic_name: "PlatformEvent_v1"
  datahub_upgrade_history_topic_name: "DataHubUpgradeHistory_v1"

These topic names are then referenced and assigned to environment variables in the upstream charts:

• mae-consumer deployment
• mce-consumer deployment
• gms deployment
• kafka-setup job

We have the same list of topics in our chart but they are not referenced sufficiently in the subcharts, so we do not have environment variables defined and this is, presumably, why the topics are not being published to or ready from correctly.

The mce-consumer has an incomplete list, but the gms and mae-consumer have none.

btullis@deploy2002:~$ kubectl get pod datahub-mce-consumer-production-8b5b4b5b5-thbhw  -o json | jq .spec.containers[0].env|grep METADATA
    "name": "METADATA_CHANGE_EVENT_NAME",
    "name": "FAILED_METADATA_CHANGE_EVENT_NAME",
    "name": "METADATA_CHANGE_PROPOSAL_TOPIC_NAME",
    "name": "FAILED_METADATA_CHANGE_PROPOSAL_TOPIC_NAME",
    "name": "METADATA_CHANGE_LOG_VERSIONED_TOPIC_NAME",
    "name": "METADATA_CHANGE_LOG_TIMESERIES_TOPIC_NAME",
btullis@deploy2002:~$ kubectl get pod datahub-mae-consumer-production-c4547866b-zwckm -o json | jq .spec.containers[0].env|grep METADATA
btullis@deploy2002:~$ kubectl get pod datahub-gms-production-65ff76fd4b-7lnmg  -o json | jq .spec.containers[0].env|grep METADATA
btullis@deploy2002:~$

Weird. I upgraded to version 0.13.3 again and now the UI is working. I can manage permissions again.
{F57631659,width=40%}

Adding to what @BTullis said, here's the list of configured topics for the mce-consumer

brouberol@deploy2002:~$ kubectl get pod datahub-mce-consumer-production-576cb6bc58-79cxk  -o json | jq .spec.containers[0].env|grep  -C1 METADATA
  {
    "name": "METADATA_CHANGE_EVENT_NAME",
    "value": "MetadataChangeEvent_v4"
--
  {
    "name": "FAILED_METADATA_CHANGE_EVENT_NAME",
    "value": "FailedMetadataChangeEvent_v4"
--
  {
    "name": "METADATA_CHANGE_PROPOSAL_TOPIC_NAME",
    "value": "MetadataChangeProposal_v1"
--
  {
    "name": "FAILED_METADATA_CHANGE_PROPOSAL_TOPIC_NAME",
    "value": "FailedMetadataChangeProposal_v1"
--
  {
    "name": "METADATA_CHANGE_LOG_VERSIONED_TOPIC_NAME",
    "value": "MetadataChangeLog_Versioned_v1"
--
  {
    "name": "METADATA_CHANGE_LOG_TIMESERIES_TOPIC_NAME",
    "value": "MetadataChangeLog_Timeseries_v1"

We can see that MetadataAuditEvent_v4 topic is nowhere to be found in that configuration. However, that seems logical, as the mce-consumer deals with ... MCE (Metadata Change Events), whereas the mae-consumer deals with .. MAE, aka Metadata Audit Events. I'll have a look at the upstream charts to gather the default values used in each case.

Change #1082151 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] datahub: add metadata topics env vars to the gms and mae-consumer

https://gerrit.wikimedia.org/r/1082151

https://datahubproject.io/docs/metadata-jobs/mae-consumer-job/

The Metadata Audit Event Consumer is a Spring job which can be deployed by itself, or as part of the Metadata Service.
Its main function is to listen to change log events emitted as a result of changes made to the Metadata Graph, converting changes in the metadata model into updates against secondary search & graph indexes (among other things)
Today the job consumes from two important Kafka topics:

• MetadataChangeLog_Versioned_v1
• MetadataChangeLog_Timeseries_v1

Where does the name Metadata Audit Event come from? Well, history. Previously, this job consumed a single MetadataAuditEvent topic which has been deprecated and removed from the critical path. Hence, the name!

So that would explain why we're not seeing any activity in MetadataAuditEvent_v4, I think.

Change #1082151 merged by Brouberol:

[operations/deployment-charts@master] datahub: add metadata topics env vars to the gms and mae-consumer

https://gerrit.wikimedia.org/r/1082151

Manually deploying https://gerrit.wikimedia.org/r/1082151 seems to have allowed the MAE to consume MetadataChangeLog_Versioned_v1, and rebuild the indices!

Here are some follow-up tickets from this incident:

• T377820: Increase the frequency of backups of the analytics_meta MariaDB instance
• T377814: Use an explicit index prefix for datahubsearch indices in production
• T377808: Enable missing healthchecks for DataHub services
• T377810: Create Grafana dashboard(s) for DataHub