2 questions:

1. where does the data for each of these sources live?
2. do we expect schema changes between ingestion runs?

is google sheets the permanent source for this data, or is there an API we can/ should pull from isntead?

@Ejegg @JMando 2 questions:

1. Do we still want this data? (I assume yes)
2. Does this data exist in the webrequest kafka streams we're currently ingesting? If the issue is that we aren't pulling those domains into pgehres but they exist in the kafka data or log files, then I can include the data in the banner impressions pipeline refactor

closing because we'll be taking a different approach to integrating P&T data with the FR Tech infra

sorry i missed this- I believe @JMando has some parsing logic that we can convert to druid. I dont know much about druid configs but if needed we can bring in P&T for guidance

i regenerated my public key- here it is:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPKOcE4nDmVZiJBqTCCEIEfmJn9YLf1Sb/h4l2rQf6Di astein@wikimedia.org

waiting on her phone number so i can send the ssl cert password

This is at the top of my list after the gsheet and sftp pipelines! I made some progress but hoping to get much further next week!

@RobH are you able to help me with this?

i've made the change

let's reduce retention to 2w

Thanks! Here's the info!

I've generated the cert and sent the pw to Neil over whatsapp!

@Ottomata is the eventlogging dataset still relevant, or is there a replacement? I see this phab and that the wikitech page is archived:
https://phabricator.wikimedia.org/T238230
https://wikitech.wikimedia.org/wiki/Analytics/Archive/EventLogging

@IAckerman-WMF do we want to compare the output of the two pipelines in the description? I don't know if there are still discrepancies between the two. On the other hand we might want to continue using the webrequest_text since we're more familiar with it. lmk what you think!

closing - dupe of https://phabricator.wikimedia.org/T252049

closing since this will be handled by the civi folks

i must have missed this earlier- this is just to copy the dbt data to dev schemas? what's the goal?

This will be part of a larger discussion on how we want to ingest P&T data

closing the phab now! I've created a subtask to work on the actual implementation
https://phabricator.wikimedia.org/T412844

we're ok with writing from the data lake to gsheets, but (at least for now) we want to lock down access so only dagster can write the data. The plan is to use duckdb to do this. I'm going to create a new phab for myself to work on this

this is my understanding too. A date on which we want to run the sync would be helpful- i can plan to meet with folks beforehand for testing to make sure we're ready to run the sync in prod!

also tagging in @BTullis

context slack thread: https://wikimedia.slack.com/archives/CSV483812/p1764777672488669

analytics ticket:
https://app.asana.com/1/3758245663860/project/1210424144163385/task/1211937797162579?focus=true

i believe this is handled by our dbt-post-merge script which runs dbt docs and dbt-metabase to sync the definitions from dbt to metabase

increasing the read timeout params for the iceberg and hive catalogs appear to have fixed this. closing

we havent seen this in a while so im going to close the ticket

Analytics folks are going to create a basic dashboard for kristine to review the data

Olga can access metabase!

the connector is merged and i materialized a small subset. Waiting on Kristine to lmk her preferred format to review the data

Olga is going to test metabase access tomorrow

yup! just sent a message to ITS about okta workflows and to Olga about logging in

just waiting for ITS to respond about adding Omari to the okta workflow table! Omari can now access metabase

Thanks! I should have an update this week for you!

Hey @thcipriani just wanted to follow up here and make sure the ticket didnt get lost!

@thcipriani re-answering your questions from above and picking this back up! Please let me know if we can provide any more info!

renamed to fundraising_analytics_dbt

pinning deepdiff fixed the issue!

i updated the user in the metabase ui and all appears to be working! marking as done!