We've deployed in staging a new application that creates a "page_view" stream, mostly based on the schema used in batch but we have to review a few things before productionizing this.
- We are not including any Maxmind transformation, like GeoIPs for now. To include this process we would require a more complex system where the maxmind database is updated inside Flink on a regular basis.
- We have to confirm that it's ok to miss this data for now.
- We are doing stateless transformations, mostly using the refinery code, but the refinery code is used inside UDFs and SQL. We are not aggregating anything.
- We should review all fields have the values we expect.
- The function (is_pageview()) that returns true/false and tells us is a webrequest is a page_view or not is based on the refinery code. There might be cases we want to do different.
- For example, redirects are not considered a page_view. Should we include them for relative trending? e.g: Maybe a page that redirects is shared in a media post and it gets trending, but it redirects to another. Which one should be trending? The current approach will only count the views on the final page.
- Performance
- It works great in staging, but we need to ensure it works well with the production throughput.
- The throughput of webrequest is very high, if we have an outage, the number of pending records will grow a lot very quickly. What do we want to do with errors?
- Do we prioritize and application that never fails, it just send messages to a DLQ, or do we want to fail if a major schema issue is happening?