Page MenuHomePhabricator

Update userenque lambda to separate querying and fetching results from cloudwatch log insights
Open, Needs TriagePublic

Description

userenque lambda is responsible for querying clouwatch logs insights within a specific duration and filtering out the logs of users who are approaching their quotas. These user information get logged in SQS queue, from where another lambda picks up and sends them email.

Problem
Query to log insights is asynchronous. The query returns a query id. In the current implementation, the userenque lambda polls with this query id, until the query gets completed. We then get the result and send it to sqs queue.
This polling incurs extra lambda cost.

Implemention steps
Separate userenque lamdba into two:

  1. logquery lambda - sends query to log insights, returns query id. Decide how to store this query id (redis ?), such that the second lambda can pick up
  2. userenque lambda - triggers half an hour after logquery. Gets the results using query id and sends to sqs queue.

QA
Messages getting logged in sqs queue.

Event Timeline

Protsack.stephan raised the priority of this task from Low to Needs Triage.Mar 1 2023, 8:33 AM