User Story:

As a developer, I want to free up KSQLDB resources as soon as a client's push query completes or the HTTP connection disconnects so that realtime queries are not affected by previous push queries.

Acceptance Criteria:

Code Changes:

• Add a QueryClose call to the Push callback function in Realtime/articles.go.
• Add unit tests to verify that the close call is executed when expected.

Integration Testing:

• Run the realtime-stress-test using the test-by-timestamp command.
• Verify that the KSQLDB Custom Dashboard no longer shows long-running active queries. See chart
  
• Specifically, ensure that after the test run (e.g., running ./sequential.sh 10 "./test-by-timestamps" in the K8s branch of the realtime-stress-test repo), the red lines indicating active queries do not persist for an extended period (20 minutes) after the test completes.

ToDo:

Implement Code Changes:

Modify Realtime/articles.go to include a QueryClose call within the push callback function.
Write and Add Unit Tests:

Create unit tests to confirm that the QueryClose call is executed when a push query completes or the connection disconnects.

Perform Integration Testing:

Execute the realtime-stress-test using the ./sequential.sh 10 "./test-by-timestamps" command in the K8s branch.
Monitor the KSQLDB Custom Dashboard to ensure that no long-running active queries remain after test completion.

Test Strategy:

Testing Type:

• Unit Testing: Verify that the new QueryClose functionality is called correctly.
• Integration Testing: Use the realtime-stress-test scenario to ensure that the issue with lingering queries is resolved.
• Manual Verification: Monitor the KSQLDB dashboard manually during the integration test to confirm that active query indicators (red lines) do not persist unnecessarily.

Potential Impact:

Broken Areas: Changes may affect how realtime queries handle resource cleanup. Improper closing of connections could lead to resource leaks or affect query performance.
Monitoring: Any failure to free resources properly might impact system performance and should be caught by monitoring tools.

• 1. Checklist for Testing:
• Confirm that the QueryClose function is being executed in the push callback.
• Verify through unit tests that the connection closes correctly when expected.
• Run the integration test using the provided command and ensure that the KSQLDB dashboard no longer shows long-running active queries.
• Monitor system performance during and after the test to ensure no residual resource locks or performance degradation.

Code Suggestions

1. In the ksqldb submodule update the PushPuller interface to include the CloseQuery interface, the method is already in the submodule in client.go, add:

// Closer is an interface to wrap the default ksqld Close method for unit testing.
type Closer interface {
	CloseQuery(ctx context.Context, queryId string) error
}

2. Add the Closer to PushPuller

3. In realtime Repo, articles.go in the cbk callback function add a line similar to, around line 333:

  defer p.KSQLDB.CloseQuery(gcx.Request.Context(), hr.QueryID)
``
4. Check if we need a Close() call inside the error logic on line 383 when the Push call is made.