reference: https://mongodb.slack.com/archives/C05P740L0VC/p1724256981011949?thread_ts=1724250854.478959&cid=C05P740L0VC

Currently, if the source is idle, we sleep 100ms between each runOnce loop in stream processor.

The idea is that we should detect new documents in the source and process them with minimal latency when they are available. This should be balanced by waiting on input or some timeout like 2seconds when the source does not have any documents. However during the brainstorming we found couple of issues:

1. if we have idle time less than the time we want to wait, then idle time semantics would break.
2. Further, kafka use case with multiple partitions, it takes non-trivial amount of code to detect if any of the partitions have data with the current structure.

So we need to wait until we build processing time windows so that #1 is no longer a problem. We might also need to find an efficient way to detect if we have input for kafka source.