By using RdKafka::KafkaConsumer class to consume messages across all partitions, we should be able to simplify our kafka source related.

Some of the advantages would be:
1) We can consume messages directly from the executor thread, this will allow us to avoid having a per partition thread and the inter thread queues to eventually be able to consume the message in the executor thread
2) We currently have a high level consumer that we use to commit offsets. We can now consolidate on just one high level consumer
3) It will be easier to support dynamically added partitions and topics
4) Using the low level per partition consumer is deprecated and so we can avoid having to use that

Some of the concerns to investigate are:
1) Does librdkafka internally try to ensure fairness across partitions wrt message consumption like we today do
2) Are there any perf issues with using the high level consumer?