GRPC has a known performance issue with overloading a single channel with too many streams that can be mitigated by the application pooling multiple channels per remote. In the presence of too many streams on a single channel, gRPC will queue the streams on the channel which may lead to timeouts.

We could resolve this by tracking streams per channel and implementing a queue of channels in the channel pool, looking something like this (though the stub tracking logic is incorrect in the POC, but the gist is here).