We should look into setting the TCP_KEEPIDLE/KEEPINTVL/KEEPCNT settings based on $maxTimeMS. This would allow us to distinguish the case of the remote host taking a long time to perform the requested operation, vs being blackholed. We need to do a bit of testing to find the optimal values here, but I think we should consider the host dead after something like 500ms <= maxTimeMS / 2 <= 15seconds. This would give us a chance to retry using a different host while still staying within the user-provided timeout. We may want to only do this for retryable operations.

Original Title: Make TCP_KEEPALIVE parameters configurable

See here.

AC: Investigate to see if this is something that we should pursue. Reach out to Kelsey and Kal for further context.