SERVER-15902

Ok I did some testing with sigaltstack with and without libunwind.
The code is available at:

https://github.com/BillyDonahue/mongo/blob/4180d90435/src/mongo/util/stacktrace_test.cpp#L374
( PQ: https://evergreen.mongodb.com/version/5da29316e3c3313e308f739e )

The very good news is that libunwind and execinfo both do fine at starting
from a frame on a sigaltstack and following the trace down into the normal
stack of the interrupted thread. Here's a libunwind stack trace,
passed through `c++filt`.

 libbase.so(mongo::stack_trace::printmongo::stack_trace::Options&)+0x40) [0x7FB9A0D85BA9]
 stacktrace_test(mongo::printStackTrace()+0x3F) [0x5590AABA7092]
 stacktrace_test(mongo::(anonymous namespace)::UnitTest_SuiteNameStackTraceSigAltStackTestTestNamePrint::_doTest()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const+0x26) [0x5590AAB97D1A]
 stacktrace_test(mongo::(anonymous namespace)::UnitTest_SuiteNameStackTraceSigAltStackTestTestNamePrint::_doTest()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*)+0x2A) [0x5590AAB97D47]
 libpthread.so.0(funlockfile+0x50) [0x7FB99E215890]
 libpthread.so.0(raise+0xC7) [0x7FB99E215727]
 stacktrace_test(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}::operator()() const+0x277) [0x5590AAB978E7]
 stacktrace_test(void std::__invoke_impl<void, mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}>(std::__invoke_other, mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}&&)+0x20) [0x5590AABA0298]
 stacktrace_test(std::__invoke_result<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}>::type std::__invoke<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}>(std::__invoke_result&&, (mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}&&)...)+0x20) [0x5590AAB9E773]
 stacktrace_test(decltype(auto) std::__apply_impl<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}, std::tuple<>>(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}&&, std::tuple<>&&, std::integer_sequence<unsigned long>)+0x24) [0x5590AAB9B991]
 stacktrace_test(decltype(auto) std::apply<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}, std::tuple<> >(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}&&, std::tuple<>&&)+0x37) [0x5590AAB9B9CB]
 stacktrace_test(mongo::stdx::thread::thread<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}, , 0>(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1})::{lambda()#1}::operator()()+0x46) [0x5590AAB9BA1A]
 stacktrace_test(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1} std::__invoke_impl<void, mongo::stdx::thread::thread<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}, , 0>(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1})::{lambda()#1}>(std::__invoke_other, mongo::stdx::thread::thread<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}, , 0>(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1})::{lambda()#1}&&)+0x20) [0x5590AABA039F]
 stacktrace_test(_ZSt8__invokeIZN5mongo4stdx6threadC4IZNS0_12_GLOBAL__N_125StackTraceSigAltStackTest10tryHandlerEPFviP9siginfo_tPvEEUlvE_JELi0EEET_DpOT0_EUlvE_JEENSt15__invoke_resultISC_JDpSD_EE4typeEOSC_SF_+0x20) [0x5590AAB9E7A4]
 stacktrace_test(decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}, , 0>(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1})::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>)+0x28) [0x5590AABA49A4]
 stacktrace_test(std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}, , 0>(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1})::{lambda()#1}> >::operator()()+0x18) [0x5590AABA497A]
 stacktrace_test(std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1}, , 0>(mongo::(anonymous namespace)::StackTraceSigAltStackTest::tryHandler(void (*)(int, siginfo_t*, void*))::{lambda()#1})::{lambda()#1}> > >::_M_run()+0x1C) [0x5590AABA482A]
 stacktrace_test(execute_native_thread_routine+0xF) [0x5590AABFC0DF]
 libpthread.so.0(start_thread+0xDB) [0x7FB99E20A6DB]
 libc.so.6(clone+0x3F) [0x7FB99DF3388F]

Without libunwind, btw, we get almost no symbolic information here:

 libbase.so(mongo::stack_trace::print(mongo::stack_trace::Options&)+0x6A) [0x7FADAE839003]
 stacktrace_test(mongo::printStackTrace()+0x3F) [0x55FEA082C012]
 stacktrace_test(+0x92C9A) [0x55FEA081CC9A]
 stacktrace_test(+0x92CC7) [0x55FEA081CCC7]
 libpthread.so.0(+0x12890) [0x7FADABF22890]
 libpthread.so.0(raise+0xC7) [0x7FADABF22727]
 stacktrace_test(+0x92867) [0x55FEA081C867]
 stacktrace_test(+0x9B218) [0x55FEA0825218]
 stacktrace_test(+0x996F3) [0x55FEA08236F3]
 stacktrace_test(+0x96911) [0x55FEA0820911]
 stacktrace_test(+0x9694B) [0x55FEA082094B]
 stacktrace_test(+0x9699A) [0x55FEA082099A]
 stacktrace_test(+0x9B31F) [0x55FEA082531F]
 stacktrace_test(+0x99724) [0x55FEA0823724]
 stacktrace_test(+0x9F924) [0x55FEA0829924]
 stacktrace_test(+0x9F8FA) [0x55FEA08298FA]
 stacktrace_test(+0x9F7AA) [0x55FEA08297AA]
 stacktrace_test(+0xF705F) [0x55FEA088105F]
 libpthread.so.0(+0x76DB) [0x7FADABF176DB]
 libc.so.6(clone+0x3F) [0x7FADABC4088F]


The problem is that we use a lot of stack. The SIGSTKSZ
which we'd assume to be a reasonable default is only 8kiB,
which is insufficient to support libunwind backtracing.

But by how much? My test installs a large sigaltstack of 1 MiB
initialized with the byte value 0xda, then `raise()`s a signal, then
scans the 1MiB sigaltstack buffer for the first non-0xda byte.

This was repeated for multiple handler actions: a minimal handler,
a handler that calls `printStackTrace`, and one that calls
`backtrace`.

sigaltstack high water mark in dynamic builds:

.
.                                    signal handler action
.  --use-libunwind : ----\       =============================
.  --dbg=on        : -\   \      minimal |  print  | backtrace
.                     =   =      ========|=========|==========
.                     N   N :      4,344 |   7,144 |     5,096
.                     Y   N :      4,424 |   7,528 |     5,160
.                     N   Y :      4,344 |  13,048 |     7,352
.                     Y   Y :      4,424 |  13,672 |     8,392
.

This makes some sense, as the loop inside of printStackTrace has to use
unwinder symbolification. The critical parameter here is that execinfo uses
slightly less than the 8kiB SIGSTKSZ, and libunwind uses significantly more.

I have heard that previous testing showed that libunwind could not handle
the sigaltstack, but I don't remember hearing what failed. The failure
may have been due to its higher high water mark.

We can use sigaltstack with libunwind but we will need to be conscious of
printStackTrace consumption.

We probably don't want to use too much if we can help it, since we have a lot
of threads.  So if we find it difficult to estimate a tight upper bound, maybe
a runtime probe like the one in the unit test can be used to make an adaptive
estimate. Maybe we can take virtual memory copy-on-write paging as a given and
allocate a large sigaltstack, knowing it probably won't be used.

We can go back in there and try to optimize the algorithm for stack slimness a
little further.  We have to try to characterize how much this high water mark
depends on the particulars of the trace. Does it vary with stack depth or with
# of shared objects, or is it fixed constant overhead? Since it doesn't
allocate it might be fixed cost but I don't know. That sort of thing.