Uploaded image for project: 'C Driver'
  1. C Driver
  2. CDRIVER-1233

Crash after Kerberos plugin cleanup runs twice

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 1.3.5
    • Component/s: None
    • None
    • Environment:
      Mac OSX 10.11.4, Apple LLVM version 7.3.0 (clang-703.0.31), Mac Ports cyrus-sasl2 @2.1.26_6+kerberos, Mac Ports kerberos5 @1.14.2 (libkrb5.3.3)

      Summary

      In some configurations, the following sequence leads to the MIT Kerberos5 library being destroyed twice during process exit:

      1. process starts, loads libmongoc, which registers its auto-destructor
      2. libmongoc loads Cyrus, which loads its Kerberos plugin, which registers the Kerberos plugin auto-destructor
      3. main() exits
      4. the Kerberos plugin's auto-destructor runs
      5. libmongoc's auto-destructor calls sasl_client_done, which closes the Kerberos plugin shared lib
      6. closing the Kerberos plugin shared lib triggers its auto-destructor again

      The solution for C programs is to configure the driver with ./configure --enable-automatic-init-and-cleanup=no and ensure they call mongoc_init and mongoc_cleanup explicitly. In version 2.0, the driver will remove the automatic init and cleanup feature.

      Original report:

      I found an assertion in kerberos originating in the C driver while testing the CXX driver master branch (commit 8dc0b5b). Backtrace follows:

      $ lldb ./src/mongocxx/test/test_driver
      
      
      (lldb) target create "./src/mongocxx/test/test_driver"
      Current executable set to './src/mongocxx/test/test_driver' (x86_64).
      (lldb) r
      Process 47444 launched: './src/mongocxx/test/test_driver' (x86_64)
      ===============================================================================
      All tests passed (903 assertions in 60 test cases)
      
      Assertion failed: (r == 0), function k5_mutex_lock, file ../../include/k5-thread.h, line 384.
      Process 47444 stopped
      * thread #1: tid = 0x9686ee, 0x00007fff97fdaf06 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main
      -thread', stop reason = signal SIGABRT
          frame #0: 0x00007fff97fdaf06 libsystem_kernel.dylib`__pthread_kill + 10
      libsystem_kernel.dylib`__pthread_kill:
      ->  0x7fff97fdaf06 <+10>: jae    0x7fff97fdaf10            ; <+20>
          0x7fff97fdaf08 <+12>: movq   %rax, %rdi
          0x7fff97fdaf0b <+15>: jmp    0x7fff97fd57cd            ; cerror_nocancel
          0x7fff97fdaf10 <+20>: retq
      (lldb) bt
      * thread #1: tid = 0x9686ee, 0x00007fff97fdaf06 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main
      -thread', stop reason = signal SIGABRT
        * frame #0: 0x00007fff97fdaf06 libsystem_kernel.dylib`__pthread_kill + 10
          frame #1: 0x00007fff922b94ec libsystem_pthread.dylib`pthread_kill + 90
          frame #2: 0x00007fff8cb626e7 libsystem_c.dylib`abort + 129
          frame #3: 0x00007fff8cb29df8 libsystem_c.dylib`__assert_rtn + 321
          frame #4: 0x0000000100879cd9 libkrb5support.1.1.dylib`krb5int_key_delete + 153
          frame #5: 0x0000000100837c18 libgssapi_krb5.2.2.dylib`gss_krb5int_lib_fini + 26
          frame #6: 0x000000010082b037 libgssapi_krb5.2.2.dylib`gssint_mechglue_fini + 40
          frame #7: 0x00007fff5fc1329c dyld`ImageLoaderMachO::doTermination(ImageLoader::LinkContext const&) + 212
          frame #8: 0x00007fff5fc01fee dyld`dyld::runImageStaticTerminators(ImageLoader*) + 206
          frame #9: 0x00007fff5fc04d19 dyld`dyld::garbageCollectImages() + 720
          frame #10: 0x00007fff5fc0c3df dyld`dlclose + 134
          frame #11: 0x00007fff9b9d874d libdyld.dylib`dlclose + 61
          frame #12: 0x00000001004938a1 libsasl2.dylib`_sasl_done_with_plugins + 46
          frame #13: 0x000000010048b0cf libsasl2.dylib`sasl_common_done + 78
          frame #14: 0x00000001004889fd libsasl2.dylib`sasl_client_done + 92
          frame #15: 0x0000000100711062 libmongoc-1.0.0.dylib`_mongoc_do_cleanup + 18 at mongoc-init.c:130
          frame #16: 0x00007fff922b6bf6 libsystem_pthread.dylib`__pthread_once_handler + 65
          frame #17: 0x00007fff9d026fc4 libsystem_platform.dylib`_os_once + 41
          frame #18: 0x00007fff922b6b95 libsystem_pthread.dylib`pthread_once + 57
          frame #19: 0x000000010071103b libmongoc-1.0.0.dylib`mongoc_cleanup + 27 at mongoc-init.c:150
          frame #20: 0x000000010071108e libmongoc-1.0.0.dylib`_mongoc_init_dtor + 14 at mongoc-init.c:170
          frame #21: 0x00007fff5fc1329c dyld`ImageLoaderMachO::doTermination(ImageLoader::LinkContext const&) + 212
          frame #22: 0x00007fff5fc0218b dyld`dyld::runAllStaticTerminators(void*) + 67
          frame #23: 0x00007fff8cb6346b libsystem_c.dylib`__cxa_finalize_ranges + 345
          frame #24: 0x00007fff8cb6376f libsystem_c.dylib`exit + 55
          frame #25: 0x00007fff9b9d95b4 libdyld.dylib`start + 8
      (lldb)
      

      During libmongoc compilation, I did notice numerous warnings of the type: clang: warning: argument unused during compilation: '-pthread'

      UPDATE: Moved libmongoc configuration output from "environment":

      libbson 1.3.5 was configured with the following options:
      
      Build configuration:
        Enable debugging (slow)                          : yes
        Enable extra alignment (required for 1.0 ABI)    : yes
        Compile with debug symbols (slow)                : yes
        Enable GCC build optimization                    : no
        Enable automatic binary hardening                : yes
        Code coverage support                            : no
        Cross Compiling                                  : no
        Big endian                                       : no
        Link Time Optimization (experimental)            : no
      
      Documentation:
        man                                              : no
        HTML                                             : no
      
      
      libmongoc 1.3.5 was configured with the following options:
      
      Build configuration:
        Enable debugging (slow)                          : yes
        Compile with debug symbols (slow)                : yes
        Enable GCC build optimization                    : no
        Enable automatic binary hardening                : yes
        Enable automatic init and cleanup                : yes
        Code coverage support                            : no
        Cross Compiling                                  : no
        Fast counters                                    : no
        Shared memory performance counters               : yes
        SASL                                             : sasl2
        SSL                                              : yes
        Libbson                                          : bundled
      
      Documentation:
        man                                              : no
        HTML                                             : no
      

      There's this warning that libraries should not call sasl_done:

      Libraries should not call sasl_done(). Applications should exercise caution when calling sasl_done() to avoid interference with any libraries that might be using libsasl.

            Assignee:
            jesse@mongodb.com A. Jesse Jiryu Davis
            Reporter:
            david.golden@mongodb.com David Golden
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: