__conn_load_extension() calls an entry function (by default wiredtiger_extension_init), which can return an error because in the case of the compressor functions, it calls conn->add_compressor() which allocates memory and can fail.
I suspect wiredtiger_extension_init (and init functions in general), should return errors, and __conn_load_extension() should check the return of the entry function.
Sorry to dump this on you, Michael, but I'm not 100% sure if the entry function should just move before the mutex and add to connection's list of libraries, or if the library needs to be linked before the entry function is called?