Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29507

Task not marked as failure if resmoke.py fails to start fixtures

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Critical - P2 Critical - P2
    • 3.5.9
    • Affects Version/s: None
    • Component/s: Testing Infrastructure
    • None
    • Fully Compatible
    • ALL
    • TIG 2017-06-19

      Happened in one of my patch builds. All mongod's in this patch fail to start with an error message like this:

      [MongoDFixture:job2] Starting mongod on port 20500...
      ./mongod --setParameter enableTestCommands=1 --dbpath /data/db/job2/resmoke --nopreallocj --storageEngine wiredTiger --wiredTigerCacheSizeGB 1 --bind_ip_all --port 20500
      [MongoDFixture:job2] mongod started on port 20500 with pid 6508.
      [MongoDFixture:job2] src/third_party/timelib-2017.05beta3/parse_tz.c:152:16: runtime error: left shift of 128 by 24 places cannot be represented in type 'int'
      [MongoDFixture:job2]     #0 0x556486bc1ca2 in read_transistions /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/third_party/timelib-2017.05beta3/parse_tz.c:152:16
      [MongoDFixture:job2]     #1 0x556486bc1ca2 in timelib_parse_tzfile /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/third_party/timelib-2017.05beta3/parse_tz.c:466
      [MongoDFixture:job2]     #2 0x556486bbe529 in mongo::DateTimeSupport::loadTimeZoneInfo(std::unique_ptr<_timelib_tzdb, mongo::DateTimeSupport::TimeZoneDBDeleter>) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/query/datetime/date_time_support.cpp:101:23
      [MongoDFixture:job2]     #3 0x556486bbe3c9 in mongo::DateTimeSupport::DateTimeSupport() /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/query/datetime/date_time_support.cpp:71:5
      [MongoDFixture:job2]     #4 0x556486873fc7 in std::_MakeUniq<mongo::DateTimeSupport>::__single_object std::make_unique<mongo::DateTimeSupport>() /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/bits/unique_ptr.h:765:34
      [MongoDFixture:job2]     #5 0x556486873fc7 in mongo::_mongoInitializerFunction_LoadTimeZoneDB(mongo::InitializerContext*) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/query/datetime/init_timezone_data.cpp:59
      [MongoDFixture:job2]     #6 0x556485762a3f in std::_Function_handler<mongo::Status (mongo::InitializerContext*), mongo::Status (*)(mongo::InitializerContext*)>::_M_invoke(std::_Any_data const&, mongo::InitializerContext*&&) /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/functional:1856:9
      [MongoDFixture:job2]     #7 0x5564871bca7a in std::function<mongo::Status (mongo::InitializerContext*)>::operator()(mongo::InitializerContext*) const /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/functional:2267:14
      [MongoDFixture:job2]     #8 0x5564871bc1c0 in mongo::Initializer::execute(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) const /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/base/initializer.cpp:57:22
      [MongoDFixture:job2]     #9 0x5564871bc5e4 in mongo::runGlobalInitializers(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/base/initializer.cpp:70:35
      [MongoDFixture:job2]     #10 0x5564871bc79e in mongo::runGlobalInitializers(int, char const* const*, char const* const*) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/base/initializer.cpp:89:12
      [MongoDFixture:job2]     #11 0x55648574f105 in mongoDbMain(int, char**, char**) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/db.cpp:1074:21
      [MongoDFixture:job2]     #12 0x55648574f105 in main /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/db.cpp:779
      [MongoDFixture:job2]     #13 0x7f88f4b4782f in __libc_start_main /build/glibc-9tT8Do/glibc-2.23/csu/../csu/libc-start.c:291
      [MongoDFixture:job2]     #14 0x55648572fdc8 in _start (/data/mci/f8008ddfb24550f379e521aaaaa119a5/src/mongod+0x28bfdc8)
      [MongoDFixture:job2] 
      [MongoDFixture:job2] mongod on port 20500 was expected to be running in _do_teardown(), but wasn't. Exited with code 1.
      

      But the task was not marked as a failure, instead thinking that no tests were run:

       [2017/06/07 16:09:03.485] [executor] 2017-06-07T16:09:03.484+0000 Shuffling order of tests for js_tests in suite aggregation. The seed is 383194046260.
       [2017/06/07 16:09:03.523] [executor:js_test] 2017-06-07T16:09:03.523+0000 Writing output of job #0 to https://logkeeper.mongodb.org/build/0662ee75067fe7d2d232f8ca185354ff.
       [2017/06/07 16:09:03.558] [executor:js_test] 2017-06-07T16:09:03.558+0000 Writing output of job #1 to https://logkeeper.mongodb.org/build/86b96d4afed95e2a0ffb66ebb7cd48f3.
       [2017/06/07 16:09:03.587] [executor:js_test] 2017-06-07T16:09:03.587+0000 Writing output of job #2 to https://logkeeper.mongodb.org/build/5430ad712810ffd63287eebd20b10e9a.
       [2017/06/07 16:09:03.617] [executor:js_test] 2017-06-07T16:09:03.617+0000 Writing output of job #3 to https://logkeeper.mongodb.org/build/3e456106bb72af96973a236912edc9e0.
       [2017/06/07 16:09:03.685] [executor:js_test] 2017-06-07T16:09:03.685+0000 Writing output of job #4 to https://logkeeper.mongodb.org/build/7f65119b4b8390edb91f734db12c6f6b.
       [2017/06/07 16:09:03.685] [executor:js_test] 2017-06-07T16:09:03.685+0000 Starting execution of js_tests...
       [2017/06/07 16:09:04.333] [executor:js_test] 2017-06-07T16:09:04.332+0000 Encountered an error while waiting for MongoDFixture (Job #0) to be ready
       [2017/06/07 16:09:04.333] Traceback (most recent call last):
       [2017/06/07 16:09:04.333]   File "/data/mci/b1773361f21e55b5703554055181ff2d/src/buildscripts/resmokelib/testing/executor.py", line 128, in _setup_fixtures
       [2017/06/07 16:09:04.333]     job.fixture.await_ready()
       [2017/06/07 16:09:04.333]   File "/data/mci/b1773361f21e55b5703554055181ff2d/src/buildscripts/resmokelib/testing/fixtures/standalone.py", line 100, in await_ready
       [2017/06/07 16:09:04.333]     " unexpectedly with code %d." % (self.port, exit_code))
       [2017/06/07 16:09:04.333] ServerFailure: Could not connect to mongod on port 20000, process ended unexpectedly with code 1.
       [2017/06/07 16:09:04.333] [executor:js_test] 2017-06-07T16:09:04.333+0000 Teardown of MongoDFixture (Job #0) was not successful.
       [2017/06/07 16:09:04.333] [executor:js_test] 2017-06-07T16:09:04.333+0000 Teardown of MongoDFixture (Job #1) was not successful.
       [2017/06/07 16:09:04.333] [executor:js_test] 2017-06-07T16:09:04.333+0000 Teardown of MongoDFixture (Job #2) was not successful.
       [2017/06/07 16:09:04.334] [executor:js_test] 2017-06-07T16:09:04.334+0000 Teardown of MongoDFixture (Job #3) was not successful.
       [2017/06/07 16:09:04.334] [executor:js_test] 2017-06-07T16:09:04.334+0000 Teardown of MongoDFixture (Job #4) was not successful.
       [2017/06/07 16:09:04.334] [resmoke] 2017-06-07T16:09:04.334+0000 ================================================================================
       [2017/06/07 16:09:04.334] [resmoke] 2017-06-07T16:09:04.334+0000 Summary of aggregation suite: No tests ran.
       [2017/06/07 16:09:04.334] Suite did not run any tests.
       [2017/06/07 16:09:04.948] [resmoke] 2017-06-07T16:09:04.947+0000 resmoke.py invocation: buildscripts/resmoke.py --suites=aggregation --storageEngine=mmapv1 --jobs=5 --shuffle --continueOnFailure --storageEngineCacheSizeGB=1 --log=buildlogger --staggerJobs=on --tagFile=etc/test_lifecycle.yml --taskId=mongodb_mongo_master_ubuntu1604_debug_ubsan_aggregation_10e3bc5992cb537a1a23722143aacc101431d95a_17_06_07_15_22_22 --includeWithAnyTags=unreliable,unreliable|aggregation,unreliable|aggregation|ubuntu1604-debug-ubsan,unreliable|aggregation|ubuntu1604-debug-ubsan|ubuntu1604-build --reportFailureStatus=silentfail --reportFile=report_unreliable.json
       [2017/06/07 16:09:04.968] [resmoke] 2017-06-07T16:09:04.968+0000 YAML configuration of suite aggregation
       [2017/06/07 16:09:04.968] test_kind: js_test
       [2017/06/07 16:09:04.968] selector:
       [2017/06/07 16:09:04.968]   roots:
       [2017/06/07 16:09:04.968]   - jstests/aggregation/*.js
       [2017/06/07 16:09:04.968]   - jstests/aggregation/bugs/*.js
       [2017/06/07 16:09:04.968]   - jstests/aggregation/expressions/*.js
       [2017/06/07 16:09:04.968]   - jstests/aggregation/sources/*/*.js
       [2017/06/07 16:09:04.968]   - jstests/aggregation/variables/*.js
       [2017/06/07 16:09:04.968] executor:
       [2017/06/07 16:09:04.968]   config:
       [2017/06/07 16:09:04.968]     shell_options:
       [2017/06/07 16:09:04.968]       readMode: commands
       [2017/06/07 16:09:04.968]   fixture:
       [2017/06/07 16:09:04.968]     class: MongoDFixture
       [2017/06/07 16:09:04.968]     mongod_options:
       [2017/06/07 16:09:04.968]       set_parameters:
       [2017/06/07 16:09:04.968]         enableTestCommands: 1
       [2017/06/07 16:09:04.968]   hooks:
       [2017/06/07 16:09:04.968]   - class: ValidateCollections
       [2017/06/07 16:09:04.968]   - class: CleanEveryN
       [2017/06/07 16:09:04.968]     n: 20
       [2017/06/07 16:09:04.968] logging:
       [2017/06/07 16:09:04.968]   executor:
       [2017/06/07 16:09:04.969]     format: '[%(name)s] %(asctime)s %(message)s'
       [2017/06/07 16:09:04.969]     handlers:
       [2017/06/07 16:09:04.969]     - class: logging.StreamHandler
       [2017/06/07 16:09:04.969]   fixture:
       [2017/06/07 16:09:04.969]     format: '[%(name)s] %(message)s'
       [2017/06/07 16:09:04.969]     handlers:
       [2017/06/07 16:09:04.969]     - class: buildlogger
       [2017/06/07 16:09:04.969]   tests:
       [2017/06/07 16:09:04.969]     format: '[%(name)s] %(asctime)s %(message)s'
       [2017/06/07 16:09:04.969]     handlers:
       [2017/06/07 16:09:04.969]     - class: buildlogger
       [2017/06/07 16:09:04.969] [executor] 2017-06-07T16:09:04.968+0000 Shuffling order of tests for js_tests in suite aggregation. The seed is 383194046701.
       [2017/06/07 16:09:04.969] [executor] 2017-06-07T16:09:04.968+0000 Skipping js_tests, no tests to run
       [2017/06/07 16:09:04.969] [resmoke] 2017-06-07T16:09:04.968+0000 ================================================================================
       [2017/06/07 16:09:04.969] [resmoke] 2017-06-07T16:09:04.968+0000 Summary of aggregation suite: No tests ran.
       [2017/06/07 16:09:04.969] Suite did not run any tests.
       [2017/06/07 16:09:05.151] Task completed - SUCCESS.
      

            Assignee:
            eddie.louie Eddie Louie
            Reporter:
            charlie.swanson@mongodb.com Charlie Swanson
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: