I was alerted to missing log files on some of our production mongod servers yesterday, the cause was that log rotate was trying to use the pid file to send a signal to mongo to tell it the log files had rotated. But the pid file wasn't where we expected to find it.
Looking in the systemd service file it should be in /var/run/mongodb/mongod.pid but nothing appeared there, even after I created the folder which was missing. Then I added this to my mongod config file:
processManagement:
pidFilePath: /var/run/mongodb.pid
Then mongod failed to start up, as it didn't have permission to create. I touched the file and gave mongo permissions and it started, which should fix our issues.
I would say that this should not be fixed by admins, but should be correctly managed by mongod's systemd service file.
I would have expected a warning or error to appear in the logs in this situation, but nothing appears:
2018-11-22T12:25:35.418+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] MongoDB starting : pid=12211 port=27017 dbpath=/srv/mongodb 64-bit host=some.host.domain.com 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] db version v3.4.15 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] git version: 52e5b5fbaa3a2a5b1a217f5e647b5061817475f9 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] allocator: tcmalloc 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] modules: none 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] build environment: 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] distmod: ubuntu1604 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] distarch: x86_64 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] target_arch: x86_64 2018-11-22T12:25:35.427+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { RESTInterfaceEnabled: false, enabled: false } }, replication: { replSetName: "dev" }, storage: { dbPath: "/srv/mongodb", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { c ollectionConfig: { blockCompressor: "snappy" }, engineConfig: { journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongodb.log" } } 2018-11-22T12:25:35.454+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7511M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60, log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress), 2018-11-22T12:25:38.052+0000 I STORAGE [initandlisten] WiredTiger message [1542889538:52406][12211:0x7fbb84eded00], txn-recover: Main recovery loop: starting at 5591/38323200 2018-11-22T12:25:38.174+0000 I STORAGE [initandlisten] WiredTiger message [1542889538:174123][12211:0x7fbb84eded00], txn-recover: Recovering log 5591 through 5592 2018-11-22T12:25:38.232+0000 I STORAGE [initandlisten] WiredTiger message [1542889538:232811][12211:0x7fbb84eded00], txn-recover: Recovering log 5592 through 5592 2018-11-22T12:25:55.956+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2018-11-22T12:25:55.956+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 39153394 records totaling to 10711736188 bytes 2018-11-22T12:25:55.956+0000 I STORAGE [initandlisten] Sampling from the oplog between Nov 8 04:49:55:1e and Nov 22 12:25:26:2 to determine where to place markers for truncation 2018-11-22T12:25:55.957+0000 I STORAGE [initandlisten] Taking 1000 samples and assuming that each section of oplog contains approximately 391418 records totaling to 107085642 bytes
My server here is running 3.4.15 but I tested with 3.14.18 from repo.mongodb.org on my laptop and saw the same issue
- causes
-
SERVER-74345 mongodb-org-server 4.4.19, 5.0.15, 6.0.5 not starting after upgrading from older version (Debian, RPM Packages)
- Closed
- is related to
-
SERVER-27241 Cannot write pid file to /var/run/mongodb/mongod.pid: No such file or directory
- Closed