Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-12698

mongod/mongos ought to collect host-level info & serve it via commands

    • Type: Icon: New Feature New Feature
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Diagnostics
    • Fully Compatible

      Very many MongoDB, Inc. "field" engagements (support, consulting, onboarding, and pre-sales interactions) require collecting information about a MongoDB deployment in order to understand, debug, tune, etc.

      A quantity of the information that is desirable in these engagements are host-level properties (hardware details, OS version, library versions, OS settings, etc.) At the moment, collecting this information is easy for an individual host, but wrangling the information for a whole cluster becomes tedious in proportion to the number of hosts running MongoDB server processes. In consequence, customer interactions can become delayed and sidetracked on these data collection steps.

      Proposal: (1) add code to mongod and mongos processes to collect the kinds of host-level information the MongoDB, Inc. field organization often wants to gather, and (2) present that information to (suitably authorized) clients via some command (maybe serverStatus, maybe some other existing command, maybe a new command; I don't think it matters).

      Benefits: streamlining the collection of diagnostic information will improve efficiency of field activities (support, services, pre-sales, etc.)

      Disadvantages: more code, all of it OS-dependent. (However, it ought be possible to implement in a very modular, incremental fashion.)

      Documentation changes needed: if the feature is added, and if the feature is not considered "internal only", document the feature.

      Driver changes needed: if it's considered desirable to have driver bindings to the proposed command, design, implement, and document them.

      Example: a sample Bourne shell script the MongoDB support team has used for Linux follows. This is intended to enumerate the kinds of things we tend to gather, not as a concrete proposal for how to collect the info or present it back to users/client applications.

      (Note: the info this script collects includes static information that only needs to be gathered once per process lifetime (OS kernel version, libraries, CPU info) and dynamic state that will change while the process runs (e.g., free disk space, I/O statistics). If this request needs to be broken up into subtasks, I'd propose that a first pass would be to collect the only static info at server startup time; a second or subsequent pass could gather the dynamic info, perhaps on a periodic basis.)

      echo "MongoDB Diag" > /tmp/system.data
      mlog() {
          echo -e "\n\n\n $1 \n =======================" >> /tmp/system.data
          shift 1
          $@ 2>/dev/null >> /tmp/system.data
      }
      
      mlog uname uname -a                          
      mlog date date                              
      mlog blockdev /sbin/blockdev --report           
      mlog libraries ls -l /lib/libc.so*               
      mlog libraries /lib/libc.so*                    
      mlog lsb lsb_release -a                  
      mlog sysctl /sbin/sysctl -a                
      mlog ifconfig /sbin/ifconfig -a             
      mlog dmesg dmesg                        
      mlog lspci lspci -vvv                  
      mlog mount mount                      
      mlog whoami whoami                    
      mlog ulimit ulimit -a                
      mlog df df -h                   
      mlog cpuinfo cat /proc/cpuinfo      
      mlog meminfo cat /proc/meminfo     
      mlog swaps cat /proc/swaps      
      mlog modules cat /proc/modules   
      mlog vmstat cat /proc/vmstat   
      mlog top -b -n 10                    
      mlog iostat iostat -xtm 5 10               
      mlog rpcinfo /usr/sbin/rpcinfo -p          
      mlog scsidevices cat /sys/bus/scsi/devices/*/model 
      mlog proc/limits
      for i in `pgrep mongo`; do echo "PID: $i"; cat /proc/$i/cmdline; echo -e "\nLimits:\n"; cat /proc/$i/limits; echo; done >> /tmp/system.data
      mlog smartctl
      /usr/sbin/smartctl --scan | sed -e 's/#.*$//' | while read i; do /usr/sbin/smartctl --all $i; done >> /tmp/system.data
      mlog nr_requests
      for d in `find /sys -name nr_requests`; do (echo $d; cat $d); done  >> /tmp/system.data
      mlog read_ahead_kb
      for d in `find /sys -name read_ahead_kb`; do (echo $d; cat $d); done   >> /tmp/system.data
      mlog scheduler
      for d in `find /sys -name scheduler`; do (echo $d; cat $d); done   >> /tmp/system.data
      

            Assignee:
            kevin.pulo@mongodb.com Kevin Pulo
            Reporter:
            richard.kreuter Richard Kreuter (Inactive)
            Votes:
            5 Vote for this issue
            Watchers:
            24 Start watching this issue

              Created:
              Updated:
              Resolved: