Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-67300

Increase in page eviction queue as well as page eviction failures

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.2
    • Component/s: None
    • Environment:
      VM details for primary and secondary: 8 vCPU, 64 GB RAM, 7.8TB SSD
      OS: Ubuntu 16.04.6 LTS
      MongoDB: 3.6.2
    • ALL
    • Hide

      Not sure if this would reproduce this issue exactly,

      1. Crash the secondary with a segmentation fault.
      2. See the impact on primary.
      Show
      Not sure if this would reproduce this issue exactly, Crash the secondary with a segmentation fault. See the impact on primary.

      Cluster state

      • Configuration
        • 1 Primary
        • 2 Secondaries (one secondary does not participate in voting and acts like a standby)
        • 1 Arbiter
      • Data size
        • 1.9TB of data in one collection
        • 30GB index size

      Incident timeline (all times are in IST)

      Incident date: 29th May 2022

      • 12:12: One of the secondaries crashed due to a segmentation fault.
      • 12:12: Increase in Pages queued for eviction along with errors in page eviction
      • 12:12 - 14:20: Increase in pages queued for eviction as well as page eviction errors still ongoing with some occasional drops in the increase. These drops correspond to checkpoint activities.
      • 14:30 - 14:52: Secondary comes back up and starts syncing from primary. Again seeing increase in page queued for eviction as well as page eviction errors.
      • 14:52: All metrics normalized.
      • 16:00 - now: Increase in page eviction queues and page eviction errors, not subsided yet. Happening as of now.

       

        1. 2022-05-29-mongo-01.log
          17.57 MB
        2. 2022-05-29-mongo-02.log
          2.16 MB
        3. metrics.2022-06-27T06-48-02Z-00000
          9.96 MB
        4. Screenshot 2022-06-15 at 17-58-14 MongoDB Overview - Grafana.png
          Screenshot 2022-06-15 at 17-58-14 MongoDB Overview - Grafana.png
          148 kB

            Assignee:
            chris.kelly@mongodb.com Chris Kelly
            Reporter:
            tejas.jadhav@gojek.com Tejas Jadhav
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: