create scons OOM retry tool

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major - P3
    • 7.0.0-rc0, 4.4.20, 5.0.16, 6.0.6
    • Affects Version/s: None
    • Component/s: None
    • None
    • Fully Compatible
    • v6.0, v5.0, v4.4, v4.2
    • 31
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      We frequently get failed builds in evergreen due to OOM and we have tried to manage this by limiting the concurrency for certain tasks, but this is not a very precise solution and has resulted in continued failed builds and maintenance. 

       

      We should create a scons tool which will modify link and compile jobs so that if the underlying command failed, we parse the stdout/stderr and identify the OOM error message depending on the compiler/platform. We then wait some random time, and try again up to some max number of retries.

            Assignee:
            Daniel Moody
            Reporter:
            Daniel Moody
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: