Uploaded image for project: 'Compass '
  1. Compass
  2. COMPASS-7292

Run accuracy tests in Compass nightly vs cloud dev

    • Type: Icon: Task Task
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • No version
    • Affects Version/s: None
    • Component/s: GAI
    • None
    • 3
    • Not Needed
    • Iteration Minmi, Iteration Nodosaurus

      We'd like to know when prompt changes or other regressions impact the accuracy of the generative ai results. To do this we'll run the accuracy tests that were recently added to Compass (scripts/ai-accuracy-tests.js) on a nightly basis. They should fail under a certain threshold. 
      Currently these tests might be synchronous, we might need to parallelize if its really slow.

            Assignee:
            rhys.howell@mongodb.com Rhys Howell
            Reporter:
            rhys.howell@mongodb.com Rhys Howell
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: