-
Type: Task
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: GAI
Update the accuracy test result format to:
- Add the "Test Id” field to provide a unique identifier for each running test.
- Instead of using "✓” and”✗" symbols, we will use boolean values to indicate if a test passed or failed.
- Add the "Response Time MS" field to track time elapsed.
- Add the "Failure" field to store a failure reason or null for successful tests.
- Would it be interesting to store user stages and operators? Even if the accuracy is 1, the generated query might not follow the best practices. We could also add the "expectedStages" and "expectedOperators" to the test runner. For example for the $unwind test case we will expect to have the $unwind stage and not something else.
- Since we explore the accuracy improvements by using sample documents, we want tests to account for those as well. We add the "With Sample Documents" filed of the boolean type for it.
Tech design: https://docs.google.com/document/d/1t8vPywHOwmr1f1Vktpv5NsTP-iK3wkR5EbCX7syoCJo/edit#heading=h.37i79gbf3cse