Scoring Methodology

For each challenge, the model receives a prompt describing the task, the index mapping, and sample documents. It must respond with a valid JSON Elasticsearch query body.

The query is executed against the backend, and the result is validated:

• Search challenges: Checks which expected documents were returned. Points for correct hits, penalties for false positives (-15 each).
• Aggregation challenges: Checks aggregation structure, bucket counts, metric values, and sub-aggregation nesting.
• Sort/order challenges: Additional points for correct result ordering.
• Partial credit: Models get proportional points for partially correct answers (e.g., finding 3 of 4 expected documents).

If the model's response is not valid JSON, or if the query throws an execution error, the score is 0.

The correctness score is multiplied by a speed factor based on the model's response latency. This rewards models that are both accurate and fast.

Final challenge score = min(100, round(correctness_score × speed_multiplier))

• Total score = sum of all challenge scores
• Max possible = 53 challenges × 100 = 5,300
• Percentage = total / max × 100
• Leaderboard rank = sorted by percentage (ties broken by avg latency)

Scores are also broken down by domain and difficulty level:

Models earn badges based on their performance:

• All models receive the same prompt for each challenge (system prompt + challenge description + mapping + sample docs)
• Temperature is set to 0 for all models (deterministic output)
• The simulated backend is deterministic — same query always produces the same result
• Latency is measured end-to-end (API call round-trip), not just model inference time
• The leaderboard shows the best run per model if multiple runs are submitted
• Anyone can reproduce results by running the same benchmark: npx elastic-quest benchmark -m openrouter:model-name -v