Scorecard - AI Eval Platform

Can I test my agent against scenarios I create myself, or does Scorecard only use predefined test cases?

Scorecard lets you create and customize scenarios for testing. The platform includes a validated metric library with industry benchmarks, but you can build your own test cases tailored to how your agent will actually be used in production.

How does Scorecard handle different versions of my agent's prompts during testing?

Scorecard includes prompt versioning built in, so you can test multiple prompt variations against the same scenarios and compare their results. This lets you identify which versions perform best before deploying to production.

What metrics can I measure when testing my agent?

You can use metrics from Scorecard's validated library, which includes industry benchmarks, or define your own custom metrics based on what matters to your business. The platform provides actionable insights that help you understand why an agent succeeded or failed in a given scenario.

Do I need a development environment set up to deploy agents through Scorecard?

No. Scorecard includes a Playground for rapid experimentation and can manage agent deployment to production without requiring an IDE, so teams can iterate and ship without being blocked by engineering infrastructure.