AgentEval is preview software (see DISCLAIMER.md). Security fixes are released in new versions, so use the latest published version:

Once AgentEval reaches stable release (1.0.0), we will support:

• Current major version (e.g., 1.x)
• Previous major version (e.g., 0.x) for 6 months after a new major release

Please do not report security vulnerabilities through public GitHub issues.

If you discover a security vulnerability in AgentEval, please report it privately:

Send details to: joslat@gmail.com

Include:

• Description of the vulnerability
• Steps to reproduce
• Potential impact
• Suggested fix (if any)

1. Acknowledgment: We'll acknowledge your report within 48 hours
2. Investigation: We'll investigate and validate the issue within 7 days
3. Fix: We'll develop a fix and release a security patch as soon as possible
4. Disclosure: We'll coordinate disclosure with you after the fix is released

1. Private Fix: Develop fix in private repository
2. CVE Assignment: Request CVE if applicable
3. Security Advisory: Publish GitHub Security Advisory
4. Patch Release: Release patched version
5. Public Disclosure: Announce fix in release notes

We follow a coordinated disclosure model:

• 90 days: Standard disclosure timeline after initial report
• Extended timeline: Available for complex vulnerabilities requiring vendor coordination
• Early disclosure: With reporter's agreement if exploit is public

AgentEval integrates with AI services (Azure OpenAI, OpenAI, etc.). Never commit API keys:

// ❌ DON'T: Hard-code credentials
var client = new AzureOpenAIClient(
    new Uri("https://my-endpoint.openai.azure.com"),
    new AzureKeyCredential("sk-XXXX...")); // DON'T DO THIS!

// ✅ DO: Use environment variables or configuration
var endpoint = new Uri(Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT")!);
var credential = new AzureKeyCredential(Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY")!);
var client = new AzureOpenAIClient(endpoint, credential);

When using snapshot testing, ensure snapshots don't contain sensitive data:

var options = new SnapshotOptions
{
    // Scrub sensitive data
    ScrubPatterns = new List<(Regex Pattern, string Replacement)>
    {
        (new Regex(@"sk-[a-zA-Z0-9]+"), "[API_KEY]"),
        (new Regex(@"Bearer [a-zA-Z0-9_\-\.]+"), "[TOKEN]"),
        (new Regex(@"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"), "[EMAIL]")
    }
};

Use secrets management in CI/CD:

# GitHub Actions
- name: Run Tests
  env:
    AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
    AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
  run: dotnet test

• Don't use production data in tests
• Don't commit test data with PII
• Use synthetic/anonymized data

None currently identified.

AgentEval includes automated security scanning in CI/CD:

• Dependency Vulnerability Scanning: Checks for known vulnerabilities in NuGet packages
• DevSkim Static Analysis: Microsoft's security-focused static analysis tool
• Secret Detection: Scans for hardcoded secrets and API keys
• Path Traversal Protection: File path inputs are validated to prevent directory traversal attacks

The project undergoes security reviews:

• Weekly automated dependency scans
• Manual security audits before major releases
• Community-driven vulnerability reporting via GitHub Security Advisories

Check Security Advisories for updates.

AgentEval includes:

• No telemetry - No telemetry, analytics, crash reporting or update check; no usage data is sent to the authors or to anyone else
• Opt-in network use - Besides the model calls you configure, AgentEval opens network connections only for the features listed in PRIVACY.md (benchmark pack downloads, the live package-registry check, the decision-model judge, the Copilot Studio connector), each of which stays off until you turn it on
• Local output - Results, transcripts and reports are written to local files and are not uploaded
• Transparent - Open source under MIT license

AgentEval depends on Microsoft .NET libraries and on third-party packages, including packages that are not from Microsoft and packages under licences other than MIT. THIRD-PARTY-NOTICES.md lists the packages in each published package, with their versions and licences; the same file ships inside the AgentEval and AgentEval.Cli NuGet packages. Directory.Packages.props pins the versions. The weekly security scan checks them for known vulnerabilities (dotnet list package --vulnerable --include-transitive).

For security questions that aren't sensitive, open a GitHub Discussion.

For private security matters, email: joslat@gmail.com

Last updated: October 2026