Issues

• Proposal: agent-evaluation-openeval-adapter for portable Test/TestSuite/TestResult interchange

  awslabs/agent-evaluation#122 · Filter by author adhabnr-ux opened on Aug 23, 2026

• Add Target for AgentCore Runtime

  awslabs/agent-evaluation#113 · Filter by author athewsey opened on Aug 21, 2025

• Upgrade to Python 3.10+ (3.9 broken anyway)

  awslabs/agent-evaluation#112 · Filter by author athewsey opened on Aug 21, 2025

• Fails to run on windows

  awslabs/agent-evaluation#110 · Filter by author kylebechtel77 opened on Apr 2, 2025

• LLM hallucinates when converting the 'steps' into actual prompts for the Evaluator

  awslabs/agent-evaluation#107 · Filter by author chrissandico opened on Mar 28, 2025

• Invalid type for parameter inputText, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

  awslabs/agent-evaluation#98 · Filter by author pipelineburst opened on Jan 24, 2025
  5

• Documentation for Knowledgebase has additional character leading to incorrect target

  awslabs/agent-evaluation#93 · Filter by author markcole99 opened on Nov 27, 2024

• [Feature Request] Guardrail option for Bedrock KB target (maybe SageMaker too?)

  awslabs/agent-evaluation#83 · Filter by author athewsey opened on Sep 18, 2024

• botocore.eventstream.InvalidHeadersLength:

  bug
  awslabs/agent-evaluation#82 · Filter by author dc2tom opened on Aug 21, 2024
  

• throttlingException when calling the InvokeAgent operation

  bug
  awslabs/agent-evaluation#64 · Filter by author tonykchen opened on May 23, 2024
  1

• ValidationException when calling the InvokeAgent operation

  bug
  awslabs/agent-evaluation#63 · Filter by author tonykchen opened on May 23, 2024
  2

• Save run artifacts to S3

  enhancementgood first issue
  awslabs/agent-evaluation#54 · Filter by author tonykchen opened on May 8, 2024
  2

• Global test cases

  enhancement
  awslabs/agent-evaluation#31 · Filter by author tonykchen opened on Apr 12, 2024

• Add exceptions to trace

  enhancement
  awslabs/agent-evaluation#30 · Filter by author tonykchen opened on Apr 12, 2024

• Formalize test plan schema

  enhancementgood first issue
  awslabs/agent-evaluation#21 · Filter by author sharonxiaohanli opened on Apr 4, 2024
  3

• Add support for Claude 3 Haiku

  enhancementgood first issue
  awslabs/agent-evaluation#13 · Filter by author tonykchen opened on Apr 4, 2024
  1