MakeMeSay get_content fails on its documented dict response type
Author: sylvesterkaczmarekCreated Aug 21, 2026Updated Aug 21, 2026
Describe the bug
evals.elsuite.make_me_say.utils.get_content() is annotated to accept either a dict or a CompletionResult:
def get_content(response: Union[dict, CompletionResult]) -> str:It handles CompletionResult through get_completions(), but the fallback assumes attribute-style OpenAI response objects:
return response.choices[0].message.contentA normal dictionary response such as:
{"choices": [{"message": {"content": "hello"}}]}therefore raises AttributeError even though dict is explicitly part of the public input type.
Expected behavior
Dictionary responses should return response["choices"][0]["message"]["content"]; existing CompletionResult and attribute-style response behavior should remain unchanged.
Proposed fix
Add an explicit dictionary branch and focused regression coverage for dict and CompletionResult inputs.
Source: openai/evals