Would an EvalPort-format export of BFCL be welcome in EvalPort's benchmarks/?
Hi Gorilla/BFCL team — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results. Its benchmarks/ directory already converts 14 public benchmarks (GSM8K, ARC, HumanEval, SQuAD, etc.) into validated EvalPort suites, each checked against the real validator in CI.
I installed bfcl-eval (2026.3.23) and read real files rather than guessing, e.g. bfcl_eval/data/possible_answer/BFCL_v4_simple_java.json:
{"id": "simple_java_1", "ground_truth": [{"SQLCompletionAnalyzer.makeProposalsFromObject": {"object": ["Customers"], "useShortName": [true], "params": [{"limit": [50], "schemaFilter": ["public"]}]}}]}and a matching question file with {"id", "question": [[{"role","content"}]], "function": [{"name","description","parameters"}]}.
That maps onto EvalPort cleanly: question + function -> TestCase.input, ground_truth -> TestCase.expected_output, graded with a code grader doing the same AST/param match BFCL already defines.
Would your maintainers be open to an EvalPort-format export of BFCL (or a subset — e.g. the non-executable simple/multiple/parallel categories) living in EvalPort's benchmarks/, with attribution and license preserved, and a link back to this repo and the leaderboard? Happy to do the conversion work and open a PR here first if you'd rather review the mapping before anything lands in EvalPort — whichever you prefer. No worries if this isn't useful for you right now.
Source: ShishirPatil/gorilla