LightRAG evaluation with reproduce gives me very different results with local LLMs

Do you need to ask a question?

• I have searched the existing question and discussions and this question is not already answered.
• I believe this is a legitimate question, not just a bug or feature request.

Your Question

Hi everyone, I tryed to implement several times an evaluation of the LightRAG pipeline using the method cited in the paper and found in the reproduce folder, but i can't get the same results expected in the paper.

The following are my results overall: LIGHTRAG EVALUATION WITH REPRODUCE WORKFLOW

	Agriculture		Legal		Mix
	Naive	Hybrid	Naive	Hybrid	Naive	Hybrid
Comprehensiveness	57,6%	42,4%	69,6%	30,4%	85,6%	14,4%
Diversity	48,8%	51,2%	61,6%	38,4%	78,4%	21,6%
Empowerment	48,8%	51,2%	68,0%	32,0%	83,2%	16,8%
Overall	54,4%	45,6%	68,8%	31,2%	83,2%	16,8%
	Naive	Mix	Naive	Mix	Naive	Mix
Comprehensiveness	49,6%	50,4%	72,8%	27,2%	50,4%	49,6%
Diversity	45,6%	56,8%	66,4%	33,6%	52,0%	48,0%
Empowerment	43,2%	56,8%	72,0%	28,0%	48,0%	52,0%
Overall	47,2%	52,8%	72,8%	27,2%	48,8%	51,2%

LIGHTRAG EVALUATION WITH RAGAS

	Agriculture			Legal			Mix
	Naive	Hybrid	Mix	Naive	Hybrid	Mix	Naive	Hybrid	Mix
Faithfullness	99,4%	88,9%	87,7%	98,9%	82,9%	82,9%	98,8%	21,3%	97,7%
Context Relevance	58,8%	55,8%	55,6%	22,4%	25,6%	24,0%	75,0%	0%	75,8%
Response Relevance	82,5%	82,4%	84,1%	42,8%	47,11%	47,9%	88,9%	63,0%	89,4%

For this study i implemented a slightly different reproduce folder (univpm_reproduce in the link down below) leveraging ollama hosting gemma4_31b for entity extraction and generation and bge3 large for embedding. If anyone from the community is interested in my work you can find in my public fork. I hope this work can also help someone facing similar issue. I would really appreciate some feedback and

Thank you very much

https://github.com/s-osimi-univpm/LightRAG/tree/reproduce-investigation

Additional Context

No response