Allow custom OpenAI ENDPOINT & Model Names - e.g. Sambanova API Cloud

Author: soufianechamiCreated Feb 10, 2025Updated Feb 23, 2026

I am trying to use Sambanova API which is compatible with OpenAI SDK. Would love to see how fast inference can benefit here and also use open source LLMs.

Below is an example of Sambanova API call:

python

import os
import openai

client = openai.OpenAI(
    api_key=os.environ.get("SAMBANOVA_API_KEY"),
    base_url="https://api.sambanova.ai/v1",
)

response = client.chat.completions.create(
    model="Meta-Llama-3.1-8B-Instruct",
    messages=[{"role":"system","content":"You are a helpful assistant"},{"role":"user","content":"Hello"}],
    temperature=0.1,
    top_p=0.1
)

print(response.choices[0].message.content)

Source: microsoft/data-formulator

View original on GitHub View discussion on GitHub