Calling the LLM API

Whether you use OpenAI, Anthropic, Google, or an open-source model behind an OpenAI-compatible endpoint, the request lifecycle is similar: authenticate, build the message list, call the API, handle the response, and deal gracefully with errors.

A basic call with the OpenAI SDK

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of Portugal?"},
    ],
    max_tokens=256,
    temperature=0.2,
)
print(response.choices[0].message.content)

Key parameters

• temperature (0–2): lower = more deterministic, higher = more creative. Use 0–0.3 for factual tasks.
• max_tokens: caps the output length; prevents runaway generation and controls cost.
• top_p: alternative to temperature for nucleus sampling — rarely need to set both.
• stop: a string or list of strings at which generation halts; useful for structured output parsing.

Streaming responses

For user-facing interfaces, stream the response so text appears incrementally rather than waiting for the full completion. The SDK exposes an iterator over delta chunks that you forward to the client.

with client.chat.completions.stream(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain async/await in Python."}],
) as stream:
    for chunk in stream:
        delta = chunk.choices[0].delta.content or ""
        print(delta, end="", flush=True)

Handling errors and retries

LLM APIs return transient errors (rate limits, 500s) regularly under load. Wrap calls in exponential backoff retry logic. The SDKs include built-in retry support; configure max retries and a sensible timeout rather than implementing raw loops yourself.