Calling an LLM: the API

In this chapter, we will learn how a program talks to a language model. We will send a real request with the OpenAI library and read the response field by field. We will watch max_tokens, the system prompt and streaming in action. We will also see the exact text the model reads, which is not quite what we sent.

What an API is

An API is an agreed way for one program to ask another program for something. The letters stand for application programming interface. For language models the agreement is simple: send a text message in a fixed format to an address on the internet, and a text message in a fixed format comes back.

The format is JSON. JSON is a way of writing down data as names and values, in plain text, with curly brackets around it.

your program                              the provider's servers
+------------------+     request (JSON)     +---------------------------------+
|  the question    |  ------------------->  |  tokenizer -> model -> tokenizer |
|  the settings    |                        |  the loop of Chapter 1          |
|                  |  <-------------------  |                                 |
+------------------+     response (JSON)    +---------------------------------+

The model runs on the provider's machines, so your program needs no special computer. It needs an internet connection and an API key, a long secret word that says who is asking and who will be sent the bill.

The request

A request holds three things: which model, what to say to it, and how to pick the tokens. This is the request we sent.

(a) The request: one JSON object
{
  "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful cooking assistant. Answer in one short sentence."
    },
    {
      "role": "user",
      "content": "I am vegetarian. What should I cook tonight?"
    }
  ],
  "temperature": 0.7,
  "max_tokens": 120,
  "seed": 7
}

Messages and roles

Every message has a role, which says who is speaking. There are three main roles. A fourth, for the results of tools, appears in Chapter 12.

The message with the role system is called the system prompt. The user of an application usually never sees it. It is where a company turns a general model into its own assistant. OpenAI's newer models call this role developer, and they call the limit max_completion_tokens.

Build a request yourself. The interactive starts with OpenAI, so it shows these two newer names. Change the message and the settings, and look at the same request as JSON, as Python and as a command for the terminal. Then switch where the request goes, to an open model on your own server, and watch the two names turn back into system and max_tokens.

The code

A few lines of code are enough to send the request. OpenAI publishes a library for Python that builds the JSON, sends it and reads the answer.

from openai import OpenAI

client = OpenAI()          # reads the key from the environment variable OPENAI_API_KEY

response = client.chat.completions.create(
    model="gpt-6-luna",
    messages=[
        {"role": "developer", "content": "You are a helpful cooking assistant. Answer in one short sentence."},
        {"role": "user", "content": "I am vegetarian. What should I cook tonight?"},
    ],
    temperature=0.7,
    max_completion_tokens=120,
)
print(response.choices[0].message.content)
print(response.usage)

The interactive shows the same program when it is set to OpenAI and Python.

The response

The response is one JSON object that holds the answer, the reason the model stopped, and the count of tokens. This one came back for the request above.

(b) The response: one JSON object
{
  "id": "chatcmpl-aa21d084-29ea-4845-aa10-f6ed59ace224",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "You might enjoy a vegetable lasagna or a chickpea shepherd's pie.",
        "role": "assistant"
      }
    }
  ],
  "created": 1790642039,
  "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 17,
    "prompt_tokens": 36,
    "total_tokens": 53,
    "prompt_tokens_details": {
      "cached_tokens": 5
    }
  }
}

The fields id and created are a serial number and the time of the answer.

total                                           36 + 17  =  53
trips through the model to write the answer     17

What the model really reads

A chat template is the fixed pattern by which the server joins the list of messages into one single text. The model of Chapter 1 continues a text. It knows nothing of lists and roles. So the server writes the messages one after the other, with special tokens that mark where each turn begins and ends.

(i) What the model really reads: the messages, joined into one text by the chat template
<|im_start|>system
You are a helpful cooking assistant. Answer in one short sentence.<|im_end|>
<|im_start|>user
I am vegetarian. What should I cook tonight?<|im_end|>
<|im_start|>assistant

  tokens in this text   36
  first ten pieces      '<|im_start|>' | 'system' | '\n' | 'You' | ' are' | ' a' | ' helpful' | ' cooking' | ' assistant' | '.'

The words between <| and |> are special tokens, which never occur in ordinary text. Every family of models has its own template, learned in training.

Here is the costume we spoke of in Chapter 1. The text ends with the line <|im_start|>assistant and then nothing. The most likely continuation of a text that looks like this is a helpful answer, followed by <|im_end|>. When that token is drawn, the loop stops, and we get finish_reason: stop.

words we wrote, system and user together            19
tokens for those words, measured                    13 + 10  =  23
tokens for the template: markers, roles, new lines  36 - 23  =  13
tokens the model read                               36

The 19 words became 23 tokens because full stops and question marks are tokens too. The count of 36 is the number the response reported as prompt_tokens. We pay for the template too.

The settings in action

A setting changes how the loop runs. It does not change the model. The experiment with the temperature is in Chapter 7: at 1.6 the names fell apart into broken text.

max_tokens cuts the answer off

We asked what a token is, with a limit of 8 tokens and then of 150.

(g) max_tokens cuts the answer off
  max_tokens 8   finish_reason 'length'  completion tokens 8   'A token is a small, discrete unit'
  max_tokens 150 finish_reason 'stop'    completion tokens 68  'A token is a small, discrete unit of information that represents something meaningful in a system. It can be a character, a number, a symbol, or a sequence of characters, and it is used to represent data or instructions in a system. Tokens are often used in programming languages to represent variables, functions, or other data structures.'

With a limit of 8 the answer stops in the middle of the sentence. The model did not plan a short answer. It was writing the same long answer, and the loop was stopped after 8 trips. max_tokens is a brake. It is not an instruction. If you want a short answer, say so in the prompt. Always check finish_reason: if it is length, the answer is incomplete.

The system prompt changes the behaviour

(f) The system prompt changes the behaviour, not the model
  system: Answer in one short sentence, for a six year old.
     "The sky is blue because the Earth's atmosphere scatters sunlight in all directions, and blue light is scattered more than other colors."
  system: Answer in one sentence, as a physics professor would.
     "The sky appears blue due to a phenomenon called Rayleigh scattering, where shorter blue wavelengths are scattered more by the Earth's atmosphere, making them visible from Earth's surface."
  system: Reply only with a JSON object that has the keys "answer" and "confidence".
     '{\n  "answer": "The sky appears blue due to a phenomenon called Rayleigh scattering. Sunlight is composed of many colors of light, and blue light is scattered more easily by the Earth\'s atmosphere. This scattering causes the blue light to be scattered in all directions, making the sky appear blue to us. The other colors of light are scattered less, which is why the sky appears white or gray when the sun is not shining.",\n  "confidence": 95\n}'

The question was the same three times, "Why is the sky blue?". The third answer is a JSON object, as ordered, which a program can read without a person. And the first answer is not written for a child of six. A small model follows the form of an instruction more easily than its spirit. A number such as "confidence: 95" is also only text that the model wrote. Nothing measured it.

Streaming

Streaming means the server sends each token as soon as it is drawn, without waiting for the whole answer. It is switched on with "stream": true.

(h) Streaming: the tokens arrive one by one
      217 ms   '1'
      296 ms   '.'
      329 ms   ' Spin'
      402 ms   'ach'
      423 ms   ' and'
      497 ms   ' Art'
      525 ms   'ich'
      598 ms   'oke'
      618 ms   ' Soup'
      695 ms   '\n'
      778 ms   '2'
      806 ms   '.'
  ... 23 more pieces
  full answer        '1. Spinach and Artichoke Soup\n2. Quinoa Salad with Feta Cheese and Arugula\n3. Lentil Soup with Potatoes and Onions'
  first piece after  217 ms
  all 35 pieces after 2097 ms  = 18 pieces per second once it started

This is why the answer of a chat assistant appears word by word. You are watching the loop. The reader can begin after 217 ms and does not have to wait 2,097 ms. Notice also how the pieces fall: "Spinach" arrives as " Spin" and "ach", two tokens.

The model remembers nothing

The model keeps nothing between two requests. Each request must carry everything the model should know. The word for this is stateless.

(e) The model keeps nothing between requests
  request 1   user: My name is Maya and I have a cat called Miso.
              assistant: "Hello Maya! It's nice to meet you. Miso sounds like a cute cat. How can I help you today?"
  request 2   user: What is my cat called?          (sent alone)
              assistant: "I'm sorry, but I'm not able to provide personal information about your cat. However, you can easily find out what your cat's name is by looking at its collar or by asking someone who"
  request 3   the whole conversation so far, then the same question
              assistant: 'Your cat is called Miso.'
  prompt tokens   request 2: 35    request 3: 83

In request 2 the model had no idea who was asking. Our question of five words cost 35 tokens, because this model's template adds a standard system prompt of its own when we send none. In request 3 we sent the earlier messages again, and then it knew.

A chat application does this for you: with every message you type, it sends the whole conversation so far.

request 2, the question alone          35 tokens
request 3, the conversation so far     83 tokens
the price of remembering               83 - 35  =  48 tokens more, on this request and on every later one

That growing cost is the subject of Chapter 10.

pip install openai mlx-lm
python -m mlx_lm server --model mlx-community/Qwen2.5-1.5B-Instruct-4bit --port 8090

# in a second terminal
python 04_call_the_api.py