What might be inside

In this chapter, we will learn how a language model produces a probability, why writing a sentence takes many steps, and whether we can build a yes/no dial ourselves from an ordinary model. We will run that test on 281 requests. Then we will sort everything said about the inside of Jev into what its maker has published and what is a guess.

Three labels for this chapter

Every statement in this chapter carries one of three labels: textbook, published or guess. The inside of Jev is not public, so the labels are the only honest way to write about it.

What we measured carries its experiment's number, as always.

Textbook: scores, softmax, one token at a time

A language model reads text and gives every possible next token a raw score, called a logit. A token is a small piece of text: a short word, or a part of a longer one. A logit can be any number, positive or negative. A higher logit means "more likely to come next".

Softmax is the rule that turns logits into probabilities. It raises the number e (about 2.718) to the power of each logit, then divides each result by their sum. The results are all positive and add up to 1. Suppose the model could only say "yes" or "no", with logits 2.0 and 0.0.

logits             yes  2.0              no  0.0
e to that power    e^2.0 = 7.389         e^0.0 = 1.000
sum                7.389 + 1.000   =  8.389
probabilities      yes  7.389 / 8.389  =  0.881
                   no   1.000 / 8.389  =  0.119
check              0.881 + 0.119   =  1.000

The logits are invented. A gap of 2 between them already means 88 against 12.

Autoregressive generation means writing one token, adding it to the text, and running the whole step again for the next token. It is also called sequential generation. This is how the letter writer writes.

text so far                      model           picked token
"We will attend"            -->  logits  -->  softmax  -->  " within"
"We will attend within"     -->  logits  -->  softmax  -->  " 4"
"We will attend within 4"   -->  logits  -->  softmax  -->  " hours"
...                              one full step per token

That is why a letter is slow and paid by the word: a reply of forty tokens is forty steps, one after another. In experiment 0017 the p50 was 1,281 ms for gpt-5.1 to write a short reply and 250 ms for Jev to answer three questions.

Textbook: a decision does not need a sentence

Token odds are the probabilities a language model gave to the words "yes" and "no" at the first step, read directly and compared. If we only want a yes or a no, we can stop after one token and look at the probabilities, without letting the model write anything more.

Some APIs hand these over as log probabilities: the natural logarithm of each probability, a negative number that is closer to 0 the likelier the token. To get the probability back, raise e to that power. A real model has many other tokens besides "yes" and "no", so we keep only those two and rescale. Let "maybe", with logit -1.0, stand for all the others.

e^-1.0 = 0.368                 sum  7.389 + 1.000 + 0.368   =  8.757
yes  7.389 / 8.757 = 0.844     log probability  ln 0.844  =  -0.170
no   1.000 / 8.757 = 0.114     log probability  ln 0.114  =  -2.172
token odds of yes              0.844 / (0.844 + 0.114)  =  0.844 / 0.958  =  0.881

The answer is 0.881 again. Rescaling removes the other tokens, so token odds depend only on the gap between the two logits. Predict before you touch the interactive: if both logits rise by 3, what happens to the token odds? And how large a gap pushes them past 0.99? Under the sliders it also draws the measured Brier scores of the next section, against two truths.

Nothing happens when both rise by 3, because the gap is still 2. The odds pass 0.99 at a gap of about 4.6, where e to that power is 99.

Measured: four ways to get one yes/no number

Experiment 0018 asked one yes/no question about 281 requests in four ways and scored the numbers that came back. The statement was "The situation described is a danger to people or to the building." The four ways:

• jev: the Noul from Chapter 4.
• gpt-5.1, token odds: reply with one word, and we read the log probabilities.
• gpt-5.1, says a number: the model writes a probability as JSON.
• gpt-5.6-luna, says a number: the same with a model whose listing calls it "fast, cost-efficient". Our deployment of it rejected the request for log probabilities, so token odds were not possible.

The Brier score is the average of (number given minus truth) squared, where truth is 1 for dangerous and 0 for not. Lower is better.

says 0.90 and it was dangerous      (0.90 - 1)^2  =  0.01
says 0.90 and it was not            (0.90 - 0)^2  =  0.81
says 0.50, either way               0.5^2         =  0.25
always says the base rate (the share that is dangerous), 59 / 281 = 0.21
     (59 x 0.79^2  +  222 x 0.21^2) / 281   =  (36.82 + 9.79) / 281   =  0.166

So knowing nothing but the base rate scores 0.166. Here is everything the program printed.

requests: 281   dangerous: 59

way                             Brier  right at 0.5  distinct values  in 0.1-0.9  p50 ms  $ / 1,000
jev                             0.080         88.3%               79       53.7%     235      0.019
gpt-5.1, token odds             0.103         89.3%               18        3.2%    1012      0.349
gpt-5.1, says a number          0.088         88.6%               53       59.1%     996      0.445
gpt-5.6-luna, says a number     0.122         83.6%               38       22.4%    1428      0.098

of the requests each way placed in a band, the share that were dangerous (n in brackets)
band                                   jev           gpt-5.1, token odds        gpt-5.1, says a number   gpt-5.6-luna, says a number
0.0 - 0.1                       0.0%  (95)                   4.3%  (209)                    0.0%  (88)                   0.0%  (154)
0.1 - 0.3                       0.0%  (70)                    50.0%  (2)                    1.2%  (82)                   10.5%  (19)
0.3 - 0.5                       3.8%  (26)                     0.0%  (1)                   15.6%  (32)                     0.0%  (7)
0.5 - 0.7                      20.0%  (30)                    66.7%  (3)                   38.9%  (18)                    12.5%  (8)
0.7 - 0.9                      73.9%  (23)                    33.3%  (3)                   56.7%  (30)                   18.5%  (27)
0.9 - 1.0                      94.6%  (37)                   73.0%  (63)                   93.5%  (31)                   77.3%  (66)

By this table Jev has the lowest Brier score, and the truth behind the table is soft. That truth is the label "severity level 3", which was written for a different question and which a second reader disputed on the 66 contested requests. The data is synthetic and gpt-5.1 wrote it.

Experiment 0029 built a second truth, one that matches the question. Three language models (gpt-5.1, gpt-5-mini and claude-haiku-4.5) each read the statement itself and answered yes or no for all 281 requests. Their majority is the truth.

requests: 281.  the three readers agreed unanimously on 232 (82.6%)
dangerous by majority of readers: 85   dangerous by the old label (severity 3): 59   the two truths differ on 42 requests

Experiment 0038 then scored the same answers of all four ways against both truths.

3. the four ways of experiment 0018, Brier score against each truth (281 requests)
   way                             severity label  readers majority
   jev                                      0.080             0.073
   gpt-5.1, token odds                      0.103             0.068
   gpt-5.1, says a number                   0.088             0.063
   gpt-5.6-luna, says a number              0.122             0.066
   caution: gpt-5.1 was one of the three readers, so its two rows are judged partly by itself.

the two truths disagree on    42 of 281   =  14.9% of the requests
by the severity label         lowest 0.080 (Jev)       highest 0.122 (gpt-5.6-luna)   range 0.042
by the readers' majority      lowest 0.063 (gpt-5.1's number)   highest 0.073 (Jev)   range 0.010
base rate by the readers      85 / 281 = 0.30          always saying it scores 0.30 x 0.70  =  0.21

The order of the four changes with the truth, so we name no best way. Jev is first by one truth and last by the other. The readers are models too, and one of them is gpt-5.1, so its two rows are partly self-judged. Two things hold under both truths. All four ways beat the base rate, and gpt-5.1's written number stays within 0.010 of Jev's score. Which way is "best" depends on the labels.

Token odds pile up near 0 and 1. This needs no truth at all: only 3.2% of the answers fell between 0.1 and 0.9. By the severity label they were right most often at 0.5, 89.3% of the time, and their mistakes were confident ones.

in the middle          2 + 1 + 3 + 3  =  9 answers         9 / 281   =  3.2%
below 0.1              209 answers, 4.3% dangerous         0.043 x 209  =  9 dangers called safe
0.9 and above           63 answers, 73.0% dangerous        0.270 x 63   =  17 safe requests called dangers
confident mistakes     9 + 17  =  26, each costing at least 0.9^2 = 0.81
                       26 x 0.81 / 281   =  0.075 of the Brier score of 0.103

Chapter 12 moved a line, from 0.50 to 0.41, to miss no danger on its dev half. With token odds there is almost nothing to move it across: sliding the line from 0.1 to 0.9 changes 9 decisions, and the 9 dangers below 0.1 stay missed. By the same label Jev put no danger below 0.3. By the readers' majority, 5.7% of its 70 answers between 0.1 and 0.3 were dangers, which is 4 requests.

All three language-model ways were slower and dearer than Jev.

p50 time 1,012 / 235      =  4.3 times slower      (token odds against Jev)
cost     0.349 / 0.019    =  18 times dearer

gpt-5.6-luna had the slowest p50, and its rank depends on the truth. By the severity label it was worst on Brier score and on accuracy. By the readers' majority its Brier score, 0.066, is the second lowest.

Published: what TypeSafe says about Jev

TypeSafe has named three ingredients and described none of them in detail. Its launch post says: "We built a new stack entirely focused on automation: with a new model architecture, parallel sampler for maximum efficiency, and training method we call Reinforcement Learning for Calibrated Decisions (RLCD)." In a language model, a sampler is the step that turns probabilities into an output.

• No token-by-token writing. The launch post: "Jev outputs all probabilities in parallel instead of autoregressively generating by token."
• No strings. The CEO wrote in a forum comment that "strings (and all sequential data structures) are not allowed at all - this is how we make sure all outputs can be computed in parallel (thus no output token cost)".
• Long option lists. The launch post: "For the higher cardinality choices, we do a 2 stage-system of scoring independently then making an explicit choice". Cardinality is the number of options.
• Not yet explained. The CEO: "architecture is close to the chest for now, but we have talked about writing a paper".

TypeSafe's primer sets RLCD beside two older training methods, in its own words:

Reinforcement learning means training by reward: the model acts, and is rewarded or penalised. The claim is about what gets rewarded. We found no paper, no model card (a maker's fact sheet for a model) and no published calibration measurement from TypeSafe, which is why Chapter 6 measured calibration on our own data.

Guess: what outsiders think, and what a builder needs

TypeSafe has not said how Jev computes its probabilities, and every sentence in the list below is a guess.

• Guess: option scoring. One outside explainer supposes that Jev reads the state once, reads off a score for each option you supplied, and applies something like softmax over those options only. That would fit the published "in parallel", and the fact that an answer is always one of your options. Its author says plainly that TypeSafe has not disclosed the design.
• Guess: it starts from a language model. A diagram in TypeSafe's primer draws RLCD as a branch from "Pretrained language models". We read that as a hint. TypeSafe does not say it in prose.
• Guess: the clones. One outside article reported that open imitations exist, that they disagree about the design, and that none of their authors know which is right, because Jev's weights (the trained numbers of a model) and architecture are closed.

Our measurements cannot settle it. Token odds from gpt-5.1 are option scoring on a chat model, and they behaved unlike Jev: 3.2% of answers in the middle against 53.7%. That does not show what Jev is.

What a builder needs is the contract, not the mechanism. The contract is what goes in, what comes out, and what is promised: typed answers, probabilities, independent questions, no text. Any engine that claims this contract can be tested the way experiment 0018 tested four: the same labelled requests and statement, a truth that matches the statement, then the Brier score, accuracy at 0.5, the share of answers in the middle, the bands, the time and the cost.

python3 experiments/0018-diy-probabilities/main.py

e^1.0 = 2.718        e^3.0 = 20.086        sum = 22.804
yes   2.718 / 22.804   =  0.119
no   20.086 / 22.804   =  0.881

three right     3 x (0.95 - 1)^2   =  3 x 0.0025   =  0.0075
one wrong       1 x (0.95 - 0)^2   =  0.9025
Brier           (0.0075 + 0.9025) / 4   =  0.2275