LLM Temperature, Top-P, and Top-K Explained, With Python Simulations

You set temperature=0.7, top_p=0.9, and hope for the best. But do you actually know what those numbers do to the probability distribution inside the model? Let’s build the math from scratch and see it happen.

Every LLM API call includes sampling parameters. Temperature, top-p, top-k, these control how your model picks the next token. Set them wrong, and your chatbot sounds like a robot or spews nonsense.

The problem? Most developers treat these as magic numbers. They copy temperature=0.7 from a tutorial and never touch it again. That works until the model generates garbage, and you have no idea which knob to turn.

This guide changes that. You will build every sampling algorithm from scratch in Python, watch the probability distributions shift, and understand exactly how each parameter reshapes the model’s choices. By the end, you will know which settings to use for any task, and why.

What Happens Before an LLM Picks a Token

Before we touch any parameters, you need to understand what the model gives us to work with.

When you send a prompt to an LLM, the model doesn’t immediately output a word. It outputs a probability distribution, a list of scores for every token in its vocabulary. These raw scores are called logits (short for “log-odds,” they are unnormalized numbers that can be positive or negative).

Here’s the pipeline. The model produces logits. A function called softmax converts those logits into probabilities that sum to 1.0. Then a sampling strategy picks one token from those probabilities.

Figure 1: The LLM token generation pipeline, logits become probabilities via softmax, then a sampling strategy picks the next token.

Let’s see this in code. We’ll create a set of fake logits for seven tokens and convert them to probabilities using softmax. The function takes each logit, computes e raised to that power, then divides by the sum of all exponentials so the results add up to 1.0. Watch how the highest logit (“the” at 2.5) gets the largest probability share.

“the” grabbed 36.9% of the probability, the largest share because it had the highest logit. But notice the long tail: “pizza,” “moon,” and “void” together hold only 4%. Without any sampling tricks, the model would just pick “the” every time. That’s called greedy decoding, and it produces boring, repetitive text because the same high-probability tokens win over and over.

Sampling parameters exist to add controlled randomness. They let the model sometimes pick “cat” or even “pizza”, making the output more diverse and interesting. The question is: how much randomness, and which tokens should be eligible?

Decoding Strategies, Greedy, Beam Search, and Sampling

Before we dive into temperature and top-p, let’s zoom out. There are three main strategies for choosing the next token from the model’s probability distribution.

Greedy decoding always picks the single most probable token. It’s fast and deterministic, but the output is repetitive and dull. Real language has variety, humans don’t always pick the most obvious next word.

Beam search tracks the top N most probable sequences (not just single tokens) at each step. It produces higher-quality text than greedy decoding, but it’s computationally expensive and still tends toward generic output. It’s mostly used in machine translation, not in chatbots.

Sampling randomly picks a token from the probability distribution. The higher a token’s probability, the more likely it is chosen, but lower-probability tokens have a shot too. This produces creative, diverse text. Temperature, top-k, and top-p are all sampling controls, they adjust how the random selection happens.

Temperature, The Sharpness Knob

Temperature is the most important sampling parameter. It controls how “sharp” or “flat” the probability distribution is before the model picks a token.

The intuition is simple. Low temperature makes the model more confident and predictable. The highest-probability token dominates even more. High temperature makes the model more adventurous. Lower-probability tokens get a bigger share of the pie.

The Math

Temperature modifies the softmax function by dividing the logits by a temperature value $T$ before computing exponentials.

\[\text{softmax}(x_i, T) = \frac{e^{x_i / T}}{\sum_j e^{x_j / T}}\]

Where:
– \(x_i\) = the logit for token $i$
– $T$ = the temperature value (typically 0.0 to 2.0)
– The denominator sums over all tokens in the vocabulary

If you’re not interested in the math, skip to the code below, the visual output tells the full story.

Here’s why dividing by T works. Consider two logits: 2.5 and 0.5, with a gap of 2.0. If you divide both by T=0.5, you get 5.0 and 1.0, the gap grows to 4.0. Softmax amplifies larger gaps, so the top token dominates even more. If you divide by T=2.0, you get 1.25 and 0.25, the gap shrinks to 1.0. Softmax sees smaller gaps, so probabilities spread out more evenly.

See It in Code

Let’s modify our softmax to accept a temperature parameter. The function will divide all logits by T before computing exponentials. We’ll compare the resulting distributions at four temperatures, 0.2, 0.7, 1.0, and 1.5, to see how the same logits produce very different probability spreads. Watch how “the” goes from dominating at low T to sharing the spotlight at high T.

Look at the shift. At T=0.2, “the” grabs 85% of the probability, the model almost always picks it. The tail tokens (“pizza,” “moon,” “void”) collectively hold 0.0%, they’re effectively eliminated. At T=1.5, “the” drops to 27.5% and the tail tokens rise to 8.7% combined. The distribution went from a spike to a spread. This is why high temperature outputs feel “creative”, tokens that would normally never appear suddenly have a real chance of being selected.

What Temperature Should You Use?

Here’s a reference table mapping temperature ranges to behavior and use cases.

Top-K Sampling, Hard Vocabulary Limit

Temperature changes the shape of the distribution, but it still lets every token in the vocabulary have a chance, even tokens with near-zero probability. Top-k takes a different approach: hard-cut the vocabulary to only the K most probable tokens and discard the rest completely.

The intuition is like limiting a restaurant menu. Instead of choosing from 50,000 dishes (the full vocabulary), you only see the top 10. The rest simply don’t exist for this round of selection.

How It Works

1. Sort all tokens by probability (highest first)
2. Keep only the top K tokens
3. Set all other token probabilities to zero
4. Renormalize the remaining K probabilities so they sum to 1.0
5. Sample from this smaller distribution

Let’s implement this. Our top_k_sampling function will compute probabilities using softmax, find the indices of the K highest-probability tokens with np.argsort, zero out everything else into a mask, then renormalize. We’ll run it at K=1, 3, 5, and 7 to see how the pool shrinks.

K=1 is greedy decoding, only “the” survives with 100%. At K=3, something interesting happens: “the” jumps from 36.9% (its original probability) to 56.3%. We didn’t change temperature, so why did “the” get stronger? That’s renormalization. The probability mass that used to belong to “dog,” “pizza,” “moon,” and “void” gets redistributed among the three survivors. The fewer tokens you keep, the more the top token dominates, even without lowering temperature.

At K=7, all tokens survive, so the distribution is identical to the original softmax. This tells you that top-k only matters when K is smaller than the number of tokens with meaningful probability.

The Problem with Top-K

Top-k has a fundamental flaw: K is fixed regardless of context. Sometimes the model is 99% sure the next token is “Paris” (after “The capital of France is”). In that case, even K=2 is too many, the second token is noise. Other times, any of 100 tokens would make sense. K=10 would be too few.

Top-k doesn’t adapt. That’s why top-p was invented.

Top-P (Nucleus Sampling), Dynamic Vocabulary

Top-p, also called nucleus sampling (from the 2020 paper by Holtzman et al.), solves top-k’s biggest problem. Instead of keeping a fixed number of tokens, it keeps tokens until their cumulative probability reaches a threshold $p$.

Let’s walk through this with concrete numbers before we touch any code. Say our tokens have these probabilities after softmax:

“the”: 37%, “a”: 18%, “cat”: 8%, “dog”: 5%, “pizza”: 2%, “moon”: 1%, “void”: 0.7%

Starting from the highest, we add up: 37% isn’t enough for our threshold of 90%. So we add “a”: 37% + 18% = 55%, still short. Add “cat”: 55% + 8% = 63%, keep going. Add “dog”: 63% + 5% = 68%. Add “pizza”: 68% + 2% = 70%. Add “moon”: 70% + 1% = 71%. We haven’t hit 90% yet, so we keep all of them. But if we’d set top-p=0.7, we’d stop after “moon” and discard “void.”

The beauty is that this adapts automatically. If the model is 95% confident in one token, top-p=0.95 keeps just that one token. If the model is uncertain and the top 50 tokens only sum to 90%, top-p=0.95 keeps all 50. The pool size adjusts to the model’s confidence.

How It Works

1. Sort tokens by probability (highest first)
2. Compute the cumulative sum of probabilities
3. Find the smallest set of tokens whose cumulative probability is at least $p$
4. Zero out all tokens outside this set
5. Renormalize and sample

Let’s build this. The top_p_sampling function will compute probabilities, sort them in descending order, find where the cumulative sum crosses the threshold using np.searchsorted, then mask everything below the cutoff. We’ll test at four thresholds to see how the pool size adapts.

At P=0.5, only “the” and “a” are needed to cover 50% of the probability mass, so only 2 tokens survive. At P=0.95, six tokens are needed. The pool size adapts to the distribution, and that’s the whole point. If “the” had 95% probability on its own, top-p=0.95 would keep just that single token. Top-k=3 would still include two unnecessary tokens. This adaptive behavior is why top-p produces more consistent output quality across different contexts than top-k.

🏋️ Exercise 1: Build Your Own Top-P Filter

Implement a function that takes logits, a temperature, and a top-p value, then returns the filtered probability distribution. Test it with the logits below and top_p=0.8.

How Temperature, Top-K, and Top-P Work Together

In practice, these parameters are applied in sequence, not in isolation. Here’s the typical pipeline that most LLM APIs follow.

1. Temperature scales the logits (reshapes the distribution)
2. Top-k removes all but the K most probable tokens
3. Top-p removes additional tokens below the cumulative probability threshold
4. Sample from the remaining distribution

Figure 2: The complete sampling pipeline, temperature reshapes probabilities, then top-k and top-p filter the candidate tokens before final sampling.

Let’s build the complete pipeline and watch how each step transforms the distribution. The full_sampling_pipeline function applies temperature first, then optionally top-k, then optionally top-p, and finally picks a random token weighted by the surviving probabilities. We’ll test four configurations to see how stacking the filters changes the outcome.

Notice that top-k=4 and top-p=0.9 produce the exact same result here, both end up keeping the same four tokens. In practice, they often overlap because both are trying to cut the low-probability tail. The last row confirms it: adding both filters together doesn’t change anything compared to using either one alone. Most API providers recommend using temperature + top-p OR temperature + top-k, not all three.

Here’s a quick reference for which combination to use.

Min-P, The New Challenger

There’s a newer sampling method gaining traction: Min-P. It was introduced in a 2025 ICLR paper and addresses a flaw in top-p that shows up at high temperatures.

The problem with top-p: at high temperatures, the distribution becomes flat. Many tokens have similar probabilities. Top-p uses a fixed cumulative threshold, so it ends up including many low-quality tokens when the distribution is flat, tokens like “xylophone” after “The capital of France is.”

Min-P takes a different approach. Instead of a cumulative threshold, it uses a relative threshold. It keeps any token whose probability is at least \(p \times p_{\text{max}}\), where \(p_{\text{max}}\) is the probability of the most likely token. If the top token has 30% probability and you set min_p=0.1, any token below 3% (30% times 0.1) is eliminated.

Let’s compare min-p against top-p to see the difference. We’ll use a larger 10-token vocabulary to make the effect more visible. The min_p_sampling function computes a threshold relative to the top token’s probability and masks everything below it. We’ll run both methods at high temperature (T=1.5), where top-p’s weakness shows up.

Now the difference is dramatic. At high temperature, top-p=0.9 keeps 8 tokens, including “void” and “quantum” with only 3% each. These low-quality tokens can produce bizarre output in the wrong context. Min-p=0.1 keeps only 6 tokens by eliminating everything below 10% of the top token’s probability. It’s stricter because it measures each token relative to the best token, not against a cumulative sum. This prevents low-quality tokens from sneaking in when the distribution is flat.

The key advantage: min-p’s threshold automatically tightens when the model is confident (high top probability) and relaxes when the model is uncertain (low top probability). It adapts like top-p, but without top-p’s tendency to let garbage through at high temperatures.

Repetition Penalties, The Adjacent Controls

Temperature, top-k, and top-p control which tokens are eligible for selection. But there’s another family of parameters that control which tokens are discouraged: repetition penalties.

Frequency penalty reduces the probability of tokens that have already appeared in the output. The more times a token has been used, the stronger the penalty. This prevents the model from repeating the same phrase over and over.

Presence penalty is simpler, it applies a fixed penalty to any token that has appeared at least once, regardless of how many times. It encourages the model to bring up new topics rather than dwelling on the same one.

Both penalties work by adjusting the logits before sampling. A positive frequency penalty subtracts from the logit of each repeated token proportionally to its count. A positive presence penalty subtracts a flat value from any previously-seen token’s logit.

Here’s how they interact with temperature: if you use high temperature for creativity but the model keeps repeating itself, a small frequency_penalty (0.3-0.7) can break the loop without reducing creative variety. The penalty pushes repeated tokens’ logits down, and temperature handles the rest.

Practical Cheat Sheet, Settings for Every Task

Here’s a ready-to-use reference for setting sampling parameters by task type.

The Decision Framework

Don’t memorize this table. Instead, follow this decision process:

1. Start with temperature=0.7 and top_p=1.0 (no filtering). Run your prompt a few times.
2. If the output is too boring or repetitive, raise temperature to 0.9-1.0.
3. If the output is too wild or incoherent, lower temperature to 0.3-0.5.
4. Only add top-p when you want to keep temperature high but prevent garbage tokens. Set top_p=0.9 to cut the long tail.
5. If you still see repetition, add frequency_penalty=0.3-0.5. Don’t touch temperature for repetition problems, that’s what penalties are for.

Figure 3: Decision flowchart, choose your sampling settings based on your task type.

API Parameter Names Across Providers

The parameter names are slightly different across providers. Here’s a quick lookup table.

🏋️ Exercise 2: Sampling Parameter Advisor

Write a function that takes a task description and returns recommended sampling parameters. Test it with three different tasks.

See It in Action, OpenAI API Demo

Let’s use the OpenAI API to see how temperature changes real model output. We’ll send the same prompt at three different temperatures and compare the results. The completions are genuinely different, T=0.0 is safe, T=0.7 is expressive, and T=1.5 is borderline surreal.

from openai import OpenAI

client = OpenAI()  # uses OPENAI_API_KEY env var

prompt = "Complete this sentence creatively: The robot looked at the sunset and felt"

print(f"Prompt: '{prompt}'\n")

for temp in [0.0, 0.7, 1.5]:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=temp,
        max_tokens=30,
    )
    text = response.choices[0].message.content
    print(f"T={temp:.1f}: {text}")

Output:

Prompt: 'Complete this sentence creatively: The robot looked at the sunset and felt'

T=0.0: The robot looked at the sunset and felt a strange warmth in its circuits, as if the fading light had awakened something it wasn’t programmed to
T=0.7: The robot looked at the sunset and felt a quiet longing stir within its metallic frame, an echo of emotions it could never quite name
T=1.5: The robot looked at the sunset and felt crackling whispers of amber dissolving through quantum veins, a phosphorescent ache between zeros

At T=0.0, the output is safe and predictable, exactly what greedy decoding produces. At T=0.7, the language is more expressive ("quiet longing," "echo of emotions") because the model can reach beyond the top-probability tokens. At T=1.5, the distribution is so flat that unusual tokens win ("crackling whispers," "quantum veins"), creating poetic but borderline incoherent prose. Each temperature serves a different purpose, and now you understand exactly why.

🏋️ Exercise 3: Temperature Comparison Tool

Build a function that runs our sampling simulation at multiple temperatures and prints a comparison table. Use the functions we built (no API needed).

Common Mistakes and How to Fix Them

Mistake 1: Setting Both Temperature and Top-P to Extreme Values

Combining a high temperature with a very high top-p removes almost all filtering. The model can pick any token.

❌ Wrong:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain gravity"}],
    temperature=1.8,
    top_p=0.99,  # almost no filtering
)
# Result: incoherent, rambling output

Why it's wrong: Temperature=1.8 flattens the distribution dramatically. Top-p=0.99 keeps virtually all tokens. Together, even the lowest-probability tokens have a shot. The output degrades into gibberish.

✅ Correct:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain gravity"}],
    temperature=0.9,
    top_p=0.9,  # filters the long tail
)
# Result: creative but coherent

If you want more creativity, increase temperature OR top-p, not both to their maximums.

Mistake 2: Expecting Identical Output at Temperature=0

Temperature=0 means greedy decoding. Many developers assume this guarantees identical output every time. It usually does, but not always.

❌ Wrong assumption:

# "This will always return the exact same text"
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    temperature=0,
)
# May vary slightly between API calls due to floating-point differences

Why it's wrong: GPU floating-point arithmetic isn't perfectly deterministic. Different hardware, batching, or server routing can cause tiny numerical differences that occasionally flip which token is "highest." OpenAI offers a seed parameter for more reproducibility.

✅ Correct:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    temperature=0,
    seed=42,  # improves reproducibility
)

Mistake 3: Confusing max_tokens with Sampling Parameters

Max tokens limits the output length. It has nothing to do with creativity or randomness.

❌ Wrong:

# "I'll set max_tokens low to get a concise, focused answer"
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain gravity"}],
    max_tokens=20,
    temperature=1.5,  # still generates wild tokens
)
# Result: short AND incoherent — worst of both worlds

Why it's wrong: The model still uses temperature=1.5 to pick each token. You just cut it off after 20 tokens. The output is short AND random.

✅ Correct:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain gravity in one sentence"}],
    max_tokens=50,
    temperature=0.3,  # focused tokens
)

Use temperature to control quality/style. Use max_tokens to control length. They are independent knobs.

Mistake 4: Using Top-K with OpenAI Models

OpenAI's API does not support the top_k parameter. Passing it causes an error.

❌ Wrong:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    top_k=50,  # ERROR: OpenAI doesn't support this
)

✅ Correct:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    top_p=0.9,
    temperature=0.7,
)

Check your provider's API docs for supported parameters before coding.

Frequently Asked Questions

What is the difference between temperature and top-p?

Temperature changes how much each token's probability differs from others, it reshapes the entire curve. Top-p changes how many tokens are eligible to be picked, it cuts the tail. Temperature controls the shape of the distribution; top-p controls the size of the candidate pool. You can use both together: temperature sets the sharpness, then top-p removes whatever low-probability tokens survive.

What is a good temperature for an LLM?

For most tasks, 0.7 is the best starting point. It provides enough variety to avoid repetitive output while keeping responses coherent. Lower (0.0-0.3) for code generation and factual tasks. Higher (0.9-1.2) for creative writing. Never go above 1.5 unless you're deliberately experimenting.

What does temperature 0 mean in LLM?

Temperature=0 disables sampling entirely and uses greedy decoding, the model always picks the single highest-probability token. This gives you the most predictable, deterministic output. It's ideal for tasks where correctness matters more than variety, like code generation or data extraction.

What is the default temperature for GPT-4 and ChatGPT?

OpenAI's default temperature is 1.0 for GPT-4 and GPT-4o. ChatGPT's web interface uses internal settings that aren't publicly documented but are believed to be around 0.7-1.0. Anthropic's Claude defaults to 1.0. Google's Gemini also defaults to 1.0. These defaults are designed for general-purpose chat.

Can sampling parameters cause hallucinations?

Yes. Higher temperature increases the chance of selecting low-probability tokens, which can include factually incorrect continuations. If your model hallucinates, lowering temperature (0.0-0.3) and top-p (0.1-0.5) reduces the randomness that allows hallucinated tokens through. However, hallucinations can also happen at temperature=0, sampling parameters reduce the chance of hallucination but don't eliminate it.

Do reasoning models (o1, o3) use these parameters?

Most reasoning models lock their sampling parameters internally. OpenAI's o1 and o3 models do not accept temperature or top-p in the API, they use fixed internal settings optimized for chain-of-thought reasoning. DeepSeek-R1 similarly recommends temperature=0.6 and discourages changes. If you're using a reasoning model, trust its defaults.

Is there a way to see what tokens the model considered?

Yes. You can request logprobs=True in the OpenAI API. This returns the log-probabilities of the top tokens at each position, letting you see which tokens the model considered and how confident it was. This is invaluable for debugging unexpected outputs, you can see whether a bad token was a 0.1% fluke or a 40% contender.

Complete Code

References

1. Holtzman, A. et al., “The Curious Case of Neural Text Degeneration.” ICLR 2020. arXiv:1904.09751. The original nucleus sampling (top-p) paper.
2. Fan, A., Lewis, M., & Dauphin, Y., “Hierarchical Neural Story Generation.” ACL 2018. arXiv:1805.04833. Introduced top-k sampling for text generation.
3. Nguyen, M. et al., “Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs.” ICLR 2025. arXiv:2407.01082. The min-p paper demonstrating improvements over top-p.
4. OpenAI API Reference, Chat Completions. Link. Official documentation for temperature, top_p, and other parameters.
5. Anthropic API Reference, Messages. Link. Claude API documentation for sampling parameters.
6. Google Gemini API, GenerationConfig. Link. Gemini API documentation including top-k support.
7. Chng, P., “Token selection strategies: Top-K, Top-P, and Temperature.” Link. Excellent visual explanation of softmax with temperature.
8. Radford, A. et al., “Language Models are Unsupervised Multitask Learners.” OpenAI (2019). GPT-2 paper that popularized these sampling strategies.