Ben Santora

Running an 8B Model CPU-Only on a Laptop Without Cooking It

I run small language models locally, on bare metal, with no GPU. Just a laptop CPU. It works, but an 8B model writing a 400-word story pushed my CPU to 94 °C, where it started throttling itself. I don’t like seeing any machine run that hot.

A few changes later, the same task on the same model now peaks at 70 °C, with no throttling. It’s about 30% slower, and the output quality is the same. Here’s what I changed and why it works.

All the scripts are on GitHub, with an installer: bensantora/slm_cool.

The machine

Check that your CPU has AVX-512:

lscpu | grep -o 'avx512[a-z_]*' | sort -u

The key fact: generation is limited by memory, not the CPU

When a model writes, each new token means reading the whole model from RAM. On a laptop, memory speed runs out long before the cores do. So pushing the CPU harder, with turbo, more threads or more watts, adds a lot of heat and not much speed.

That means you can hold the CPU back quite a lot and lose surprisingly little.

Part 1: The Linux side (this is where the cooling comes from)

RAPL, the power limit

Intel CPUs have RAPL (Running Average Power Limit). Linux exposes it through the powercap framework (the intel_rapl driver) as plain files in /sys. There are two limits:

Read yours (values are in microwatts):

cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw   # PL1
cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw   # PL2

Set them (needs root, and resets at reboot):

echo 15000000 | sudo tee /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw  # 15 W
echo 18000000 | sudo tee /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw  # 18 W

Heat comes from watts, so this is the single most important setting. The CPU can’t go over the cap, whatever is running.

Make the everyday cap stick: a small systemd service

/usr/local/sbin/cpu-powercap:

#!/bin/bash
RAPL=/sys/class/powercap/intel-rapl/intel-rapl:0
echo 15000000 > "$RAPL/constraint_0_power_limit_uw"
echo 18000000 > "$RAPL/constraint_1_power_limit_uw"

/etc/systemd/system/cpu-powercap.service:

[Unit]
Description=CPU package power cap (RAPL PL1/PL2)
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/cpu-powercap
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
sudo chmod 755 /usr/local/sbin/cpu-powercap
sudo systemctl enable --now cpu-powercap

I use 15/18 W every day. Normal desktop use feels the same, and heavy jobs are capped. A 60-second all-thread stress test (stress-ng --cpu 8) peaks at 81 °C with this cap, with no throttling.

Turbo and energy preference, from the intel_pstate driver

The intel_pstate CPU frequency driver gives you two more switches:

# turbo off: the CPU stays at its base clock (2.8 GHz here instead of up to 4.7)
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo

# energy/performance preference: lean towards saving power
for f in /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference; do
  echo power | sudo tee "$f" >/dev/null
done

High clocks need high voltage, and power rises steeply with voltage. Dropping turbo removes the least efficient part of the CPU’s range.

“Cool mode” only while the model runs

I didn’t want turbo off all the time, so I made a small script that switches cool mode on and off. On: turbo off, power preference, a 12 W cap. Off: back to normal.

/usr/local/sbin/slm-cool:

#!/bin/bash
set -euo pipefail
RAPL=/sys/class/powercap/intel-rapl/intel-rapl:0
PSTATE=/sys/devices/system/cpu/intel_pstate
epp() { for f in /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference; do echo "$1" > "$f"; done; }
cap() { echo $(( $1 * 1000000 )) > "$RAPL/constraint_0_power_limit_uw"
        echo $(( $2 * 1000000 )) > "$RAPL/constraint_1_power_limit_uw"; }
case "${1:-}" in
  on)  echo 1 > "$PSTATE/no_turbo"; epp power;               cap 12 12 ;;
  off) echo 0 > "$PSTATE/no_turbo"; epp balance_performance; cap 15 18 ;;
  *)   echo "usage: slm-cool on|off" >&2; exit 2 ;;
esac

To let the launcher call it without a password prompt, a sudoers rule allows that one script and nothing else (always edit with visudo):

# /etc/sudoers.d/slm-cool   (sudo visudo -f /etc/sudoers.d/slm-cool)
youruser ALL=(root) NOPASSWD: /usr/local/sbin/slm-cool on, /usr/local/sbin/slm-cool off

It’s handy for other heavy jobs too, like a video render or a long compile: sudo slm-cool on, and later sudo slm-cool off.

Part 2: The llama.cpp flags (a smaller help)

The launcher switches cool mode on, runs the model, and switches cool mode off again on exit, including Ctrl+C:

#!/bin/bash
set -euo pipefail
MODEL="$HOME/.llama/models/L3-8B-Lunaris-v1-Q4_K_M.gguf"

sudo -n /usr/local/sbin/slm-cool on
trap 'sudo -n /usr/local/sbin/slm-cool off' EXIT
trap 'exit 130' INT TERM HUP

llama-cli \
  -m "$MODEL" \
  -ngl 0 \
  -t 3 -tb 3 \
  -c 8192 \
  -fa on -ctk q8_0 -ctv q8_0 \
  -n 1536 \
  --temp 0.9 --min-p 0.05 --repeat-penalty 1.05 \
  -sys "You are a fiction author." \
  "$@"

Don’t use exec for the last line. The EXIT trap has to run after llama-cli quits.

What the flags do:

Flag Why
-ngl 0 No layers on the GPU: pure CPU
-t 3 -tb 3 3 threads on a 4-core CPU. Since memory is the bottleneck, the 4th core adds heat more than speed. Hyperthreads don’t help here.
-c 8192 Llama 3’s trained context length
-fa on Flash attention: the same result, computed more efficiently
-ctk q8_0 -ctv q8_0 Stores the model’s working memory (KV cache) at 8-bit instead of 16-bit. Half the memory traffic, with a negligible quality change. Needs -fa on.
-n 1536 Maximum reply length. Shorter replies mean less time under full load.
--temp, --min-p, --repeat-penalty Sampling. These affect the writing, not the heat.

I changed the thread count, flash attention and the cache type together with the power settings, so I can’t say how much each one did on its own. The power settings set the ceiling, and these trim a little under it.

Measuring it

I log every sensors reading to a CSV once a second while the model runs, then turn the CSV into a colour-coded HTML table. Any logger works. What matters is starting it when the model starts and stopping it when the model finishes. With llama-cli’s -st (single turn: answer once, then exit), one run becomes one command.

Also worth watching: the CPU counts every time it throttles itself because of heat.

cat /sys/devices/system/cpu/cpu0/thermal_throttle/package_throttle_count

Read it before and after a run. If it went up, the CPU hit its thermal limit.

Results

The same prompt (a 400-word story), the same model, once a second, starting from idle:

Before After (cool mode)
Peak, CPU package 94 °C 70 °C
Average while running 82 °C 62 °C
Thermal throttling yes, held at 94 for ~13 s none
Generation speed 6.1 tokens/s 4.2 tokens/s
Prompt reading 29 tokens/s 14 tokens/s
Total time 1 min 42 s 2 min 22 s

“Before” means turbo on, 4 threads, flash attention off, a 16-bit cache, and a 20/22 W cap.

A second Llama 3 8B fine-tune with the same settings peaked at 64 °C.

The “before” curve was still climbing when it hit the throttle point. The “after” curve rises for about a minute, then levels off at 63–66 °C. Heat doesn’t build up without limit: with the power held at 12 W, the temperature settles where the cooling keeps up.

Does slower mean worse?

No. A lower clock does the same maths, just more slowly. The only setting that touches the numbers at all is the 8-bit cache, and that difference is tiny next to the 4-bit quantisation the model already has. If you want zero change there, keep -ctk f16 -ctv f16 and accept a little more heat.

Caveats

Bottom line

Running an 8B model CPU-only on a laptop is very usable, as long as you treat the power limit as the setting that matters. Turbo and full power buy little speed for a model that’s waiting on memory anyway. Turn them down, and a heavy model run on a daily-driver laptop goes from close to 94 °C and throttling to about 70 °C and calm, at about 4 tokens a second.