Running an 8B Model CPU-Only on a Laptop Without Cooking It
I run small language models locally, on bare metal, with no GPU. Just a laptop CPU. It works, but an 8B model writing a 400-word story pushed my CPU to 94 °C, where it started throttling itself. I don’t like seeing any machine run that hot.
A few changes later, the same task on the same model now peaks at 70 °C, with no throttling. It’s about 30% slower, and the output quality is the same. Here’s what I changed and why it works.
All the scripts are on GitHub, with an installer: bensantora/slm_cool.
The machine
- HP ENVY 17m laptop, Intel Core i7-1165G7: 4 cores / 8 threads, AVX-512
- Debian 12, CPU only (
-ngl 0, nothing on the GPU) - llama.cpp, Linux release build. It ships CPU code for several instruction sets and loads the best one for your processor at startup, so the AVX-512 code gets used with no extra setup.
- Model: L3-8B-Lunaris-v1, a Llama 3 8B fine-tune for fiction, quantised to Q4_K_M (about 4.9 GB)
Check that your CPU has AVX-512:
lscpu | grep -o 'avx512[a-z_]*' | sort -u
The key fact: generation is limited by memory, not the CPU
When a model writes, each new token means reading the whole model from RAM. On a laptop, memory speed runs out long before the cores do. So pushing the CPU harder, with turbo, more threads or more watts, adds a lot of heat and not much speed.
That means you can hold the CPU back quite a lot and lose surprisingly little.
Part 1: The Linux side (this is where the cooling comes from)
RAPL, the power limit
Intel CPUs have RAPL (Running Average Power Limit).
Linux exposes it through the powercap framework (the
intel_rapl driver) as plain files in /sys.
There are two limits:
- PL1: the long-term limit, the wattage the CPU can hold indefinitely
- PL2: a short-term boost above PL1, allowed for a limited time
Read yours (values are in microwatts):
cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw # PL1
cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw # PL2
Set them (needs root, and resets at reboot):
echo 15000000 | sudo tee /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw # 15 W
echo 18000000 | sudo tee /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw # 18 W
Heat comes from watts, so this is the single most important setting. The CPU can’t go over the cap, whatever is running.
Make the everyday cap stick: a small systemd service
/usr/local/sbin/cpu-powercap:
#!/bin/bash
RAPL=/sys/class/powercap/intel-rapl/intel-rapl:0
echo 15000000 > "$RAPL/constraint_0_power_limit_uw"
echo 18000000 > "$RAPL/constraint_1_power_limit_uw"
/etc/systemd/system/cpu-powercap.service:
[Unit]
Description=CPU package power cap (RAPL PL1/PL2)
After=multi-user.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/cpu-powercap
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
sudo chmod 755 /usr/local/sbin/cpu-powercap
sudo systemctl enable --now cpu-powercap
I use 15/18 W every day. Normal desktop use feels the same, and heavy
jobs are capped. A 60-second all-thread stress test
(stress-ng --cpu 8) peaks at 81 °C with this cap, with no
throttling.
Turbo and energy preference, from the intel_pstate driver
The intel_pstate CPU frequency driver gives you two more
switches:
# turbo off: the CPU stays at its base clock (2.8 GHz here instead of up to 4.7)
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
# energy/performance preference: lean towards saving power
for f in /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference; do
echo power | sudo tee "$f" >/dev/null
done
High clocks need high voltage, and power rises steeply with voltage. Dropping turbo removes the least efficient part of the CPU’s range.
“Cool mode” only while the model runs
I didn’t want turbo off all the time, so I made a small script that
switches cool mode on and off. On: turbo off, power
preference, a 12 W cap. Off: back to normal.
/usr/local/sbin/slm-cool:
#!/bin/bash
set -euo pipefail
RAPL=/sys/class/powercap/intel-rapl/intel-rapl:0
PSTATE=/sys/devices/system/cpu/intel_pstate
epp() { for f in /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference; do echo "$1" > "$f"; done; }
cap() { echo $(( $1 * 1000000 )) > "$RAPL/constraint_0_power_limit_uw"
echo $(( $2 * 1000000 )) > "$RAPL/constraint_1_power_limit_uw"; }
case "${1:-}" in
on) echo 1 > "$PSTATE/no_turbo"; epp power; cap 12 12 ;;
off) echo 0 > "$PSTATE/no_turbo"; epp balance_performance; cap 15 18 ;;
*) echo "usage: slm-cool on|off" >&2; exit 2 ;;
esac
To let the launcher call it without a password prompt, a sudoers rule
allows that one script and nothing else (always edit with
visudo):
# /etc/sudoers.d/slm-cool (sudo visudo -f /etc/sudoers.d/slm-cool)
youruser ALL=(root) NOPASSWD: /usr/local/sbin/slm-cool on, /usr/local/sbin/slm-cool off
It’s handy for other heavy jobs too, like a video render or a long
compile: sudo slm-cool on, and later
sudo slm-cool off.
Part 2: The llama.cpp flags (a smaller help)
The launcher switches cool mode on, runs the model, and switches cool mode off again on exit, including Ctrl+C:
#!/bin/bash
set -euo pipefail
MODEL="$HOME/.llama/models/L3-8B-Lunaris-v1-Q4_K_M.gguf"
sudo -n /usr/local/sbin/slm-cool on
trap 'sudo -n /usr/local/sbin/slm-cool off' EXIT
trap 'exit 130' INT TERM HUP
llama-cli \
-m "$MODEL" \
-ngl 0 \
-t 3 -tb 3 \
-c 8192 \
-fa on -ctk q8_0 -ctv q8_0 \
-n 1536 \
--temp 0.9 --min-p 0.05 --repeat-penalty 1.05 \
-sys "You are a fiction author." \
"$@"
Don’t use exec for the last line. The EXIT
trap has to run after llama-cli quits.
What the flags do:
| Flag | Why |
|---|---|
-ngl 0 |
No layers on the GPU: pure CPU |
-t 3 -tb 3 |
3 threads on a 4-core CPU. Since memory is the bottleneck, the 4th core adds heat more than speed. Hyperthreads don’t help here. |
-c 8192 |
Llama 3’s trained context length |
-fa on |
Flash attention: the same result, computed more efficiently |
-ctk q8_0 -ctv q8_0 |
Stores the model’s working memory (KV cache) at 8-bit instead of
16-bit. Half the memory traffic, with a negligible quality change. Needs
-fa on. |
-n 1536 |
Maximum reply length. Shorter replies mean less time under full load. |
--temp, --min-p,
--repeat-penalty |
Sampling. These affect the writing, not the heat. |
I changed the thread count, flash attention and the cache type together with the power settings, so I can’t say how much each one did on its own. The power settings set the ceiling, and these trim a little under it.
Measuring it
I log every sensors reading to a CSV once a second while
the model runs, then turn the CSV into a colour-coded HTML table. Any
logger works. What matters is starting it when the model starts and
stopping it when the model finishes. With llama-cli’s -st
(single turn: answer once, then exit), one run becomes one command.
Also worth watching: the CPU counts every time it throttles itself because of heat.
cat /sys/devices/system/cpu/cpu0/thermal_throttle/package_throttle_count
Read it before and after a run. If it went up, the CPU hit its thermal limit.
Results
The same prompt (a 400-word story), the same model, once a second, starting from idle:
| Before | After (cool mode) | |
|---|---|---|
| Peak, CPU package | 94 °C | 70 °C |
| Average while running | 82 °C | 62 °C |
| Thermal throttling | yes, held at 94 for ~13 s | none |
| Generation speed | 6.1 tokens/s | 4.2 tokens/s |
| Prompt reading | 29 tokens/s | 14 tokens/s |
| Total time | 1 min 42 s | 2 min 22 s |
“Before” means turbo on, 4 threads, flash attention off, a 16-bit cache, and a 20/22 W cap.
A second Llama 3 8B fine-tune with the same settings peaked at 64 °C.
The “before” curve was still climbing when it hit the throttle point. The “after” curve rises for about a minute, then levels off at 63–66 °C. Heat doesn’t build up without limit: with the power held at 12 W, the temperature settles where the cooling keeps up.
Does slower mean worse?
No. A lower clock does the same maths, just more slowly. The only
setting that touches the numbers at all is the 8-bit cache, and that
difference is tiny next to the 4-bit quantisation the model already has.
If you want zero change there, keep -ctk f16 -ctv f16 and
accept a little more heat.
Caveats
- Intel only, as written.
intel-raplandintel_pstateare Intel. AMD hasamd_pstateand sets power limits differently. - Every laptop’s cooling is different. 12 W suited mine. Find yours by testing: change one setting, run the same prompt, compare peak temperature and tokens/s.
- Writing to
/sysresets at reboot. That’s what the systemd service is for. - Fan speed: many laptops don’t report it to Linux (mine shows 0 RPM), so go by temperature.
Bottom line
Running an 8B model CPU-only on a laptop is very usable, as long as you treat the power limit as the setting that matters. Turbo and full power buy little speed for a model that’s waiting on memory anyway. Turn them down, and a heavy model run on a daily-driver laptop goes from close to 94 °C and throttling to about 70 °C and calm, at about 4 tokens a second.
