Running Edge LLMs on a Raspberry Pi

What if a Raspberry Pi could generate text entirely on its own, without sending prompts to a cloud-based AI service?

To find out, I installed several compact large language models on a Raspberry Pi 5 with 8GB of RAM and tested how well they could respond to prompts written in Japanese.

I tried the following four combinations:

• Gemma 4 E2B Instruction Tuned + LiteRT-LM

• Qwen3 0.6B Mixed INT4 + LiteRT-LM

• Qwen2.5 1.5B Instruct Q8 + LiteRT-LM

• Qwen3 1.7B GGUF Q8_0 + llama.cpp

Note: This experiment focused on Japanese-language performance. For reproducibility, the original Japanese prompts are preserved in the commands.

---

What Is an Edge LLM?

“Edge LLM” is not the name of one strict standard. In general, it refers to an LLM designed to run on a device outside the cloud, such as a smartphone, PC, single-board computer, or embedded system.

Running a model with tens or hundreds of billions of parameters directly on an edge device is usually impractical. Edge LLM deployments therefore rely on techniques such as:

1. Keeping the model size between roughly 0.5B and a few billion parameters

2. Quantizing model weights to 4-bit or 8-bit precision

3. Converting the model into a format optimized for on-device execution

4. Using an inference runtime optimized for CPUs, GPUs, or NPUs

5. Limiting the context length and maximum output size for the intended task

Advantages of Edge LLMs

Edge LLMs offer several benefits that cloud-hosted LLMs do not:

• After downloading the model, inference can work without a network connection

• Input data can be processed without sending it to an external inference API

• You can experiment without worrying about per-token API charges

• Processing can happen close to sensors, cameras, and other field devices

• You control the model, prompts, logs, and storage location

However, running a model locally does not automatically make the system secure. The operating system, logs, network configuration, malware protection, and access permissions still need to be managed properly.

Limitations of Edge LLMs

• They may have less knowledge and weaker instruction-following ability than large cloud models

• Hallucinations and unnatural wording can be more noticeable with smaller models

• Long-form generation can be slow when running on a CPU

• Sustained workloads make cooling and power quality important

• The same question may not produce exactly the same answer every time

• Model formats and inference tools are not always interchangeable

---

Models Tested in This Experiment

Model Comparison

Model	Provider	Size	Runtime format	Downloaded or published file size	Main characteristics
Gemma 4 E2B it	Google	E2B	.litertlm	About 2.4 GiB downloaded	Designed for mobile and edge devices; produced the most natural response in this test
Qwen3 0.6B	Qwen Team	0.6B	.litertlm Mixed INT4	474.6 MiB	Extremely compact; supports thinking and non-thinking modes
Qwen2.5 1.5B Instruct	Qwen Team	1.54B	.litertlm Q8	About 1.5 GiB downloaded	Multilingual instruction-tuned model targeting more than 29 languages
Qwen3 1.7B	Qwen Team	1.7B	GGUF Q8_0	1.83 GB	GGUF model that is easy to run with llama.cpp; supports thinking mode

Google presents the E2B and E4B variants of Gemma 4 as models for mobile and edge devices. In Google's published Raspberry Pi 5 16GB CPU measurements, Gemma 4 E2B achieved a decode speed of 8 tokens per second, a time to first token of 7.8 seconds, and peak CPU memory usage of 1,546 MB.

I did not record the equivalent LiteRT-LM metrics on my 8GB Raspberry Pi, so the results in this article should not be treated as a direct comparison with Google's figures.

Qwen3 is a hybrid model that can switch between thinking and non-thinking modes. For simple questions where speed matters, /no_think can be used. For tasks such as mathematics or coding, /think can allow the model to spend more tokens reasoning.

A model file's size is not the same as its runtime memory usage. The runtime, KV cache, input tokens, output tokens, and other data also require memory.

---

LiteRT-LM, llama.cpp, and Ollama

Model names and runtime names can be easy to confuse.

• Gemma, Qwen, and Llama are model families

• LiteRT-LM, llama.cpp, and Ollama are tools used to run or manage models

Item	LiteRT-LM	llama.cpp	Ollama
Primary role	On-device LLM runtime and orchestration layer from Google AI Edge	Lightweight LLM inference engine written in C and C++	Tool that simplifies local model installation, management, and API serving
Main model format	.litertlm	GGUF	Ollama-managed models; GGUF models can also be imported
Ease of setup	Straightforward when a compatible model is available	Requires building the software and understanding its options	Usually easy to start because the commands are short
Model selection	Mainly LiteRT-LM-compatible or pre-converted models	Very large selection of published GGUF models	Models from the Ollama library or custom Modelfile definitions
Fine-grained control	CLI, Python, and C++ interfaces with on-device features	Detailed control over threads, context, quantization, and backends	Hides many low-level settings to simplify operation
Typical Raspberry Pi use	Quickly testing supported models such as Gemma 4	Comparing many GGUF models and quantization levels	Simple model management and local API use
Used in this experiment	Yes	llama-cli only	No

For quickly trying a supported model with a short command, LiteRT-LM is convenient. If you want to compare many GGUF models and quantization methods, llama.cpp offers more flexibility.

.litertlm and GGUF

Format	Mainly used with	Models used in this experiment
.litertlm	LiteRT-LM	Gemma 4 E2B, Qwen3 0.6B, and Qwen2.5 1.5B
GGUF	llama.cpp and compatible applications	Qwen3 1.7B Q8_0

A .litertlm package contains the model and related information required by LiteRT-LM. GGUF is widely used by llama.cpp-based runtimes, and many pre-quantized models are available in this format.

---

Test Environment

The following conditions can be confirmed from my execution logs:

Item	Configuration
Board	Raspberry Pi 5
Memory	8 GB
CPU	Broadcom BCM2712, four Arm Cortex-A76 cores at 2.4 GHz
Inference device	CPU
llama.cpp threads	4
llama.cpp context length	2,048
llama.cpp maximum generation	256 tokens
llama.cpp build	b10077-5735e10c4
LiteRT-LM version	Not recorded in the execution log
Operating system	The reproduction steps below assume 64-bit Raspberry Pi OS

Raspberry Pi recommends a high-quality 5V/5A power supply and active cooling for the Raspberry Pi 5. LLM inference keeps the CPU under load for extended periods, so a fan or the official Active Cooler is a good idea for stable operation.

---

Installation and Execution

1. Update Raspberry Pi OS

First, confirm that you are running a 64-bit operating system.

After reconnecting, install the required packages:

sudo apt install -y \
  python3-venv \
  python3-pip \
  git \
  curl \
  cmake \
  build-essential \
  pkg-config \
  libcurl4-openssl-dev \
  jq \
  htop \
  time

---

2. Install LiteRT-LM

The LiteRT-LM CLI requires Python 3.10 or later. Rather than installing it directly into the system Python environment, I used a virtual environment:

mkdir -p ~/venvs
python3 -m venv ~/venvs/litert-lm
source ~/venvs/litert-lm/bin/activate

python -m pip install --upgrade pip
python -m pip install --upgrade litert-lm

Check that the CLI is available:

litert-lm --help

---

3. Run Gemma 4 E2B with LiteRT-LM

I first downloaded and ran Gemma 4 E2B:

litert-lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  --prompt='あなたは日本語で簡潔に回答するアシスタントです。Raspberry Pi 5の特徴を150字程度で説明してください。'

The Japanese prompt means:

You are an assistant who answers concisely in Japanese. Describe the main features of the Raspberry Pi 5 in approximately 150 Japanese characters.

On the first run, the tool downloaded a model of approximately 2.4 GiB:

Downloading 'gemma-4-E2B-it.litertlm'
from 'litert-community/gemma-4-E2B-it-litert-lm' (2.4GiB)...
[####################] 100%

Actual Output

The following is an English translation of the original Japanese response:

The Raspberry Pi 5 features a high-performance CPU, fast RAM, and an improved
GPU, delivering a major increase in processing power compared with previous
models. This makes it possible to run a wider range of projects and applications
more comfortably, including more complex AI processing, advanced graphics, and
high-speed data processing.

The model summarized the main points in natural Japanese and stayed reasonably close to the requested length. As a first functional test, the result was encouraging.

However, a precise description of Raspberry Pi 5 specifications should still be checked against official documentation, including the exact CPU model, number of cores, clock speed, and supported graphics APIs. Natural-sounding LLM output is not automatically factually correct.

Register the Model in the Local Registry

To avoid entering the long repository name every time, Gemma 4 E2B can be registered locally with the ID gemma4-e2b:

litert-lm import \
  --from-huggingface-repo litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  gemma4-e2b

List the registered models:

litert-lm list

My system displayed:

Listing models in: /home/yfpi/.litert-lm/models
ID          SIZE    MODIFIED
gemma4-e2b  2.4 GB  2026-07-21 23:53:15

The model can then be started with the shorter ID:

litert-lm run gemma4-e2b

---

4. Try Gemma 4 E2B in Interactive Mode

I launched the registered model in interactive mode:

litert-lm run gemma4-e2b

I used the following question, which was also used to compare the other models:

日本語で、ローカルLLMを利用できる用途を5つ挙げてください。

The prompt means:

In Japanese, list five use cases for a local LLM.

Actual Output

Open the full English translation of the Gemma 4 E2B response

Local LLMs can be used for a wide variety of purposes. They are especially
powerful when privacy, security, or offline operation is important.

Here are five practical use cases.

---

### 1. Summarizing and analyzing confidential internal documents

Use case:

Processing highly confidential information that must not be disclosed outside
an organization, such as customer data, patent information, internal design
documents, and personnel records.

Benefits of a local LLM:

- Avoiding data exposure: Because the data does not need to be sent to an
  external server over the internet, the risk of external data leakage can be
  completely eliminated.
- Secure processing: Document summarization, keyword extraction, and compliance
  checks can be performed in an environment that applies the organization's own
  security policies and access controls.

### 2. Advanced customer support and FAQ systems in offline environments

Use case:

Providing knowledge-based question answering and troubleshooting in locations
with unreliable internet access, such as factories, remote areas, or restricted
security zones.

Benefits of a local LLM:

- Independence from the network: The service can continue operating locally
  without a constantly available internet connection.
- Retaining specialist knowledge: Knowledge not covered by general-purpose
  external models, such as industry terminology and internal manuals, can be
  fully incorporated into a locally trained model.

### 3. Specialized content generation and brainstorming for niche fields

Use case:

Drafting highly specialized content or brainstorming ideas in a particular
field, such as a specific area of law, historical literature, documentation for
a particular programming language, or a certain historical period.

Benefits of a local LLM:

- Higher domain-specific accuracy: Fine-tuning the model with a domain-specific
  dataset, such as a database of legal provisions, can dramatically improve its
  accuracy in that field.
- Easier copyright and licensing management: Generating and reviewing content
  locally before sending it to an external service can make licensing and
  copyright issues easier to check.

### 4. Code generation and review in development and debugging environments

Use case:

Allowing developers to request code suggestions, bug identification, and
refactoring advice for proprietary coding standards or a specific legacy
codebase inside a private environment.

Benefits of a local LLM:

- Source-code confidentiality: Customer source code and unpublished algorithms
  can be reviewed or used for code generation without being disclosed outside
  the organization.
- A fast feedback loop: Developers can run the model locally and receive
  immediate assistance without interrupting their workflow.

### 5. Personalized, locally hosted chatbots focused on user experience

Use case:

Building highly personalized conversational assistants for a particular user
group, such as a hobby community or a group of learners, using that group's
context and previous conversations.

Benefits of a local LLM:

- Deeper contextual understanding: Training locally on past questions, speaking
  styles, and community-specific terminology can produce deeper contextual
  understanding and more human-like responses than a general-purpose model.
- Ownership of user data: Conversation data can be processed entirely locally
  rather than stored on an external server, increasing users' confidence in how
  their data is handled.

The response clearly separated the five use cases and explained the benefits of local execution for each one. Among all the outputs in this experiment, it had the best structure, natural Japanese, and alignment with the question.

However, I would not reuse the following claims without checking and revising them:

• “The risk of data leakage can be completely eliminated”

• “Specialist knowledge can be fully incorporated into the model”

• “Fine-tuning can dramatically improve accuracy”

Information can still leak from a local environment. Simply giving internal documents to a model also does not mean that the model permanently learns them. Retrieval-augmented generation, inserting documents into a prompt, and fine-tuning are different mechanisms.

The writing quality was high, but the strong claims still needed human review. That is a very typical LLM result.

---

5. Run Qwen3 0.6B Mixed INT4 with LiteRT-LM

Next, I tested the much smaller Qwen3 0.6B:

litert-lm run \
  --from-huggingface-repo=litert-community/Qwen3-0.6B \
  qwen3_0_6b_mixed_int4.litertlm \
  --prompt='日本語で、ローカルLLMを利用できる用途を5つ挙げてください。 /no_think'

The first download was 474.6 MiB:

Downloading 'qwen3_0_6b_mixed_int4.litertlm'
from 'litert-community/Qwen3-0.6B' (474.6MiB)...
[####################] 100%

Actual Output

The Japanese response contained several unclear or unnatural phrases. The following translation preserves that awkwardness as closely as possible:

<think>
</think>

The following are uses of a local LLM:

1. Product construction and development: Build a language model suitable for
   "rollover" topics and provide real-time processing.
2. Diagnosing the user's body: Embed a model suitable for "rollover" to diagnose
   the state of a vehicle's operating mechanism in real time.
3. Understanding the structure of the user's body's vehicle: Analyze a vehicle's
   configuration and mechanical state in real time using a language model
   suitable for "rollover."
4. Understanding the contents of a mobile application: Process the contents of
   a vehicle's mobile application in real time using a language model suitable
   for "rollover."
5. Processing mobile-application data: Process data in a vehicle's mobile
   application in real time using a language model suitable for "rollover."

This was the smallest model in the experiment, so its low download and storage requirements are attractive. However, the output had several problems:

• It used unnatural terms that were difficult to interpret

• It unexpectedly focused on vehicles, although the question did not mention them

• The differences between the five items were unclear

• It did not successfully explain general use cases for local LLMs

Because /no_think was included, the <think> block was empty. The fact that the empty tags were still displayed was probably related to the output formatting or chat template.

Based only on this result, Qwen3 0.6B Mixed INT4 may be more suitable for tasks with tightly constrained outputs, such as short classification, structured extraction, or selecting from fixed candidates, rather than open-ended Japanese explanations.

---

6. Run Qwen2.5 1.5B Instruct with LiteRT-LM

I also tested Qwen2.5 1.5B Instruct, which is slightly larger than the 0.6B model:

litert-lm run \
  --from-huggingface-repo=litert-community/Qwen2.5-1.5B-Instruct \
  --prompt='日本語で、ローカルLLMを利用できる用途を5つ挙げてください。'

Because the repository contained multiple model files, LiteRT-LM displayed a selection screen:

? Please select a model file
Qwen2.5-1.5B-Instruct_multi-prefill-seq_q8_ekv4096.litertlm

The selected file was approximately 1.5 GiB:

Downloading
'Qwen2.5-1.5B-Instruct_multi-prefill-seq_q8_ekv4096.litertlm'
from 'litert-community/Qwen2.5-1.5B-Instruct' (1.5GiB)...
[####################] 100%

Actual Output

The following is an English translation of the original Japanese response:

1. Create a project-management application: This centralizes information such
   as team profiles, task schedules, and tablet status.
2. Record meetings in audio and video: This centralizes important information
   and makes it possible to create recordings that can serve as ledgers or
   reference materials.
3. Create and analyze service-improvement metrics: This makes it possible to
   analyze user flows related to a service and create service-improvement
   indicators.
4. Cloud-based file sharing: This makes it easy to share files and data and
   streamline workflows.
5. Translation services: This makes it possible to translate foreign-language
   text.

The Japanese was easier to read than the Qwen3 0.6B response, and the model successfully produced a list of five items. However, its alignment with the question was still weak:

• “Recording meetings in audio and video” is not itself an LLM use case

• “Cloud-based file sharing” does not fit the topic of local LLMs

• The role of the LLM in project management and metrics analysis was unclear

• Translation is a reasonable use case, but the answer did not explain the benefit of running it locally

The 1.5B model produced more readable language than the 0.6B model, but this result also shows that readability and correctness are not the same thing.

---

7. Build llama.cpp

To test the GGUF version of Qwen3 1.7B, I built llama.cpp from source:

cd ~
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

Create a release build:

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build \
  --config Release \
  -j4 \
  --target llama-cli

Check the executable:

./build/bin/llama-cli --help

llama.cpp is a C/C++ inference engine that supports multiple CPU instruction sets, including ARM NEON, as well as many quantization formats. The -hf option can download and run a compatible model directly from Hugging Face.

---

8. Run Qwen3 1.7B GGUF Q8_0 with llama.cpp

I used the following command:

./build/bin/llama-cli \
  -hf Qwen/Qwen3-1.7B-GGUF:Q8_0 \
  --jinja \
  --simple-io \
  -t 4 \
  -c 2048 \
  -n 256 \
  -p "日本語で、ローカルLLMを利用できる用途を5つ挙げてください。" \
  -st

The Japanese prompt again means:

In Japanese, list five use cases for a local LLM.

The main options are:

Option	Meaning
-hf	Download and run a model from Hugging Face
Q8_0	Select the 8-bit quantized version
--jinja	Use the Jinja chat template included with the model
--simple-io	Simplify terminal input and output
-t 4	Use four CPU threads for inference
-c 2048	Set the context length to 2,048
-n 256	Limit generation to 256 tokens
-p	Provide the initial prompt
-st	Run one turn and exit

Actual Output

Open the Qwen3 1.7B console output

Loading model...

build      : b10077-5735e10c4
model      : Qwen/Qwen3-1.7B-GGUF:Q8_0
ftype      : Q8_0
modalities : text

> 日本語で、ローカルLLMを利用できる用途を5つ挙げてください。

[Start thinking]

Okay, the user wants me to list five uses where a local LLM can be utilized. Let me think about the different areas where local LLMs are valuable.

First, customer service is a big one. Companies can use local LLMs for chatbots that handle FAQs and support, which is more efficient than using cloud-based services. That's a solid point.

Second, content generation. Local LLMs can help with writing articles, generating code, or creating marketing materials. That's another key use case.

Third, data analysis. Local models can process and analyze data, especially if the company has sensitive data that they want to keep on-premise. That makes sense for compliance reasons.

Fourth, personalization. Local LLMs can tailor recommendations or content based on user behavior, which is important for e-commerce or streaming services.

Fifth, development tools. Like code generation or debugging assistance, which can speed up the development process for developers.

Wait, are there any other areas? Maybe real-time translation or language processing? But the user asked for five, so I need to pick the most relevant ones. Let me check each point again to make sure they're distinct and cover different applications. Yeah, those five seem comprehensive. I should present them

[ Prompt: 36.3 t/s | Generation: 5.9 t/s ]

Exiting...

Speed

llama.cpp reported the following speeds:

Processing stage	Measured speed
Prompt processing	36.3 tokens/s
Generation	5.9 tokens/s

Prompt is the rate at which the model processes the input. Generation is the rate at which it produces the response one token at a time. Generation speed generally has a greater effect on how fast the answer feels to the user.

Why Was There No Final Answer?

The model loaded successfully and inference clearly ran. However, it stopped after producing the English thinking text, before displaying a final answer in Japanese.

The likely explanation is the following combination:

1. Qwen3 can use thinking mode by default

2. Generation was limited to -n 256

3. The thinking process probably consumed the available output-token budget

4. With -st, the process ended after the single turn

This does not mean that Qwen3 1.7B cannot answer the question on a Raspberry Pi. A more reasonable interpretation is that the token-budget settings did not match the task.

Improved Command for Prioritizing a Normal Answer

For a simple question, add /no_think to the prompt:

./build/bin/llama-cli \
  -hf Qwen/Qwen3-1.7B-GGUF:Q8_0 \
  --jinja \
  --simple-io \
  -t 4 \
  -c 2048 \
  -n 256 \
  -p "日本語で、ローカルLLMを利用できる用途を5つ挙げてください。 /no_think" \
  -st

To use thinking mode, increase the maximum number of generated tokens:

./build/bin/llama-cli \
  -hf Qwen/Qwen3-1.7B-GGUF:Q8_0 \
  --jinja \
  --simple-io \
  -t 4 \
  -c 2048 \
  -n 1024 \
  -p "日本語で、ローカルLLMを利用できる用途を5つ挙げてください。 /think" \
  -st

Increasing the output limit also increases execution time. For a simple list question on a Raspberry Pi, /no_think is probably the more practical default.

---

Comparison of Measured Results

The following table is a subjective assessment based on one output from each model in this experiment:

Model and runtime	Model download size	Answer completed	Natural Japanese	Alignment with question	Recorded speed	Impression from this test
Gemma 4 E2B / LiteRT-LM	About 2.4 GiB	Yes	High	High	Not recorded	Most practical response
Qwen3 0.6B INT4 / LiteRT-LM	474.6 MiB	Yes	Low	Low	Not recorded	Very small, but struggled with open-ended generation
Qwen2.5 1.5B Q8 / LiteRT-LM	About 1.5 GiB	Yes	Medium	Low to medium	Not recorded	Readable, but required careful fact checking
Qwen3 1.7B Q8_0 / llama.cpp	1.83 GB	Stopped before final answer	No final answer	Not assessable	Prompt: 36.3 tokens/s; generation: 5.9 tokens/s	Worth retesting with revised settings

This was a functional test, not a controlled benchmark. The following conditions were not standardized:

• LiteRT-LM speed and peak memory usage were not recorded

• The model formats and quantization methods differed

• Thinking mode was active only for Qwen3 1.7B

• The maximum generation settings were not identical

• Each model was tested only once

• Sampling parameters were not fixed

• The first Gemma 4 test used a different question

It would therefore be incorrect to conclude that “Gemma 4 is always better than Qwen” or that “Qwen3 0.6B is unusable.”

The result supported by this experiment is more limited:

On this hardware, with these model files, these prompts, and these individual runs, Gemma 4 E2B produced the best response.

---

What I Learned

1. Small LLMs Can Run on a Raspberry Pi 5 with 8GB of RAM

I successfully loaded and ran generation with 0.6B, 1.5B, 1.7B, and Gemma 4 E2B models on the CPU.

With compact, quantized models, the Raspberry Pi 5 is an enjoyable platform for learning about and testing local LLM inference.

However, the work described here is mainly inference. I did not perform large-scale pretraining or substantial fine-tuning on the Raspberry Pi itself.

2. The Smallest Model Was Lightweight, but Quality Differences Were Visible

Qwen3 0.6B Mixed INT4 was only 474.6 MiB, but its open-ended Japanese answer was noticeably unnatural.

From a single test, it is not possible to separate the effects of model size, quantization, conversion format, and generation settings. Smaller models may be easier to evaluate when the task is narrowly defined, for example:

• Classify a message as an “inquiry,” “incident,” or “request”

• Extract only a few fields in JSON

• Answer yes or no

• Choose one item from a fixed list

• Convert an instruction into a short command

3. Settings Matter as Much as Model Size

Qwen3 1.7B was larger than the 0.6B model, but it did not reach the final answer in this run. The likely issue was not raw processing capability, but the combination of thinking mode and -n 256.

A fair comparison should control at least the following:

• Whether thinking mode is enabled

• Maximum output tokens

• Context length

• Sampling settings such as temperature

• Prompt wording

• Number of test runs

4. The Runtime Also Affects Usability

LiteRT-LM made it easy to run compatible .litertlm models with short commands. Registering a model under a local ID was also convenient.

llama.cpp required a build step, but it offered a much wider selection of GGUF models and more control over thread count, context length, and maximum generation.

5. Natural-Sounding Answers Still Need Verification

The Gemma 4 E2B response was readable and persuasive, but it also included overly strong claims such as saying that a risk could be “completely eliminated.”

I found it useful to review LLM output in three separate stages:

1. Is the writing natural?

2. Does it answer the question?

3. Is the content factually correct?

Official documentation and real measurements are still essential.

---

Conclusion

I installed several edge-oriented LLMs on a Raspberry Pi 5 with 8GB of RAM and ran Gemma 4 and Qwen models entirely on the CPU.

Within the scope of this experiment, Gemma 4 E2B with LiteRT-LM made the best overall impression in terms of setup simplicity and Japanese response quality.

Qwen3 0.6B was extremely lightweight, but its open-ended Japanese generation was unnatural in this run. Qwen2.5 1.5B produced more readable text, although its response did not always match the question closely.

Qwen3 1.7B running through llama.cpp achieved a measured generation speed of 5.9 tokens per second on the Raspberry Pi 5 CPU. However, the thinking process probably consumed the 256-token output budget before the final answer could be generated. Using /no_think or increasing the output-token limit should improve the result.

An edge LLM is not simply a smaller copy of a cloud LLM. It requires balancing the model, quantization method, runtime, prompt design, context length, and output budget within limited memory and compute resources.

That tuning process is part of what makes the Raspberry Pi such an interesting platform for learning about edge AI.