A local model with several ways in
Hugging Face has published a guide to running Thinking Machines Lab's Inkling 2 Mini on a personal computer. The November 20 release brings support through Transformers, Llama App and llama.cpp, with an MLX route for Apple silicon. The guide also covers SGLang for larger deployments. For someone trying the model at home, those choices raise a practical question: which download belongs with which application? [1]
The first choice is the software that will run the model. A desktop application suits someone who wants to manage downloads and chat in one place. A command-line server or Python package gives developers more control. Before downloading anything, decide whether the job calls for a chat window, an endpoint for a coding editor, or image input. Then follow one set of instructions through to a working example. Mixing steps from different runtimes can turn a simple configuration problem into a confusing installation.
What the model's size actually means
Thinking Machines describes Inkling 2 Mini as a mixture-of-experts model with 26.4 billion total parameters and 3.4 billion active parameters. Its middle layers process each token three times. That reuse produces 5.4 billion weight activations per token, according to the lab. The distinction matters when comparing specification sheets: the active count describes computation, while the total count includes weights the computer still needs to accommodate. [2]
The lab puts the checkpoint at roughly 20 GB and demonstrates offline use on a MacBook Pro with 48 GB of memory. Its model card lists Apple silicon machines with at least 32 GB of memory among supported hardware, alongside NVIDIA DGX Spark; other GPU setups are possible, with 32 GB or more of VRAM recommended. Those are vendor guidelines for the checkpoint and supported setups, rather than a guarantee that every quantized download will behave identically. [2] [4]
Choose the GGUF file deliberately
The ggml-org repository provides a useful explanation of the storage tradeoff. Its file table lists IQ4_NL at 17.6 GB, Q8_0 at 27.8 GB and BF16 at 52.2 GB. The smaller IQ4_NL version uses four-bit quantization for selected expert weights while retaining higher precision elsewhere. The repository also lists separate vision-encoder files, so users following an image-capable setup should check its complete download requirements. [5]
These numbers describe files, not a complete memory budget for a running conversation. Users should leave room for the operating system, other applications and the runtime's working data. A laptop that can store a download on its SSD may still struggle to run it. Before downloading several variants, readers can choose one, record its exact name and check whether the runtime recognizes it. That simple record also makes later comparisons useful: a speed change means little if the model format changed at the same time.
Llama App offers a desktop starting point
Llama App runs on Mac and Windows and supplies a browser chat interface plus a local API. Its website describes automatic choices for model precision and context size based on available memory. The application can work offline once the necessary files are present. Developers can point compatible software at localhost:9931/v1, the address documented for its OpenAI-compatible endpoint. Compatibility refers to the API format; the selected model still runs locally. [6]
A short prompt in the supplied chat interface makes a useful first check. Once that works, connect the editor and confirm which model it selects. A failure confined to the editor points toward connection settings or a requested feature that the setup does not support. A working chat response establishes only that the basic conversation path works. Tool calls and image handling need their own small examples, using the same client and settings that the eventual workflow will use.
MLX gives Mac developers another route
Hugging Face's guide says its MLX integration supports text and images for Inkling 2 Mini and can use the checkpoint's native mixed-precision weights. The standard model uses a fixed three-pass loop, while a separate research variant allows experiments with the number of passes. Users who simply want to try the released model can follow the standard-model instructions without introducing that extra variable. [1]
MLX LM's own documentation provides command-line generation and chat tools, along with a Python API. Its examples use the –model option to select a compatible model explicitly. The documentation also explains prompt caching and smaller prefill steps for controlling memory use while reading long prompts. These are useful troubleshooting references after the Inkling-specific installation works. A generic quick-start command may select a different default model, so the command's model argument deserves a check before anyone judges Inkling's responses. [7]
Start with a modest context
Inkling 2 Mini supports a context window of up to one million tokens, according to its model card. That upper limit does not describe the amount of text a particular laptop can process comfortably. The card also identifies the model as accepting text and images while producing text. Users should separate those model capabilities from the limits imposed by their chosen runtime and available memory. [4]
Ollama's local FAQ offers a concrete diagnostic for readers using that runtime: ollama ps shows loaded models and whether they reside on the GPU, CPU or a mixture of both. The FAQ also documents changing context length. A useful first experiment is a short, familiar document with a question whose answer can be checked directly. Users can increase the document size after that works, recording memory use and response time. That process identifies a useful operating range without assuming the largest advertised context is the right default. [8]
Check useful work before trusting the output
Thinking Machines warns that looping adds computation without giving the model a proportionate increase in stored factual knowledge. Its release discussion identifies knowledge limitations as a reason tools can help. Readers should therefore judge a local assistant on the work they can verify, rather than its willingness to answer an unfamiliar question. A confident paragraph remains an output to check. [2]
For coding, a first evaluation could use a small function with existing tests and ask for an explanation before requesting a change. For document work, users could compare a summary against a short source and inspect whether names, dates and exceptions survive. An image test should use a picture whose contents are already known. These small checks expose different failure modes without requiring a benchmark suite. Users who record the model variant, runtime version and settings can repeat the same examples after an update and see whether the setup actually improved.
A test log should separate installation failures from wrong answers. If the model never loads, record the file selected, available memory and runtime error. If it loads but answers incorrectly, save the prompt and the expected answer before changing the setup. For a coding assistant, add whether the proposed change passed the existing tests and which test failed. That record gives a developer something specific to repeat after an update. It also makes a comparison between two model variants more useful, because both have faced the same work under documented settings.
Sources
[1] Hugging Face: Inkling 2 Mini local inference guide
[2] Thinking Machines Lab: Introducing Inkling 2 Mini
[3] Hugging Face: maintained source of the guide
[4] Thinking Machines Lab: Inkling 2 Mini model card
[5] ggml-org: Inkling 2 Mini GGUF file specifications
[6] Llama App: desktop chat and local API
[7] MLX LM: official installation and usage documentation
Have a correction or a useful perspective?
Get in touch with the publication.


