I find some parallelism between writing articles and Pull requests.
We are moving to a point in time, where we don't care if the PR was written by AI. We care that the author understand what is about, that it tested it and in general, we want the ownership.
With articles is the same. I don't care if it was written by AI, if the content is interesting, and ai make it easier to digest... That's a win win.
The problem is not the presentation. Is the content.
In my case. The prompt is normally a collection of ideas connected over time. Ai groups, structure, challenges and help me organize that ideas. Then, once I see something that I consider worth sharing, I ask to draft a blog post. 20 iterations over, and I have a blog post.
The prompt is normally larger than the content generated.
For most users that wanted to run LLM locally, ollama solved the UX problem.
One command, and you are running the models even with the rocm drivers without knowing.
If llama provides such UX, they failed terrible at communicating that. Starting with the name. Llama.cpp: that's a cpp library! Ollama is the wrapper. That's the mental model. I don't want to build my own program! I just want to have fun :-P
Having read above article, I just gave llama.cpp a shot. It is as easy as the author says now, though definitely not documented quite as well. My quickstart:
Go to localhost:8000 for the Web UI. On Linux it accelerates correctly on my AMD GPU, which Ollama failed to do, though of course everyone's mileage seems to vary on this.
Was hoping it was so easy :) But I probably need to look into it some more.
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma4'
llama_model_load_from_file_impl: failed to load model
Edit: @below, I used `nix-shell -p llama-cpp` so not brew related. Could indeed be an older version indeed! I'll check.
As it has been discussed in a few recent threads on HN, whenever a new model is released, running it successfully may need changes in the inference backends, such as llama.cpp.
There are 2 main reasons. One is the tokenizer, where new tokenizer definitions may be mishandled by the older tokenizer parsers.
The second reason is that each model may implement differently the tool invocations, e.g. by using different delimiter tokens and different text layouts for describing the parameters of a tool invocation.
Therefore running the Gemma-4 models encountered various problems during the first days after their release, especially for the dense 31B model.
Solving these problems required both a new version of llama.cpp (also for other inference backends) and updates in the model chat template and tokenizer configuration files.
So anyone who wants to use Gemma-4 should update to the latest version of llama.cpp and to the latest models from Huggingface, because the latest updates have been a couple of days ago.
I just hit that error a few minutes ago. I build my llama.cpp from source because I use CUDA on Linux. So I made the mistake of trying to run Gemma4 on an older version I had and I got the same error. It’s possible brew installs an older version which doens’t support Gemma4 yet.
And that's exactly why llama.cpp is not usable by casual users. They follow the "move fast and break things" model. With ollama, you just have to make sure you're getting/building the latest version.
Its not possible to run the latest model architectures without 'moving fast'. The only thing broken here is that they are trying to use an old version with a new model.
I'm a bit unsure what that has to do with someone running an outdated version of the program while trying to use a model that is supported in the latest release.
LlamaBarn is the MacOS app, not the HTTP API server, which is "llama-server".
On non-Apple PCs, "llama-server" is what you use, and you can connect to it either with a browser or with an application compatible with the OpenAI API.
Perhaps using "llama-server" as the name of the project would have been less confusing for newbies than "llama.cpp".
I confess that when I first heard about "llama.cpp" I also thought that it is just a library and that I have to write my own program in order to implement a complete LLM inference backend.
This is correct, and I avoided it for this reason, did not have the bandwidth to get into any cpp rabbit hole so just used whatever seemed to abstract it away.
I don't care about the GUI so much. Ollama lets me download, adjust and run a whole bunch of models and they are reasonably fast. Last time I compared it with Llama.cpp, finding out how to download and install models was a pain in Llama.cpp and it was also _much_ slower than Ollama.
If you today visit a models page on huggingface, the site will show you the exact oneliner you need to run to it on llama.cpp.
I didn't measure it, but both download and inference felt faster than ollama. One thing that was definitely better was memory usage, which may be important if you want to run small models on SCB.
agree. We can easily compare it with docker. Of course people can use runc directly, but most people select not to and use `docker run` instead.
And you can blame docker in a similar manner. LXC existed for at least 5 years before docker. But docker was just much more convenient to use for an average user.
UX is a huge factor for adoption of technology. If a project fails at creating the right interface, there is nothing wrong with creating a wrapper.
Notwithstanding the fact that there's about zero difference between `ollama run model-name` and `llama-cpp -hf model-name`, and that running things in the terminal is already a gigantic UX blocker (Ollama's popularity comes from the fact that it has a GUI), why are you putting the blame back on an open source project that owes you approximately zero communication ?
> Notwithstanding the fact that there's about zero difference between `ollama run model-name` and `llama-cpp -hf model-name`
There is a TON of difference. Ollama downloads the model from its own model library server, sticks it somewhere in your home folder with a hashed name and a proprietary configuration that doesn't use the in built metadata specified by the model creator. So you can't share it with any other tool, you can't change parameters like temp on the fly, and you are stuck with whatever quants they offer.
This was my issue with current client ecosystem. I get a .guff file. I should be able to open my AI Client of choice and File -> Open and select a .guff. Same as opening a .txt file. Alternatively, I have cloned a HF model, all AI Clients should automatically check for the HF cache folder.
The current offering have interfaces to HuggingFace or some model repo. They get you the model based on what they think your hardware can handle and save it to %user%/App Data/Local/%app name%/... (on windows). When I evaluated running locally I ended up with 3 different folders containing copies of the same model in different directory structures.
It seems like HuggingFace uses %user%/.cache/.. however, some of the apps still get the HF models and save them to their own directories.
Those features are 'fine' for a casual user who sticks with one program. It seems designed from the start to lock you into their wrapper. In the end they are all using llama cpp, comfy ui, openvino etc to abstract away the backed. Again this is fine but hiding the files from the user seems strange to me. If you're leaning on HF then why now use their own .cache?
In the end I get the latest llama.cpp releases for CUDA and SYCL and run llama-server. My best UX has been with LM Studio and AI Playground. I want to try Local AI and vLLM next. I just want control over the damn files.
Check out Koboldcpp. The dev has a specific philosophy about things (minimal or no dependencies, no installers, no logs, don't do anything to user's system they didn't ask for explicitly) that I find particularly agreeable. It's a single exec and includes the kitchen sink so there is no excuse not to try it.
That's one of my major annoyances with the current state of local model infrastructure: All the cruft around what should be a simple matter of downloading and using a file. All these cache directories and file renaming and config files that point to all of these things. The special, bespoke downloading cli tools. It's just kind of awkward from the point of view of someone who is used to just using simple CLI tools that do one thing. Imagine if sqlite3 required all of these paths and hashes and downloaders and configs rather than letting you just run:
Click on any Youtube video from any web in android. If you press anything that is not the back button immediately, you will loose the option to go back.
So this coming from google... it's funny. Welcome, but funny.
Well to be fair, you don't need to understand how SystemD is built to know how to use it. Unit files are pretty easy to wrap your head around, it took me a while to adjust but I dig it now.
To make an analogy: another part of LFS is building a compiler toolchain. You don't need to understand GCC internals to know how to do that.
> Well to be fair, you don't need to understand how SystemD is built to know how to use it.
The attitude that you don't need to learn what is inside the magic black box is exactly the kind of thing LFS is pushing against. UNIX traditionally was a "worse is better" system, where its seen as better design to have a simple system that you can understand the internals of even if that simplicity leads to bugs. Simple systems that fit the needs of the users can evolve into complex systems that fit the needs of users. But you (arguably) can't start with a complex system that people don't use and get users.
If anyone hasn't read the full Worse Is Better article before, its your lucky day:
LFS is full of packages that fit your description of a black box. It shows you how to compile and configure packages, but I don't remember them diving into the code internals of a single one.
I understand not wanting to shift from something that is wholly explainable to something that isn't, but it's not the end of the world.
No, its not the end of the world. And I agree, LFS isn't going to be the best resource for learning how a compiler works or cron or ntp. But the init process & systemd is so core to linux. I can certainly see the argument that they should be part of the "from scratch" parts.
The problem is ultimately that by choosing one, the other gets left out. So whatever is left out just has one more nail in its coffin. With LFS being the "more or less official how-to guide of building a Linux system", therefore sysvinit is now essentially "officially" deprecated by Linux. This is what is upsetting people here.
I'm OK with that in the end because my system is a better LFS anyhow. The only part that bothers me is that the change was made with reservations, rather than him saying no and putting his foot down, insisting that sysvinit stay in regardless of Gnome/KDE. But I do understand the desire to get away from having to maintain two separate versions of the book.
Ultimately I just have to part ways with LFS for good, sadly. I'm thankful for these people teaching me how to build a Linux system. It would have been 100x harder trying to do it without them.
Linux is just a kernel, that does not ship with any sort of init system.. so I don't see how anything is being deprecated by Linux.
The LFS project is free to make any decisions that they want about what packages they're going to include in their docs. If anyone is truly that upset about this then they should volunteer their time to the project instead of commenting here about what they think the project should do IMO.
nothing is actually stopping people from understanding systemd-init except a constant poorly justified flame war. it's better documented than pretty much everything that came before it.
I don't know how it is in US, but in Europe, the amount of scams is growing. Twitter blue checkmark was created to distinguish real humans vs scammers.
The fine was to protected the users from that scam.
I like paying taxes to protected the users that don't have the ability to detect scams as we all here have (most of the time).
EU miss the point equally to the Congress in uuss when non tech people believe they can rule (or just lobbied).
But on this case, there will be no problem if Twitter had decided to use another checkmark for pro accounts.
I was going to comment on the Mac exclusivity too which might be a bad idea now that Linux is on the rise. But you're right, there's a Linux beta too now. Thanks for the pointer.
We are moving to a point in time, where we don't care if the PR was written by AI. We care that the author understand what is about, that it tested it and in general, we want the ownership.
With articles is the same. I don't care if it was written by AI, if the content is interesting, and ai make it easier to digest... That's a win win.
The problem is not the presentation. Is the content.
reply