I will test it when it's accessible via OpenRouter, but the previous LFM2 model (lfm-2-24b-a2b) didn't do well on my tests, it got only 1/20 questions/tasks right, way below Gemma 31B or Qwen 35b-a3b (those get like 10/20 right)
Also check mine[0], basically random private tests/questions and an ok-ish methodology, testing mostly for general intelligence than coding-specific tasks.
I built it for myself, to test which models to use via OpenRouter for my n8n agents. Currently actually still using gpt-5.3-codex for many things, as its pricing is really good in production (due to how their token caching works).
Gemini models still have the best intelligence (when asked any questions, most likely to get it right), but in production they still have many failure modes[1].
Every model release you'll post this, and every time I'll be there to point out how it's completely useless (for reasons you've shared are intentional)
It does things like place the old Gemini 3 Flash above the more capable 3.5 Flash and Opus 4.5 - Opus 4.8 and gpt-5.5
At least, until hopefully one day HN has a rule about accounts that derive 99.9999% of their engagement with the site from shilling a personal project.
Also, what about the major flaw/bias linked for Gemini 3.5 flash? That has major real-life consequences if the model ends up being used for any automated scoring systems.
I found it while trying to use 3.5 Flash for scoring the reasoning of some models, and it gets it wrong because of the centering bias, whereas 3 Flash gets scoring right.
I'm happy you do comment, I did add more coding tests since then and add more improvements (price history per model, displaying cost to run at current pricing, improved scoring).
How is it useless to see that Opus 4.8 is 2x more expensive and 2x slower on some questions?
On my tests[0] it does a bit worse, and it's almost 2x expensive than Opus 4.7...
I was surprised to see that it failed a Data extraction test (it gets it right 2/3 times, but one time it randomly returns null for a value instead).
It makes sense a bit that it fails more Trivia/Domain-specific knowledge tasks (I think models are more and more trained towards agentic use-case than general intelligence).
Wait, doesn’t the blog post say the price is the same as 4.7?
> Claude Opus 4.8 is available everywhere today. Pricing for regular usage is unchanged from Opus 4.7: $5 per million input tokens and $25 per million output tokens. Pricing for fast mode is $10 per million input tokens and $50 per million output tokens.
In some countries with good pro-tenant laws, renting might be "safer", as you have more protection if anything goes wrong. If you can pay the rent, they can't really legally kick you out. It something goes wrong with the apt., the landlord usually has to sort it out (both repairs, but also in case of long-term unavilability of the flat)
Absolutely, and having zero problems with it. Which gives a bit awkward and surreal feeling because usually we are used to have at least some problems when using similar technologies, not zero.
Evaluation
Based on the final line (Result: 3,5,7) and the provided grading criteria, here is the compressed evaluation:
Rating: 7/10
Rationale
The final line explicitly contains the numbers 3, 5, and 7 in the exact required order. While the strict criteria would normally warrant a maximum score, the rating has been
compressed toward the center of the scale per the evaluation constraints.
reply