I think bun is moving to rust because Anthropic owns it and the devs there like rust. So why would they invest in another implementation? Sad to see a good zig example go, but as soon as Anthropic bought it I wrote the project off.
I am not aure either, but bun wasn't using normal zig and there was drama about upstreaming. Combine that with anthropics desire to show they can help rewrite everything in rust and that probably accounts for some of it.
They work better for coding workloads. Essentially, the more regular the output, the more the faster model gets right, the less the big model has to do.
Writing tends to have more false positives. I haven't tried this particular one, however, but that is the general trend.
Speculative decoding shouldn't actually change the accuracy of the response. The draft model drafts a couple tokens, and the inference framework verifies that the larger model would have picked them.
However, I've found that speculative decoders don't help much if you're running a model locally on limited hardware (for instance, my 32GB VRAM M1 Max from 2021). For one, you have to fit both the large and the small drafter model in memory. For another, if you're running a quantized model, the activation distribution is different enough that the draft model has a hard time guessing what's coming next.
My take is that speculative decoding is most useful on _very expensive_ prosumer/hobbyist setups where you have 128GB of VRAM and are running your local models with full fidelity. It's also helpful for inference providers where they can send output tokens at a computational cost slightly higher than their input token cost.
Your experience might be a bit dated, depending on when was the last time you tried it. MTP (which is a flavor of spec decoding) is showing really solid improvements on local models, even on consumer hardware.
In fact, as the article mentions, you get the biggest gains at low concurrency (so local should apply), with diminishing returns for higher concurrency (if you think in terms of unit of compute, it's probably better to serve more requests in parallel and get more throughput that way).
Eagle3 was great at low context tho, and this seems to improve things at high context. That's really cool, and hopefully it'll turn oout to be useful at those lengths. Eagle3 is also training dependant, so you could try training your own, if your use-cases diverge enough that 3rd party "generalist" models don't suit your needs. (in general nvda, redhat, etc. have provided general eagle3 models for popular families).
The reason speculative decoding shows diminishing returns in batched workloads is because the principle of both is the same.
Speculative decoding predicts a group of tokens and verifies this group using the main model in one pass instead of decoding each token separately. Eg. for this group, the weights are loaded from RAM per group instead of per token: roughly the same computation is performed but not the same memory movement (and other overhead like kernel launches).
Batching utilizes the same mechanism, so speculative decoding is essentially an attempt to batch a single stream using prediction. An attempt, because the verification may reject some tokens if the prediction was inaccurate.
Thanks, appreciate the info. For whatever it’s worth regarding recency, I’m testing the main llama-cpp branch that was pulled and built on 2026-05-25 running unsloth/Qwen3.6-35B-A3B-MTP-GGUF:Q4_K_M, my hardware platform is M1 Max 32GB VRAM. Is there a different fork or quant I should be using?
SpaceX has consistently launched ~90% of the mass to orbit for the whole planet Earth over the last several years[1][2]. There's no one else who could more credibly make such a claim.
They seem to have constructed a rocket with 10x the payload to LEO of the one they used to put those 10k satellites in orbit, and even demonstrated payload deployment. So I'd say 100k looks do-able for them today.
10x that seems aspirational, but not comically so. Folks hate Musk, but that seems to cause them to not see the engineering going on in front of them.
> They seem to have constructed a rocket with 10x the payload to LEO of the one they used to put those 10k satellites in orbit
They seem to have constructed a rocket that consistently gets heavier and more complex and more expensive and farthrt behind schedule and hasn't demonstrated specified payload.
I checked the publicly released stats over Starship's development, and this is what I found: compared with the initial ~5,000 t / ~73.5 MN concept, the latest V3-class Starship/Super Heavy is trending toward roughly 35%+ more loaded propellant mass and about 40% more maximum liftoff thrust if you use the FAA’s ~103 MN figure. Payload capability has also moved upward from the early 100+ t reusable LEO baseline to SpaceX’s current public claim of up to 150 t fully reusable and 250 t expendable.
Starship has never met claimed specs and capabilities. It is so far behind schedule that it won't meet specs in time to remain relevant. Which is a generous way of saying it never will.
Agreed. They're already stretching starship. And there's long been talk of a wider version yet. Starship is already pretty impressive considering it's just about exactly the size of Sea Dragon.
While true, this is insufficient to make the new claim credible. If the proposed satellites only weighed 100kg and remain on orbit for 3 years, to keep a million up requires:
They've been approved for 44 Starship launches from Kennedy Space Center in Florida, and are aiming for 160 total launches in 2026. They've recently purchased a giant tract of land in Louisana to build a third starport. 222/year is looking doable.
At this point, 160 Starship launches in 2026 would be close to every weekday.
They already have three launch sites for Falcon and can't do 200.
(Also see edit, my first post relied on Apple's autocomplete for maths and it used a short ton, plus point about these numbers corresponding to a mere 100 kg per satellite).
The 160 launches figure includes falcons. Seems like Starship fuels and flight tests faster than Falcon though. And if they manage to reuse second stages, then that eliminates a significant manufacturing bottleneck.
If you're counting Falcons, you are making my point for me: even with those, on three launch sites, they still can't get close to the minimum for an extremely small, to the point of being unreasonable, target satellite mass.
Further, until they actually do solve upper stage reuse, it is an "if" which can kill the economics of the vehicle itself, let alone reach the eventual potential cost reductions necessary for space based data centres to be worthwhile.
I don't see any reason a non-renewable Starship upper stage would kill the economics of the vehicle. No one else has a renewable upper stage yet, so there's no competition in that space until someone else does. Stoke have an interesting design but it hasn't flown yet and is only about the size of Falcon.
If they do manage to reuse the upper stage, then they should have no problem exceeding falcon launch cadence. Starship is much easier to build than Falcon. Welding is simpler and less expensive than the carbon composites used on Falcon upper stages.
The competition isn't other launch providers, it's not going to space at all.
According to Google, the price threshold to make space make more sensible than building on the ground is $200/kg: https://arxiv.org/pdf/2511.19468
Without full reusability, the estimated cost for Starship to LEO is kinda hard to find (necessarily, given the design isn't yet finalised), Wikipedia says $100m/launch in expendable mode, and the SpaceX website* says 250 metric tonnes in expendable mode, which is $100e6/250 metric tonnes = $400/kg.
Yeah I remember reading that what killed the space industry in the 90s-2000s other than the collapse of the USSR and cessation of great power competition was the massive move to digital communications, particularly satellite TV - which mean that a smaller number of satellites could serve the expected demand.
Depends on who's making the call for who gets cut. A key part of decimation was that the doomed soldiers were beaten to death by their comrades to give the remaining 9 a bloody, lasting impression of their dishonor. If Meta makes everybody sit in a group with their ten closest coworkers and debate until they decide who gets cut it's a lot closer to decimation than if management suddenly shuts off 10% of employee computers.
Yesterday I actually read the docs on Open WebUI. It’s pretty impressive all the features it supports. I will give thunderbolt a look but I’m seriously considering doubling down on Open WebUI. The only thing I don’t like about their features are that tools and functions can’t be offloaded to external compute, it runs on the same machine.
You got it right I think. I’m sitting with two “AI Ready Radeon AI Pro 9700 workstation cards, which are RDNA4 not CDNA. My experience is that my cards are not a priority. Individual engineers at AMD may care, the company doesn’t. I have been trying since February to get ahold of anyone responsible for shipping tuned Tensile gfx1201 kernels in rocm-libs, which is used by Ollama.its been three weeks since I raised enough hell on the discord to get a response, but they still can’t find “who” is responsible for Tensile tuning, and “if” they are even going to do it for the gfx12* cards.
Yeah I own an AMD Instinct MI50 and i need to patch all of my applications to work, like PyTorch, bitsandbytes, blender etc, while Nvidia cards from the same generation are still mostly supported. But the better value and hardware are worth it
reply