Rendered at 21:43:56 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
NitpickLawyer 1 days ago [-]
Supposedly this is a Kimi k3 rival. Bit of a chonker, especially since they only released bf16 and fp8. So at launch this will be harder to serve than k3. No QAT on q4 means that someone with deep pockets (nvda?) will have to quant it, with plenty of calibration data. Should bring it ~1.3TB, so around k3 size.
License pretty similar to k3 with some caveats. Free to use for internal or <50M$ revenue / year. Limitations above that threshold for serving the model or services targeting coding / productivity agents.
Benchmarks are looking good, trading blows w/ opus4.8 and sol, generally 10-20p under fable. But that's neither here nor there w/ qwen, their benchmark to real world usage correlation has been iffy in the past.
The local model 3.8-27B announced for Friday, same time so ~48 hours from now. That'll be a bit more exciting for a lot more people, since 3.6 was quite good for local inference, and their 3.7-max -> 3.8-max shows a lot of improvement.
tinco 10 hours ago [-]
From my evaluation[1] neither Kimi K3 nor Qwen 3.8 are as good as GLM 5.2 at coding. I wonder if there's a marketing gap that's got people underestimate it.
It uses twice as much tokens to achieve the same but the results are significantly better and because it's so much cheaper it's the most economical choice too.
I had several issues with unsloth gguf, even for models released a few months back like gemma 4, I have 0 confidence in their models, at this stage, I feel several uncensored are more reliable.
danielhanchen 23 hours ago [-]
Hey sorry what are the problems that you're experiencing - we're more than happy to help fix them!
LeBit 21 hours ago [-]
Thank you for what you are doing.
danielhanchen 19 hours ago [-]
Thanks for the support and to the community!
SwellJoe 17 hours ago [-]
Counterpoint: I've been using the Unsloth Gemma 4 quants extensively since very soon after release (on ROCm and Apple Silicon, I don't have any Nvidia hardware big enough), pretty much every quantization down to 4 bits (the QAT is the business, indistinguishable from the full-fat version, runs great on a slightly chonky desktop or laptop), and I haven't had any issues. The reason I use Gemma 4 so much often comes down to how reliable it is; when I want to experiment with llama.cpp settings, MTP, n-gram, etc. it's my go-to because I know there isn't anything wrong with the model or the quantizations that could interfere with the experiment.
It did take a little while for Unsloth to update the Laguna S 2.1 quants to fix the yarn_attn_factor, and so it was a bit frustrating getting that quantization running right, but almost always, I pick the unsloth quantization if there is one. (Still waiting/hoping for a Ling 3.0 Flash.)
danielhanchen 14 hours ago [-]
We will investigate Ling!
BonerWiener 14 hours ago [-]
> I feel several uncensored are more reliable.
I have had the same experience with gemma 4 on same tasks being refused. But this is when working with cyber offensive tasks and the like. It excels in coding and is very fast on consumer hardware. So I would say use the right tool for the right task.
What uncensored models can you recommend?
segmondy 1 days ago [-]
... because they are often the first to quant it. sometimes the actually model providers will release wrong chat templates or values in the model config which leads to bad quants. how would you know a quant is good if you don't make one? you don't. so they make it first, then they run a lot of tests, KD, perplexity, etc, they publish it. They take feedback from the community, then they update if needed. if you want to try it right now, you grab it else wait for a week or 2.
Foobar8568 1 days ago [-]
Gemma4 was released a few weeks ago? The problems are still there. Today I started using another "provider" and the problems disappeared. Thanks but no.
danielhanchen 23 hours ago [-]
Hey yes - if you could describe what the issues are - we will gladly fix them!
Foobar8568 5 hours ago [-]
Mea-culpa, dry-multiplier generated crap and even more so on Gemma4.
arcanemachiner 1 days ago [-]
They are very responsive, and would probably be happy to help you fix your issues.
segmondy 23 hours ago [-]
I've 0 issues with gemma4 and I downloaded it early.
LeBit 21 hours ago [-]
Same. Used the 12B, 26B A4B and 31B.
No issues with llama.cpp.
chlorion 19 hours ago [-]
Huh I have had great luck with unsloth quants so far. What issues are you having?
jacquesm 11 hours ago [-]
Without substantiating what issues you have with unsloth gguf files you are just adding noise, no signal.
jokethrowaway 24 hours ago [-]
I've used their gemma 4 quants since when they were not still working in llama.cpp and ik-llama.cpp and I don't remember any problems
They are the most reliable in my experience, but if you have alternatives you trust I'd love to know
codedokode 1 days ago [-]
I wonder who is unsloth and where they got time, hardware and knowledge to quantize them?
NitpickLawyer 1 days ago [-]
unshloth started as a finetuning library with lots of optimisations so you could finetune on lower end hardware. Kind of OGs of the local community. Started by two brothers Michael and Daniel(?) a math wiz and a community builder/communicator. They've since gotten some VC backing, are active in quantising lots of models on release day (work w/ labs to prepare things), known for their optimised quants (use different bits for different layers). Recently I saw they launched some sort of a desktop app, like lmstudio if you're familiar with it. They're really cool people and known in the local model places.
They started with offering training methods for quantized models to save memory and added new things over time. They are very active in the local model community and have extensive documentation and tooling to help with running and training models locally.
kittikitti 1 days ago [-]
I quantize my models with llama.cpp and it's usually one command. Some of their quants are fine-tuned by architecture but it's only to squeeze out every little performance benefit.
suprjami 24 hours ago [-]
Unsloth imatrix data puts their quants at lower KLD than almost all others.
It's true they make architecture-specific changes like keeping certain layers at F16 but it's also more than that.
unleaded 23 hours ago [-]
What hardware would you even be able to run this on?
adrian_b 15 hours ago [-]
Even the lowliest hardware could run this, but at an unlikely to be useful low speed, e.g. of 3 or 4 tokens per minute (by reading the weights from a couple of 4 TB SSDs for the BF16 model, or from a 4 TB SSD for the FP8 variant).
The question about LLMs is never whether they can be run, because that has a trivial answer, they can always be run. The right question is what speeds are achievable for representative hardware configurations.
At launch, it is difficult to estimate the speed. That should be known after someone reports experimental results. Moreover, for many LLMs the speed improved sometimes later after their release, after tweaks in inference backends, like llama.cpp or vLLM.
badcafe23423435 12 hours ago [-]
tell me how run 35B on my 8GiB VRAM
(linux)
speed is not problem when You run agents and forget for 2-3 days
zepearl 24 hours ago [-]
The parameter "reasoning_effort" is something new, or am I wrong?
Qwen3.8 comes with official support for reasoning_effort, which can be used to adjust reasoning depth and control cost:
- xhigh (default): for complex tasks demanding thorough analysis
- medium: balancing accuracy and speed
- low: efficient reasoning optimizing for speed and cost
In addition, preserve_thinking is enabled by default for all workloads for the best out-of-the-box experience.
Asking because in my case (OCR of scanned historical "National Geographic" magazines) the LLM trying to merge text split into separate columns was running in circles from time to time and needed a lot of prompt tuning when using Qwen 3.0/3.5/3.6 (still needs from time to time).
philipkglass 23 hours ago [-]
I'm using Qwen 3.5 for OCR, and reasoning_effort is supported there too. I found that it can be loop-prone (though somewhat less so) even if you set reasoning_effort to low.
alanwreath 1 days ago [-]
> 3.8-27B announced for Friday
Maybe I’m misreading this or some other post, I thought QWEN was stepping away from releasing these models for local consumption
Sadly they seem to not be releasing a sparse 35b A3b or anything inbetween "too large to host for mortals" and "fits into a consumer rtx". Probably not to eat away their profits on their API serving. 120b - 300b is a dead space right now, very few good releases in that size range. (I know there are, but the big labs aren't releasing stuff here)
NitpickLawyer 1 days ago [-]
3.8-27B is confirmed for Friday. They didn't release their whole 4B-400B range of models since 3.5. And 3.6 only got 27B and 35B MoE. So yeah, slowing down, but not completely out of the small model game.
trollbridge 1 days ago [-]
They reversed course and now are saying they'll be releasing their Max style models in open weights.
arthurcolle 1 days ago [-]
thank you china!
FooBarWidget 1 days ago [-]
Rather, thank you competition in China. This is Chinese "overcapacity" (of talent pool) at work.
embedding-shape 1 days ago [-]
Quite literally the government in China came out and said "It'd be better for us if we did more open models and collaborated with other countries also doing open models" and then Qwen changed their tune. It's literally thanks to China in this case, not competitors/peers in China. Their government is horrible for a lot of stuff, but in this case they do deserve praise for forcing the "right" (according to me) direction.
matheusmoreira 20 hours ago [-]
Realpolitik I guess. They want to destabilize the USA, and we want open weights we can run on our own machines. As long as those interests coincide, we are allied.
The best outcome for us is the one where they all keep competing and undermining each other until the end of time while providing us all with better models and cheaper hardware to run them with. The US corporations in particular should never be allowed to achieve their "you'll buy intelligence from us on a meter" rent seeking dream.
vintermann 17 hours ago [-]
The US is destabilizing itself plenty on its own. I think this is just a question of common sense, and the party's long standing habit of pushing some competition but not too much competition, to avoid wasted effort.
arthurcolle 20 hours ago [-]
when Mistral finally releases Le Chaton Fat, I will switch to that but until then, I will count my lucky starts that this exists. Cheers to Qwen3.8-2.4T release day
Zetaphor 17 hours ago [-]
Unfortunately they're currently too busy trying to patent tool calling
Our desire for better local models just happens to coincide with China's desire to destroy the western AI company business model by releasing local models. I doubt there's any philanthropy involved.
horacemorace 21 hours ago [-]
Which coincidentally helps average people far more than the already rich investors in a few western mega corps.
If it weren’t for these big Chinese model releases the western companies wouldn’t release anything at all. The field would be advancing at a snail’s pace.
Manfrednotfunny 12 hours ago [-]
Its not just that.
China also puts pressure on rich chinese flaunting their riches.
They have a common prosperity initiative.
FooBarWidget 15 hours ago [-]
I find it really weird why people keep framing decisions like this in terms of morals and selflessness, e.g., "is/isn't philantropy". This is about relationships and mutual benefit.
They literally announced their motivations and world few a few weeks ago at the Shanghai AI conference. They want to ally with the global south. They see AI like the industrial revolution: the global south was left behind for a long time and, as a result has been exploited and has struggled to develop for a long time. They see open AI as a way to level the playing field to prevent such "new historical injustices" (in the sense of the Century of Humiliation and the Opium Wars). Concrete policies to back this rhetoric include technology transfer and training programs for the global south. They frame this latter not as philantropy but as generosity, in the sense that it generates goodwill and what goes around comes around. They believe that helping the global south and cultivating relationships will eventually help China.
Think about it. Your local businesses are not charities either. That doesn't make them bad, nor does it mean you derive no benefit. It still benefits you to cultivate good relationships with them.
yfontana 14 hours ago [-]
China has been using that rhetoric of "helping" the global south ever since it emerged as a super power. In most cases, it has a lot less to do with generosity than with securing natural resources and international influence.
embedding-shape 11 hours ago [-]
> In most cases, it has a lot less to do with generosity
Nothing happens on a geopolitical scale, from the US, China or anyone else, simply because of generosity.
FooBarWidget 13 hours ago [-]
Generosity and securing natural resources and international influence don't have to be mutually exclusive. If you interview Africans, then they say that while cooperation with China has problems, they sure are glad they at least have more choices now, and on the whole China's existence is beneficial. The alternative — only western choices and no China — sure hasn't served them well in the past half century.
No matter what you believe is their "true" intentions, offering 5000 training and tech transfer positions to the global south is a very concrete and unambiguous move. As are forgiving African loans and unilaterally offering zero trade tariffs.
yfontana 9 hours ago [-]
If what China has been doing is considered generous, then so should the tens of billions in foreign aid that the West has sent to the South over the years.
I'm not saying that China's investment in the South hasn't had positive effects. But "generosity" is rarely a relevant lens when analyzing international relations.
FooBarWidget 8 hours ago [-]
> then so should the tens of billions in foreign aid that the West has sent to the South over the years.
Well, yes? Why does it have to be either-or?
> But "generosity" is rarely a relevant lens when analyzing international relations.
Automatically assuming nefarious intentions behind all moves is also rarely a relevant lens.
And as I said, and I'm not sure why you keep ignoring it, but I define "generosity" in the sense of mutual benefit. Being nice to your neighbors and helping them, benefits you due to generated goodwill. I'm explicitly not defining generosity in the sense of selfless philanthropy where you get nothing back. The idea that doing good things for others eventually results in good things coming your way, and thus that one should do good things for others even it's selfishly motivated (and also that there's nothing wrong with this), is not a crazy idea.
In a lot of cultures (Chinese included), gifts are not simply gifts. There is the social expectation that the gift is reciprocated. Western cynicists may call this "manipulation" or "influence". The Chinese see this as the start of a relationship of a cycle of mutual gift giving.
bsder 20 hours ago [-]
> Their government is horrible for a lot of stuff, but in this case they do deserve praise for forcing the "right" (according to me) direction.
It's not just according to you.
Without open weights, what happens if you get blacklisted from Anthropic and OpenAI? If AI becomes a standard tool for programming like a compiler, you've effectively been Blackballed from the field of programming. Full Stop. This is "Right to Read" coming home: https://www.gnu.org/philosophy/right-to-read.en.html
In addition, without open source competitors to your core tools, we KNOW what happens. Cadence and Synopsys and a megabuck per engineer per year ... that's what happens.
jokethrowaway 24 hours ago [-]
The obvious goal is to destabilize the western economy and prove that US tech is a worthless bubble - but I agree, OSS AI is great for everybody and what OpenAI was supposed to be
DrBenCarson 22 hours ago [-]
There’s an alternate universe in which OpenAI stays open, licenses according to revenue, Chinese models don’t gain traction in the US because domestic models take all the capacity…whatever, $1T IPO beats the right answer ever time
ethbr1 19 hours ago [-]
> Chinese models don’t gain traction in the US because domestic models take all the capacity…whatever, $1T IPO beats the right answer ever time
The trouble here is how more infrastructure helps OpenAI and Anthropic continue billing at 10/100x Chinese model rates.
Either their models have to be better (to justify the higher prices and margin) or their inference has to be lower cost (which isn't going to happen until they move away from Nvidia).
arthurcolle 15 hours ago [-]
Black market operators can resell stolen account tokens at a lower price than authentic premier tokens from frontier labs and can host their own infra too. I'm not super convinced frontier model serving without downstream model development on a vertical specific software / knowledge worker "factory" model can work
FooBarWidget 15 hours ago [-]
And what do you think happens after the govt said that? Going to companies and force them to open models? That's only how westerners' misconception of China works. If you study Chinese EV industrial policy history you'll know they're not based on coercion of private parties but on incentives (and that, ironically, private EV companies succeeded despite incentives, not because).
The real policy mechanisms around open AI models are also incentives. Various cities have programs to pay companies for releasing open models. They subsidize compute through vouchers. They reward universities and students for open source collaboration.
This isn't some black box. The policies are written down, anybody can read their AI+ policy papers.
Alibaba went back to releasing open models way before the Xi speech from a few weeks ago. The cause is pressure from researchers, who believe in openness, as well as the competition who keeps releasing open models. This is Chinese "involution" at work. And the subsidies also help, of course.
ngl999 18 hours ago [-]
that "collaborate" means US GPUs and training set, then deployed in censored data centers for profit
PerkFuel 1 days ago [-]
[dead]
1 days ago [-]
MrDrMcCoy 1 days ago [-]
Llama.cpp can quantize without special training, but I'm not sure if any special model architecture support is needed to read it in the first place. If it can be converted to gguf at all and you know what tensors to target, it can get the full ternary bonsai treatment today.
NitpickLawyer 1 days ago [-]
Sure, but that's for "personal" serving. I meant for 3rd party providers. Usually we get a good indication on what it costs to host this, as the prices settle on open router. That's why I said it's tougher to serve than kimi k3 on launch. As a provider you'd do fp8 if the model creator didn't do QAT on q4, or until someone does a good calibrated nvfp4. And that's usually nvda :)
MrDrMcCoy 1 days ago [-]
That makes sense, but your specific phrasing precluded the possibility of non-QAT quantization.
NitpickLawyer 1 days ago [-]
Should have worded that better, my bad.
binary132 1 days ago [-]
QAT is an optimizing quantization algorithm, not naive quant.
dragonwriter 22 hours ago [-]
Isn’t QAT a training approach (roughly, simulating quantization in the forward pass during training so that quantization of the level targeted in training has close-to-optimal behavior), not a quantization algorithm? Hence, the name?
binary132 3 hours ago [-]
sounds like a repeatable method of optimizing quantization to me, friend
MrDrMcCoy 1 days ago [-]
Right, but the way they phrased it suggested that without QAT it could not be quanted at all.
badcafe23423435 12 hours ago [-]
Show me any open source model quantisite, distile or reduce size from nvidia ;)
esafak 1 days ago [-]
Now that they have reached the frontier in raw performance, I would like to see Chinese models improve their reasoning efficiency.
JSR_FDED 24 hours ago [-]
I bet they could do it faster if they weren’t blocked from buying GPUs
verdverm 1 days ago [-]
For all the talk about over reasoning, K3 on low thinking has been rather nice
esafak 1 days ago [-]
It looks like a work horse! Is it yours?
jingpostmedia 1 days ago [-]
[flagged]
verdverm 1 days ago [-]
quanting is actually cheap and you can compress a model that does not fit on a GPU. You can process layer by layer, this is what the sequential processor in llm-compressor does.
dhx 1 days ago [-]
Also of interest: DeepSeek V4-Pro-0813 (1.6T-A49B) benchmark scores have apparently just been announced on the DeepSeek WeChat channel and they're sitting about Fable 5 level.[1]
Isn't this quite a bit behind Sol and Fable and even ChatGPT 5.5 xhigh and Opus 5 max?
In terms of what you get for what you pay for, it's incredible - probably by far the best.
But unless I'm reading things wrong, it does not appear to be top-of-the-line.
neosat 1 days ago [-]
This may not 'quite a bit behind' those at all. If you look at the benchmark numbers they are very comparable to Fable, but beyond a certain point the benchmark numbers don't tell you much. Opus #5 beats Fable on some benchmarks but given similar cost almost everyone who has used those two models will prefer to use Fable.
At this price range $0.87per 1M they will get a lot of usage of people trying it out. Given the benchmark numbers, for many people and many use cases this will become their primary driver. There are people and use cases where Fable, Sol will work better but those are likely not the target of DeepSeek anyway.
In terms of performance and price pareto curve I don't think any model can beat this today (though openAI is doing some exciting recent work in efficiency) - which is a remarkable feat for the DeepSeek team.
Either way, what a time for consumers of these models :)
SwellJoe 17 hours ago [-]
I stopped picking Fable because it refuses based on guardrails so often. I do a lot of security related work, and Fable just won't do any of it. So, I don't bother. Unfortunately, Opus 5 also refuses quite a bit of security work, now, as well, so my Anthropic subscription becomes less useful by the day. Fable may be better, but if it won't do the work...
DeepSeek and Kimi K3 will happily do security work, and they do it pretty well.
vinnymac 8 hours ago [-]
Same, I have gotten good results out of Opus 4.6,4.7,4.8 for my security work though. So I continue to use them for this.
Curious if you’ve find yourself enjoying DS or Kimi more than Opus 4?
SwellJoe 4 hours ago [-]
DeepSeek is more fun, because it's cheap-as-free, Good Enough, quite fast. I use it for all API stuff, automated runs, testing of the security auditing harness and benchmarks I'm working on, etc.
Kimi K3 is smarter, though. At least smarter than DeepSeek V4 Flash 0731. I haven't tried the new Pro version, but will this weekend when I'm working on my personal projects But, K3 has been what I've been using for the actual coding of the harness and such (after Claude models, and then OpenAI models, began refusing to do that work). K3 is very expensive, though. Much more expensive than pretty much everything except Anthropic, and their subscription plans are stingy.
I also like Reasonix quite a bit, as an agent harness, though Kimi Code is also very good. I guess I'll try out the new DeepSeek official harness, as well.
dhx 5 hours ago [-]
Now that people have played with the model for a bit, there is one reported real world win for DeepSeek v4 Pro 0813 that is perhaps quite consequential given the drama in the US about Mythos.
In a benchmark, DeepSeek v4 Pro 0813 found 87.5% of selected real world software vulnerabilities publicly reported and with CVEs assigned, which is above runner ups Opus 5 and Qwen 3.8 which both found only 81.3%. However there is a downside to this--DeepSeek v4 Pro 0813 is less accurate with a 35% false positive rate versus GPT-5.6-Sol's 15% false positive rate. For vulnerability analysis though, it's probably worth finding that one extra vulnerability no other model has found even if requires significantly more triage to remove false positives, or additional cost to run every vulnerability detection through other models to verify.
Does that mean they're using the new pricing now? I no longer see the warning/notice about "Things are about to get a lot more expensive soon" on https://platform.deepseek.com/usage anymore, so I guess yes?
ive not paid any attention to this space much, is deepseek referring to paid-sub service like claude/gemini/etc ? or is this local llm
pinkgolem 19 hours ago [-]
deepseek is a lab, the models are open weight(atleast after a bit)
they have a somewhat selfhostable model(flash), but are mostly known for having super cheap api access
no zdr ofcourse.
kadoban 17 hours ago [-]
> no zdr ofcourse.
Since the models are open, there's other infra providers who have different policies.
pinkgolem 5 hours ago [-]
Yes, but those are more expensive normally
trueno 10 hours ago [-]
any recs?
kadoban 3 hours ago [-]
Tbh I just go through openrouter because it's easy, so you can just go down the list in that and see who sounds good. I honestly don't spend much time at it.
trueno 10 hours ago [-]
i see. appreciate the reply this was concise and just what i needed
The 1bit quant model is at an astonishing 397GB with 95B active per MOE. This literally puts Opus 4.5 performance level into a machine a normal person could buy, and still gets usable tokens/second.
The full lossless model BF16 is clocking at 4.9TB. The model card claims the model to be between Opus 4.8 and Fable 5. Again that's astonishing as getting a machine with 7TB RAM (with context + KV cache) is still within the realm of medium size companies.
Bad things: The open source version has its vision capability removed, and the context capped at 250k . I expect someone to bolt a Kimi 2.6 vision tower to it to restore the vision capability (at less performance of course). For context, I played around with extending the context to 600k for Qwen 3.5 397b, and the context remained stable up to around 480k. It'd be interesting to see if the same can be done to Q3.8 .
Also no out of the box DSpark/DFlash support. MTP is present so we should at least get some boost in TP speed.
ilc 1 days ago [-]
To compare a 1 bit quant to the full fat model is misleading.
Honestly this model people at home can tinker with, if you have a big enough Mac. Maybe 4 Strix Halo/DGX Spark, and then at 1 bit quant? Nah.
Use the right sized model, for your hardware. You'll get better results.
guardiangod 1 days ago [-]
Extremely large 1 bit models are usually within 50-60% of KV divergence to lossless models. In this case I think the comparison to Opus 4.5 is a fair assessment.
Extremely large models don't suffer as much from quantization due to its weight topology also contains encoded information, so the loss of info from any one weight is somewhat mitigated.
dist-epoch 1 days ago [-]
KL divergence (you misspelled it) doesn't tell you anything about capability drop - how much did this particular benchmark (thus ranking among models) change after 10% or 50% KL divergence?
frotaur 13 hours ago [-]
Not only that, but KL divergence is not a '%'. It's just a number ranging from 0 to infinity that tells you the 'distance' between two probability distributions.
ilc 1 days ago [-]
Any one weight, but all of them. And also crushing the architecture itself?
I wouldn't pick up 400gb of hardware to run in that mode. I might try it for fun, but even then you are looking at handling a 95GB active parameter set.
This is NOT a model for most home labs. I'm sure some can and will use it. But most, should steer clear.
kadoban 17 hours ago [-]
I wouldn't just rush out and buy hardware, but there will be benchmarks after a while to make an informed decision.
95GB active is not _too_ bad, would require some creativity and $$, but I bet I could do that at home for less than a cheap car.
pil0u 1 days ago [-]
I don't understand the logic behind model sizes and quantization.
Suppose I have 100GB of unified memory, how should I know which model suits it best? I understand how a 2.4T model wouldn't fit, but I don't understand the impact of quantization and whether I should use a 200G model quantised to fit say 90GB of memory, or a non-quantised 90G model.
numpad0 1 days ago [-]
Usually the largest Q4 model that fits and has best reputations. Usually the performance degradation is not considered tolerable below Q4. Usually the model of choice ends up being either Qwen 3.6 27B or 35B-A3B.
What's weird about local LLM models is that closed door improvements in training/RLHF dataset have been so significant that it's rare for larger but older models to make sense - everyone seem to always hard switch to the newest one and report step changes in capabilities(or maybe people running Kimi K2 since release just don't talk about it on the public Internet, giving me that impression).
silentbob7 16 hours ago [-]
It depends on your usecase and the size of your ram is not the only driver. I think the primary performance drivers (without sacrificing precision) right now are QAT, MTP/DFlash, MoE and Hybrid approaches to avoid full attention in parts by replacing with linear / sparse attention.
So Gemma4 26B A4B QAT+MTP (from unsloth) would be a good pick atm. I'd love to see some smaller models with all that features.
NitpickLawyer 1 days ago [-]
It really depends. It used to be easier to have a rule of thumb, but now it's not clear anymore. Now there are a lot of things to consider, such as a model's kv efficiency (how much context you can fit), MoE v. dense, QAT or not (Quant aware training) and so on.
The old rule of thumb was that a lower quant of a larger model > higher quant of a smaller model. That being said, for some things going lower than fp8 will see a lot of degradation in generation quality. Except if the model comes with QAT 4bit quants. Then there's also nvfp4 w/ calibration data, which also can improve things. So it's really not easy to tell "at a glance" you'd have to test them yourself on your hardware.
onlyrealcuzzo 1 days ago [-]
Standard models are designed to quantize down to 4-bits relatively well.
Anything below that, and especially 1.58b - is typically complete garbage, and you're much better off running a model 100x smaller at regular precision (compared to one 7x smaller quantized into complete garbage).
If the model was designed specifically to quantize down to 1.58b, then it's different.
AFAIK, there's no large models designed for this yet.
richardfey 1 days ago [-]
> If the model was designed specifically to quantize down to 1.58b, then it's different.
> AFAIK, there's no large models designed for this yet.
Isn't BitNet b1.58 2B4T what you are looking for? (haven't tried it myself though)
onlyrealcuzzo 1 days ago [-]
2B is pretty small...
No 100B+ param (certainly no 2T+ param) models have been trained natively to quantize down to 1.58b.
codedokode 1 days ago [-]
Usually 4-bit 200B model is better than 8-bit 90B. But if you go below 4 bits, I am not sure what is better.
markasoftware 1 days ago [-]
There's no rhyme or reason to it. Quants aren't benchmarked much. Generally 4bit better than smaller model 8bit
auspiv 1 days ago [-]
Opus 4.5 level of performance is also accessible with deepseek-v4-flash-0731 (0731 being the july 31 update) which is much, much, much smaller. 2x RTX pro 6000 blackwell can run it. 4x can run it very comfortably
guardiangod 1 days ago [-]
I am running DS v4 flash 0731 lossless at 80t/s right now. It really is not at Opus 4.5 level (for my workload). I would say it's around 3.7 Sonnet, which is still pretty good, but other models such as GLM 5.2 are still leaps better. Of course I run DSv4 flash over GLM 5.2 for a few very good reasons, but intelligence is not 1 of them.
MrDrMcCoy 1 days ago [-]
Despite fitting into VRAM, I can't get DSV4 to run at usable speeds on my AMD hardware. The upcoming qwen3.8 27b greatly excites me, and I hope it can outperform Stepfun 3.7 Flash, which is the best thing I can run today.
kennywinker 17 hours ago [-]
I'm just trying out Muse-Glimmer 30b, and my initial vibe is this might be better than qwen3.6-27b. No idea how it compares to Stepfun, because I can't run that model - but worth checking out while you wait for qwen3.8-27b
Philpax 1 days ago [-]
What do you need the extra 2 for? Tensor parallelism?
arjie 1 days ago [-]
Longer context and more cache. The problem is that native format with DSpark enabled you have very little room on the VRAM.
Philpax 1 days ago [-]
I was under the impression that you could fit the full 1M context within the 192GB VRAM as a result of DeepSeek's various architectural advancements, but I'll grant that DSpark + a larger pool for concurrency may necessitate more VRAM, yes.
cbeach 1 days ago [-]
Anyone thinking of buying 2x RTX Pro 6000 Blackwells - beware: unlike other cards e.g. RTX 5090, The RTX Pro 6000 cards cannot be NV-Linked, so you'll be going through the PCIe bus instead (7x higher sync cost)
ThunderSizzle 22 hours ago [-]
My understanding is the last consumer card that supported that was the 3090. A Google search seems to agree the 5090 does NOT support NVLink...
wolttam 1 days ago [-]
Opus 4.5, even 4.6-level performance has been around since July 31st in 284B total params and just 160GB of weights at native FP4 quantization- DSv4 Flash.
ekianjo 1 days ago [-]
> The 1bit quant model i
at this kind of quantization is it useful though?
l72 1 days ago [-]
> In particular, Qwen3.8-Max is the official version based on Qwen3.8-2.4T-A95B with more features, such as vision input & non-thinking support, 1M context length by default, official built-in tools, etc.
That is unfortunate, that the open weight model doesn't have vision support or the 1M context length...
wren6991 1 days ago [-]
People have had surprising success adding vision to open-weight LLMs that ship without it, like DSV4 Flash [1] or GLM-5.2 [2]. Given this model is already vision-trained I expect that approach will work well here.
Qwen3.5 was awesome: fairly open and fully featured. 3.8 lacking vision, nerfing thinking modes, and low context length feels pointless.
edg5000 17 hours ago [-]
Shall we bet on when the hardware needed for this (without quantizing and at good speed) will reach < 10k USD? I'm betting 2040. I can download it now, and then get the hardware later. Eventually we can all have these things running 24/7 in our home if we wanted to. I currently would not have any task for it that would really utilize the hardware 24/7, but maybe in 20 years I will.
ak_t 15 hours ago [-]
I think it is more likely that a smaller model (<400B) with similar intelligence gets developed long before the hardware to serve a 2.4T model gets cheaper than 10k.
edg5000 13 hours ago [-]
Ah, I hadn't thought of that. To what degree have we seen this already? What would you consider as the biggest jump in intelligence per weight?
ak_t 4 hours ago [-]
It's been pretty consistent, the smaller models (hundreds of billions of params) usually catch up in 6 months or so to their frontier counterparts, at least on benchmarks.
Manfrednotfunny 12 hours ago [-]
I would say 5 years.
The whole industry is now pushing through memory.
In 5 years you have either some type of explosion which willjust make all the hardware from today affordable or you have such an AI explosion, that the today hardware is written off and not efficient enough anymore that you can buy it for cheap.
In parallel, its clear that we need more memory.
In parallel models in hardware will become a thing on mass market.
In parallel everything gets more efficient. The 30B parameter model will be for sure more intelligent in 5 years than it is today.
edg5000 10 hours ago [-]
Arguably no one was really prepared for AI, and NVidia kind of happended to coincidentally have suitable hardware. I have the feeling that something simpler and more efficient is possible when designed for the ground up purely for AI. But I could be wrong. That would mean there is a lot of room left for improvement.
cmrdporcupine 8 hours ago [-]
It is not at all just quantities of memory (or speed of computation)
It's in large part a problem of bandwidth, too. Mostly really. HBM memory can do up to 3TB/s vs DDR5 like 250GB/S. The latter is just too slow to process 2.5B parameter models, it simply can't move the values back and forth fast enough. It would drag to a crawl. Much smaller dense models at that speed on the NVIDIA Spark can't do more than 15tok/sec.
Real serving systems for these models involve large numbers of parallel GPUs with massive memory bandwidth, hooked up via NVlink.
It will take a long time for that level of tech to get down to consumer level.
(An ideal computing architecture built for LLMs would in fact offer some way of colocating computation with memory. If you can put matmul etc right in the DRAM and avoid going back and forth over the bus...)
arthurcolle 17 hours ago [-]
approximately $20 million for 750TB unified memory custom interconnect right now
edg5000 16 hours ago [-]
Wow, that's way higher than I assumed. I looked into it and remember ending up with something like 200k, but I must have been off. That's crazy.
arthurcolle 15 hours ago [-]
[dead]
ByteWarden 1 days ago [-]
More curious about how qwen3.8-27B performs. That's the size that I can run locally.
gilgoomesh 14 hours ago [-]
Yeah, I must have misread the press release last week as I thought it would be released at the same time.
Interestingly, the high variant does a lot worse and failed to generate a valid SVG (and the low variant use more tokens than the high one, so maybe their reasoning efforts are not working properly).
The solar system animation is also the coolest looking I've seen, unfortunately the animation doesn't work:
I've been wanting to run open weight models lately to give them a shot with OpenCode. However, I get the impression that models like Qwen and Kimi k3 are impossible to run locally? I have a RTX 5090 and 64 GB of RAM but the models seem to be much larger than that. What's the route to start using these models? Bedrock?
daemonologist 1 days ago [-]
Bedrock seems to have stopped adding new open-weights models, and mostly only has Anthropic and OpenAI stuff now. You can get Qwen 3.8 directly from Alibaba: https://www.qwencloud.com (proprietary variant) or from DigitalOcean (this variant, probably also from others soon).
On your 5090 you could easily run a smaller model like Qwen 3.6 27B: https://huggingface.co/collections/Qwen/qwen36 or Gemma 4 etc., or as mentioned there's a Qwen 3.8 27B coming out in a few days.
cautiouscat 1 days ago [-]
What does the number before the B signify?
Eisenstein 1 days ago [-]
Its number of parameters. The 'B' is billions. If you have bf16 weights each parameter would be 16 bits.
mswphd 1 days ago [-]
OpenRouter is (roughly) a single proxy between you + many different models + providers. it works with opencode (+ many other products), and is relatively convenient for trying out a bunch of models.
note that they add some fee ontop of things (maybe 10% of spend?). it isn't htat big of a deal for general experimentation, but if you end up wanting to use a single model in a higher-volume way, it likely makes sense to cut them out of your stack.
zeeveener 1 days ago [-]
You could easily run any of their 30B-or-less models which is what most people are waiting for.
Apparently the ~30B variant will be released on Friday?
julianlam 22 hours ago [-]
Define "easily". My laptop (2023) was configured with 16gb.
Getting to the point where I was able to run a 30B model required $500 in memory.
Mashimo 15 hours ago [-]
I think he is specificlly talking to the person with "RTX 5090 and 64 GB of RAM"
Not the greater "you"
gessha 1 days ago [-]
5090 is plenty for the Q4_K_M quantized version of 3.6 27B with reduced context size.
I run it on a 3090(24GB) and 64k context using GGUF format and llama-cpp. Double 3090 gives you 128k, quad 3090 gets you to full context - 256k.
FeepingCreature 13 hours ago [-]
Or you can run quantized context, there's some degradation but it fits in a lot less.
verdverm 1 days ago [-]
Fireworks or OpenCode Go
jannishan 8 hours ago [-]
Has anyone compared the programming capabilities of Qwe3.8 and Kimi K3? Which one is better?
octocop 1 days ago [-]
when will we see MIT license Qwen again?
simonw 1 days ago [-]
Is this the largest ever open weight model release by parameter count? I think it is.
NitpickLawyer 1 days ago [-]
No, Kimi k3 is 2.8T params. This is 2.4T params but ~5TB weights because it was released in bf16 and ~2.5TB for the fp8 version. Kimi k3 launched with QAT 4bit, so ~1.5TB weights.
Mercuriusdream 1 days ago [-]
KIMI K3 was the biggest open weight release afaik; It is 2.8T-A100B if I'm correct
CamperBob2 1 days ago [-]
Not seeing the upside versus K3 here, especially with the intentional capability loss.
Read the room, Qwen. It's not a good time to hobble your releases.
Bob_bo 20 hours ago [-]
People online say this model's performance isn't very good; what do you think of it after using it?
frozenseven 1 days ago [-]
Vocabulary size ~248k. A bit bigger than other recent Chinese models (Kimi K3 ~164k, DeepSeek-V4 ~129k, and GLM-5.2 ~155k).
Make of this what you will.
Ey7NFZ3P0nzAe 18 hours ago [-]
> Make of this what you will.
I'm interested in your take on it. IIRC Gemma family models too have a ~250k vocabulary size
miohtama 24 hours ago [-]
Does this mean its tokenizer is somehow tuned?
theanonymousone 1 days ago [-]
Do we know if AA and DeepSWE benchmarks are on bf16 or fp8 quantisations?
License pretty similar to k3 with some caveats. Free to use for internal or <50M$ revenue / year. Limitations above that threshold for serving the model or services targeting coding / productivity agents.
Benchmarks are looking good, trading blows w/ opus4.8 and sol, generally 10-20p under fable. But that's neither here nor there w/ qwen, their benchmark to real world usage correlation has been iffy in the past.
The local model 3.8-27B announced for Friday, same time so ~48 hours from now. That'll be a bit more exciting for a lot more people, since 3.6 was quite good for local inference, and their 3.7-max -> 3.8-max shows a lot of improvement.
It uses twice as much tokens to achieve the same but the results are significantly better and because it's so much cheaper it's the most economical choice too.
[1] https://blog.bosun.ai/software-maintenance-with-open-weight-...
It did take a little while for Unsloth to update the Laguna S 2.1 quants to fix the yarn_attn_factor, and so it was a bit frustrating getting that quantization running right, but almost always, I pick the unsloth quantization if there is one. (Still waiting/hoping for a Ling 3.0 Flash.)
I have had the same experience with gemma 4 on same tasks being refused. But this is when working with cyber offensive tasks and the like. It excels in coding and is very fast on consumer hardware. So I would say use the right tool for the right task.
What uncensored models can you recommend?
No issues with llama.cpp.
They are the most reliable in my experience, but if you have alternatives you trust I'd love to know
It's true they make architecture-specific changes like keeping certain layers at F16 but it's also more than that.
The question about LLMs is never whether they can be run, because that has a trivial answer, they can always be run. The right question is what speeds are achievable for representative hardware configurations.
At launch, it is difficult to estimate the speed. That should be known after someone reports experimental results. Moreover, for many LLMs the speed improved sometimes later after their release, after tweaks in inference backends, like llama.cpp or vLLM.
speed is not problem when You run agents and forget for 2-3 days
Maybe I’m misreading this or some other post, I thought QWEN was stepping away from releasing these models for local consumption
Sadly they seem to not be releasing a sparse 35b A3b or anything inbetween "too large to host for mortals" and "fits into a consumer rtx". Probably not to eat away their profits on their API serving. 120b - 300b is a dead space right now, very few good releases in that size range. (I know there are, but the big labs aren't releasing stuff here)
The best outcome for us is the one where they all keep competing and undermining each other until the end of time while providing us all with better models and cheaper hardware to run them with. The US corporations in particular should never be allowed to achieve their "you'll buy intelligence from us on a meter" rent seeking dream.
https://news.ycombinator.com/item?id=49243397
China also puts pressure on rich chinese flaunting their riches.
They have a common prosperity initiative.
They literally announced their motivations and world few a few weeks ago at the Shanghai AI conference. They want to ally with the global south. They see AI like the industrial revolution: the global south was left behind for a long time and, as a result has been exploited and has struggled to develop for a long time. They see open AI as a way to level the playing field to prevent such "new historical injustices" (in the sense of the Century of Humiliation and the Opium Wars). Concrete policies to back this rhetoric include technology transfer and training programs for the global south. They frame this latter not as philantropy but as generosity, in the sense that it generates goodwill and what goes around comes around. They believe that helping the global south and cultivating relationships will eventually help China.
Think about it. Your local businesses are not charities either. That doesn't make them bad, nor does it mean you derive no benefit. It still benefits you to cultivate good relationships with them.
Nothing happens on a geopolitical scale, from the US, China or anyone else, simply because of generosity.
No matter what you believe is their "true" intentions, offering 5000 training and tech transfer positions to the global south is a very concrete and unambiguous move. As are forgiving African loans and unilaterally offering zero trade tariffs.
I'm not saying that China's investment in the South hasn't had positive effects. But "generosity" is rarely a relevant lens when analyzing international relations.
Well, yes? Why does it have to be either-or?
> But "generosity" is rarely a relevant lens when analyzing international relations.
Automatically assuming nefarious intentions behind all moves is also rarely a relevant lens.
And as I said, and I'm not sure why you keep ignoring it, but I define "generosity" in the sense of mutual benefit. Being nice to your neighbors and helping them, benefits you due to generated goodwill. I'm explicitly not defining generosity in the sense of selfless philanthropy where you get nothing back. The idea that doing good things for others eventually results in good things coming your way, and thus that one should do good things for others even it's selfishly motivated (and also that there's nothing wrong with this), is not a crazy idea.
In a lot of cultures (Chinese included), gifts are not simply gifts. There is the social expectation that the gift is reciprocated. Western cynicists may call this "manipulation" or "influence". The Chinese see this as the start of a relationship of a cycle of mutual gift giving.
It's not just according to you.
Without open weights, what happens if you get blacklisted from Anthropic and OpenAI? If AI becomes a standard tool for programming like a compiler, you've effectively been Blackballed from the field of programming. Full Stop. This is "Right to Read" coming home: https://www.gnu.org/philosophy/right-to-read.en.html
In addition, without open source competitors to your core tools, we KNOW what happens. Cadence and Synopsys and a megabuck per engineer per year ... that's what happens.
The trouble here is how more infrastructure helps OpenAI and Anthropic continue billing at 10/100x Chinese model rates.
Either their models have to be better (to justify the higher prices and margin) or their inference has to be lower cost (which isn't going to happen until they move away from Nvidia).
The real policy mechanisms around open AI models are also incentives. Various cities have programs to pay companies for releasing open models. They subsidize compute through vouchers. They reward universities and students for open source collaboration.
This isn't some black box. The policies are written down, anybody can read their AI+ policy papers.
Alibaba went back to releasing open models way before the Xi speech from a few weeks ago. The cause is pressure from researchers, who believe in openness, as well as the competition who keeps releasing open models. This is Chinese "involution" at work. And the subsidies also help, of course.
[1] https://www.reddit.com/r/LocalLLaMA/comments/1vmi0fg/deepsee...
In terms of what you get for what you pay for, it's incredible - probably by far the best.
But unless I'm reading things wrong, it does not appear to be top-of-the-line.
At this price range $0.87per 1M they will get a lot of usage of people trying it out. Given the benchmark numbers, for many people and many use cases this will become their primary driver. There are people and use cases where Fable, Sol will work better but those are likely not the target of DeepSeek anyway.
In terms of performance and price pareto curve I don't think any model can beat this today (though openAI is doing some exciting recent work in efficiency) - which is a remarkable feat for the DeepSeek team.
Either way, what a time for consumers of these models :)
DeepSeek and Kimi K3 will happily do security work, and they do it pretty well.
Curious if you’ve find yourself enjoying DS or Kimi more than Opus 4?
Kimi K3 is smarter, though. At least smarter than DeepSeek V4 Flash 0731. I haven't tried the new Pro version, but will this weekend when I'm working on my personal projects But, K3 has been what I've been using for the actual coding of the harness and such (after Claude models, and then OpenAI models, began refusing to do that work). K3 is very expensive, though. Much more expensive than pretty much everything except Anthropic, and their subscription plans are stingy.
I also like Reasonix quite a bit, as an agent harness, though Kimi Code is also very good. I guess I'll try out the new DeepSeek official harness, as well.
In a benchmark, DeepSeek v4 Pro 0813 found 87.5% of selected real world software vulnerabilities publicly reported and with CVEs assigned, which is above runner ups Opus 5 and Qwen 3.8 which both found only 81.3%. However there is a downside to this--DeepSeek v4 Pro 0813 is less accurate with a 35% false positive rate versus GPT-5.6-Sol's 15% false positive rate. For vulnerability analysis though, it's probably worth finding that one extra vulnerability no other model has found even if requires significantly more triage to remove false positives, or additional cost to run every vulnerability detection through other models to verify.
[1] https://nitter.net/pilvar222/status/2087691659953815783#m
https://openrouter.ai/deepseek/deepseek-v4-pro-0813
The former has a button to dismiss the dialog. Maybe you clicked on it by accident, or maybe it does not work right.
they have a somewhat selfhostable model(flash), but are mostly known for having super cheap api access
no zdr ofcourse.
Since the models are open, there's other infra providers who have different policies.
The 1bit quant model is at an astonishing 397GB with 95B active per MOE. This literally puts Opus 4.5 performance level into a machine a normal person could buy, and still gets usable tokens/second.
The full lossless model BF16 is clocking at 4.9TB. The model card claims the model to be between Opus 4.8 and Fable 5. Again that's astonishing as getting a machine with 7TB RAM (with context + KV cache) is still within the realm of medium size companies.
Bad things: The open source version has its vision capability removed, and the context capped at 250k . I expect someone to bolt a Kimi 2.6 vision tower to it to restore the vision capability (at less performance of course). For context, I played around with extending the context to 600k for Qwen 3.5 397b, and the context remained stable up to around 480k. It'd be interesting to see if the same can be done to Q3.8 .
Also no out of the box DSpark/DFlash support. MTP is present so we should at least get some boost in TP speed.
Honestly this model people at home can tinker with, if you have a big enough Mac. Maybe 4 Strix Halo/DGX Spark, and then at 1 bit quant? Nah.
Use the right sized model, for your hardware. You'll get better results.
Extremely large models don't suffer as much from quantization due to its weight topology also contains encoded information, so the loss of info from any one weight is somewhat mitigated.
I wouldn't pick up 400gb of hardware to run in that mode. I might try it for fun, but even then you are looking at handling a 95GB active parameter set.
This is NOT a model for most home labs. I'm sure some can and will use it. But most, should steer clear.
95GB active is not _too_ bad, would require some creativity and $$, but I bet I could do that at home for less than a cheap car.
Suppose I have 100GB of unified memory, how should I know which model suits it best? I understand how a 2.4T model wouldn't fit, but I don't understand the impact of quantization and whether I should use a 200G model quantised to fit say 90GB of memory, or a non-quantised 90G model.
What's weird about local LLM models is that closed door improvements in training/RLHF dataset have been so significant that it's rare for larger but older models to make sense - everyone seem to always hard switch to the newest one and report step changes in capabilities(or maybe people running Kimi K2 since release just don't talk about it on the public Internet, giving me that impression).
The old rule of thumb was that a lower quant of a larger model > higher quant of a smaller model. That being said, for some things going lower than fp8 will see a lot of degradation in generation quality. Except if the model comes with QAT 4bit quants. Then there's also nvfp4 w/ calibration data, which also can improve things. So it's really not easy to tell "at a glance" you'd have to test them yourself on your hardware.
Anything below that, and especially 1.58b - is typically complete garbage, and you're much better off running a model 100x smaller at regular precision (compared to one 7x smaller quantized into complete garbage).
If the model was designed specifically to quantize down to 1.58b, then it's different.
AFAIK, there's no large models designed for this yet.
> AFAIK, there's no large models designed for this yet.
Isn't BitNet b1.58 2B4T what you are looking for? (haven't tried it myself though)
No 100B+ param (certainly no 2T+ param) models have been trained natively to quantize down to 1.58b.
at this kind of quantization is it useful though?
That is unfortunate, that the open weight model doesn't have vision support or the 1M context length...
[1] https://old.reddit.com/r/LocalLLaMA/comments/1vl6ior/i_gave_...
[2] https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4
The whole industry is now pushing through memory.
In 5 years you have either some type of explosion which willjust make all the hardware from today affordable or you have such an AI explosion, that the today hardware is written off and not efficient enough anymore that you can buy it for cheap.
In parallel, its clear that we need more memory.
In parallel models in hardware will become a thing on mass market.
In parallel everything gets more efficient. The 30B parameter model will be for sure more intelligent in 5 years than it is today.
It's in large part a problem of bandwidth, too. Mostly really. HBM memory can do up to 3TB/s vs DDR5 like 250GB/S. The latter is just too slow to process 2.5B parameter models, it simply can't move the values back and forth fast enough. It would drag to a crawl. Much smaller dense models at that speed on the NVIDIA Spark can't do more than 15tok/sec.
Real serving systems for these models involve large numbers of parallel GPUs with massive memory bandwidth, hooked up via NVlink.
It will take a long time for that level of tech to get down to consumer level.
(An ideal computing architecture built for LLMs would in fact offer some way of colocating computation with memory. If you can put matmul etc right in the DRAM and avoid going back and forth over the bus...)
[0]: https://aibenchy.com/compare/x-ai-grok-4-6-high/bytedance-se...
[1]: https://aibenchy.com/compare/x-ai-grok-4-6-high/bytedance-se...
The solar system animation is also the coolest looking I've seen, unfortunately the animation doesn't work:
https://aibenchy.com/compare/qwen-qwen3-8-2-4t-a95b-low/qwen...
On your 5090 you could easily run a smaller model like Qwen 3.6 27B: https://huggingface.co/collections/Qwen/qwen36 or Gemma 4 etc., or as mentioned there's a Qwen 3.8 27B coming out in a few days.
for example, they already have qwen3.8-max
https://openrouter.ai/discover?model=qwen/qwen3.8-max
note that they add some fee ontop of things (maybe 10% of spend?). it isn't htat big of a deal for general experimentation, but if you end up wanting to use a single model in a higher-volume way, it likely makes sense to cut them out of your stack.
Apparently the ~30B variant will be released on Friday?
Getting to the point where I was able to run a 30B model required $500 in memory.
Not the greater "you"
I run it on a 3090(24GB) and 64k context using GGUF format and llama-cpp. Double 3090 gives you 128k, quad 3090 gets you to full context - 256k.
Read the room, Qwen. It's not a good time to hobble your releases.
Make of this what you will.
I'm interested in your take on it. IIRC Gemma family models too have a ~250k vocabulary size
best crypto-bro impression I can do...