I picked up a pre-ai-price-insanity AX162-R at Hetzner a while back and loaded it up on memory to max out the 12 channels the 48c EPYC 9454P as a "this will be the last mysql box I'll need" and have I been _wildly_ impressed with it's performance. The things I throw at it are honestly laughable at times, wildly irresponsible queries against a rather large database, the redis qps metrics are ridiculous and I just load it up with random ggufs since the memory bandwidth is... not terrible and 384GB of it is... useful.
The web app that's hosted on it deals with lots of images and text - over 100mil of each deduplicated, embedded, simhashed - it does the hashing, and the embedding in real time during ingest. Just handles it.
> I did not personally test the open weight models beyond the old Qwen 3.6 27B, which produced unusably bad results for me.
So you don't have much perspective on things, it seems. Let me introduce you to the GLM 5.2 and then 5.3/5.3 flash series of... "oh, wow, I should have bought some RTX PRO 6000's while they were 'cheap'" stage of progression.
As someone carrying multiple max subscriptions to both claude and codex - primary workhorse is glm 5.3 flash running on rented GPUs for less than a latte/hr.
I also found qwen 3.6 27B nearly useless for my own needs. DS4 flash 0731 and then 4.1 have been nearly as eye opening as glm 5.3 flash, but have their own warts.
Have you tried Qwen 3.8 Flash Next? You can run it on one spark with reasonable context sizes at about 30 tps, and it's as good as DS Flash 0731. Maybe even a tie with GLM 5.3, though like everything it depends on the use case.
I’m having the same issue. Hold max subscriptions on both frontier labs but I’ve been forced to use open source models because token limits are not what they used to be. So I end up using Astra and Fable for reviewing, and open source models for implementing.
But who is publishing the outage information to X/Twitter if not the utility itself? I'd be surprised if they do that more timely and reliably on a third-party platform than their own app. That's not the case for any European or US transit authorities I know.
Can any other user of public transport in Paris comment?
1. the service may not be - every small provider has downtime, perhaps proportionally to their size.
2. the tools to update Twitter are well understood, and you can get software that does scheduling of posts across multiple platforms, previews of what posts will look like, etc etc, and said preview works well on the many platforms that people are already using. People doing the updating may prefer to use those tools over what the internal tool gives them.
I think you're arguing the same general point that the person you're responding to is. But you're saying he's not understanding - he understands that they report a cache hit % but you can't look at that public metric with any level of accuracy _because_ most people aren't pinning their providers and they _are_ getting juggled around which is bringing that metric down. That's not to say that specific providers might have issues or worse cache implementations - but it stands that if openrouter is juggling the requests back and forth by default then _that alone_ is breaking caches on those requests in huge numbers.
They're not "docking points", they're calculating it in the most straightforward way. If I start a session and the majority of requests are sent to Provider A, and my last request gets routed to Provider B, I have a 0% cache hit rate with Provider B. I'm very curious how else you expect this to be calculated? Do you think they're completely omitting requests that switch providers mid-session?
FWIW, I get significantly higher than listed cache hit rates when I pin my session to a specific provider, which is further evidence of the above.
> OpenRouter randomizes which provider gets your request by default right?
I'm not sure it's wholey accurate to say they "randomize" the provider, rather my assumption based on usage is that it's something like cheapest-ish/responded to the request within some reasonable-ish time/etc algorithm that chooses the provider on each request - which seems, remarkably questionable in terms of optimizing for user experience or hidden user costs.
> This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.
I so very much recommend this approach. My avenues that automate llm calls to openrouter are setup to make api reqs to openrouter to determine best price/response/etc and then pin the request to that (and, preferably, a fallback if there's reasonable difference between #1 and #2) provider for that session. Otherwise you're going to have a bad time.
I'd imagine this could make things interesting in cases where one provider is offering different quants than the others and openrouter is just swapping you back and forth on a long agentic session.
> All the American companies you mentioned still follow American law and regulation. Skirting that blatantly has big consequences.
> Chinese companies do not follow American laws and there are absolutely no consequences for violating it.
... lmk when anthropic/openai/spacex/xai are held accountable for anything. Anything at all. Hard to be when you're _writing_ the rules.
At the current point in time I'd argue it's more about opportunity cost/value.
If I'm a professional photographer chasing the best possible end product, I'm not buying cameras because they're economical. I'm buying the best camera I can get my hands on to get the best product I can produce within reason under the understanding that it doesn't have to equate to the best economic decision to be the _right_ decision.
If you're in a position to be able to take advantage of the local inference - it's a no brainer. If you're not sure how that would be done, then it's not a good move.
The web app that's hosted on it deals with lots of images and text - over 100mil of each deduplicated, embedded, simhashed - it does the hashing, and the embedding in real time during ingest. Just handles it.
These things are absolutely insane.
reply