Proximity to borders leaves the government the possibility that if it really, really wanted to search a property, they'd just have to cede some land in a 100mile radius to a foreign country and instantly create an international border wherever they need one.
Vulkan compute is not really designed or intended to be a CUDA competitor, its feature set is much more restricted, and Vulkan host side code is much more verbose than CUDA. OpenCL or SYCL are much closer in features to CUDA. I found that when using SYCL on Nvidia, debugging symbols etc can be passed through and you can use tools like NSight Compute to profile it as if it were CUDA.
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.
From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension.
The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
So when I said "a couple of nvidia engineers", I indeed meant Jeff.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
> I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable
__VK_ENABLE_DEVICE_GROUPS=1.
It could easily be a competitor to Cuda, if it just made things easier. Like, why does it take 50 lines of code to allocate memory in vulkan, and just one single line in cuda? Vulkan should just provide a single-line gpuMalloc convenience function. And not just for allocation, for all the other nonsense as well.
For a performant portable language, we’d have to go to a higher level where you describe what to do and leave the how to do to the compiler. It would then need to be able to adjust memory layout, access patterns, data type choice to the underlying hardware. I’m not sure if this is possible to do reliably - the closest we have right now may in fact be highly detailed plain English descriptions of the algorithms fed to an LLM prompted to produce assembly.
It's not possible to do reliably. None of the models in current use today use any esoteric math. It's extremely easy to implement the math behind both the inference and learning of all modern models.
Not everything is AI and dot products of massive vectors, there are still applications that do other maths on GPUs
My thinking was rather that most of our current programming languages put memory layout fully into the programmer’s responsibility - I can think off hand of a language where the compiler makes performance decisions like whether your structure are SoA, AoS or SoAoS, what alignment, padding, strides and float types to use.
Automatic decisions about when to use cooperative loads through shared local mem versus gathers from global mem and hardware caches are also something that such a hypothetical compiler would have to make.
I mean if you don't care about perf, opencl does what you want, and exists today.
As for ai and matrix vector performance... I mean matrices are extremely useful across many domains. The hands off language that exists today is called blas. That's fine but won't lead to the best perf on any GPU today.
SoA and AoS data layouts are not even a worthwhile point of contention. Same with shared v global mem. Today's hardware has specialized memory depending on the operation. The hardware on these processors is so specialized as to make anything but first class support for the feature be completely pointless. If you look at Nvidia code that's open source even they will special case a lot of their chips. Literally if you want the best perf you write the kernel exactly for the exact chip. That's intra vendor .. you can only imagine inter vendor
TIL, I didn’t know that. I always assumed it came from level zero”, the Intel computer layer that ZLUDA was translating to before its developer was hired by AMD to target HIP.
Yeah talking about the (vendor-preferred) compute part here
Vulkan's SPIR-V dialect is substantially different from the OpenCL one, notably with the former having structured control flow. They're incompatible between each other.
What do you mean by holistic? SYCL is an open versioned standard that allows for vendor specific extensions. The problem is not that there isn’t a proper standard, the problem is that many hardware vendors - or software developers simply don’t want to adopt it.
Intel (via Codeplay) was handing it out on a silver platter - Nvidia on SYCL, full top chain, and people still wouldn’t want it.
Isn't OneAPI a good example of the problem, alongside Mojo/ONNX/TensorRT? The industry doesn't need a fifteenth competing standard. They need hardware buy-in.
By holistic, I mean hardware architecture cooperation. Nvidia can hold onto their lead forever if GPU designers fight over what a GPGPU hardware baseline looks like. The current ecosystem fragmentation is not competitive, and future fragmentation probably wouldn't work either. I think the fastest way to kill Nvidia would be a hardware consortium.
The problem with OneAPI is naming. It leads people to believe that is another competing standard where in fact is is simply just an implementation of a standard compliant SYCL compiler. If it just had been named “Intel SYCL compiler”, similar to the existing and accepted Intel OpenCL compiler, it would have been easier.
What would you expect the hardware consortium to coordinate on? Unified ISA?
They were maintained by Codeplay - paid for my Intel. Nvidia can make contributions anytime they want, and here is the problem: Nvidia does not want to. Until each vendor starts pitching in with contributing their backend to an open standard, you will have to rely on others doing it for them.
AI will take down Nvidia’s moat. When it becomes trivial to translate CUDA/PTX to HIP, SYCL or Metal, CUDA is no longer the moat, it becomes the intermediate representation.
trivial to translate (or transpile) - okay. trivial to understand the result - not so much. trivial to then evolve it - hm... perhaps a different story. still, it seems very likely now, that such "quick rewrites" are viable, not sure if an open approach to them is viable. a newly born open project that was LLM-derived, and not by a credible author, which spans hundreds of files no human eye has ever looked at, can only work for a closed organization, but will never be trusted by the general audience... just like that.
I don't see a hard reason. If it works it works. No hard need for a good, universal, and long lasting solution. At some point you just stack slop on top of slop and it works for your use case - and if it doesn't you'll slop it out yourself.
tbh... it seems to work contrary to all expectation that slop shouldn't work. but, then, just when I think about the incredible corporate slop i've seen here and there, that not only works, but pays. well I guess we have to agree - slop works. and that's fine.
i swear people who are outsiders here have only clickbait takes; if you've never had to ship GPU code professionally you should just not comment on these things.
the source language has never been the moat. Nvidia sells to hyperscalers. Hyperscalers have armies of kernel authors who have no issue translating shaders by hand (or now with claude). Nvidia's moat is (and will remain for the foreseeable future) the entire stack. you cannot fathom the pain and misery of working on literally any other stack. if you've never debugged a GPU synchronization error or kernel panic due to some GPU firmware bug or fought absolute shit profilers hunting for perf you really have no idea what you're talking about.
EDIT: i can't believe this really requires saying but graphics and compute are not the same domain at all. if you work in graphics for GPU but not compute then you are still way out of your depth commenting. to wit: graphics people do not (and cannot) write CUDA kernels/shaders.
But claude will not be able to debug the whole stack? I have no experience in the domain. But as an outsider, it seems to me, that if the rate of progress is maintained, i don't see why agents wouldn't be able to do it in a couple of years
If your stack is open source, why not? I’ve had good results with pointing an LLM to a debug build of the GPU compiler when I saw it crash. It just seems you need to periodically remind them that gdb exists to prevent it from littering your drivers with printfs.
this isn't a "pissing contest"? you made a speculative claim in a public forum and i'm challenging your authority to make such a claim. a "pissing contest" would be if i had said i've shipped hundreds of thousands of lines of shader code into prod and thus you clearly have no idea what you're talking about because you haven't (which is also true).
I could post the GitHub URLs of all the shader code I wrote that’s running on countless GPUs right now, but what would it change? I’m still just a random guy on the internet with an opinion that happens to be different from your opinion.
You can simply disagree with me, regardless of my experience (or lack thereof).
that's exactly what i did and made an argument for why i think you're wrong. in response you provided exactly zero substantive remarks other than "i've written shaders" and then accused me of pissing.
also FYI it's clear from your profile that you've only worked on graphics (embree, blender, etc) and not compute. so i'll repeat: you're an outsider and you have absolutely no idea what you're talking about.
You can't know if he is an outsider so your argument runs into nothing. If you kept explaining why AI won't remove the CUDA moat and doubled down on that, then swerner would have kept silent, meanwhile your comments would stay so your current strategy is one blunder after the other.
“if you've never debugged a GPU synchronization error or kernel panic due to some GPU firmware bug or fought absolute shit profilers hunting for perf”
I have done all of those things. As part of my full time job, for years.
Now that we’ve put all of that aside, can we stop talking about me and go back to discussing moats? What do you think are top three things that are holding customers back from buying AMD GPUs instead of Nvidia GPUs?
I could be misremembering, but I think Jensen Huang himself once called CUDA or the CUDA ecosystem their moat, and it certainly seems to be accepted narrative in the tech press. They may be wrong there, and you sharing your first hand experience here would be helpful to many of us readers here.
That’s not strictly true, they officially support some of them.
My 7900XTX is supported, I run local models via rocm all the time recently, mostly to play with/experiment on, Vulcan works as well and for some models works better (or the trade offs are better for that use case).
Their mistake was simply not picking and going all in earlier, they let nvidia become the defacto standard without even contesting it on both the hardware side and software side and that’s a hard though not impossible comeback to make.
Long term I think they’ll catch up in capability if not market share because simply too much money on the table not to.
Most modern graphics is compute. Pixar, Dreamworks, Sony, etc do not use Vulkan to render their movies. It’s CPUs or CUDA.
“graphics people do not (and cannot) write CUDA kernels/shaders” is just not true at all. All it would take to verify that would be things like reading the introduction of the OptiX documentation, a small sample of SIGGRAPH GPU papers or the Blender/Cycles source code.
Wait, I have an idea:
What is your qualification for talking about graphics programming? If you cannot provide credentials, I will have no other choice than to claim you don’t know what you’re talking about, based on the criteria by which you judged my comments so far.
I don't think we're at a point yet where anyone would trust ZLUDA enough to ship commercial products that rely on it. I would be delighted though, if anyone can prove me wrong.
Because while many want to work on the same project generally, the people who can or want to do that has so little time.
I personally had to put all my toy projects into hiatus to be able to keep up with life. Maybe if things get a little lighter, we can thaw them or start working on a couple of interesting ones which I stumble on.
yeah yeah, "when" an often keyword with AI it seems. As Mr. E. Nigma put it - what always comes but never arrives? Meanwhile the moat deepens and it's build on inertia and laziness and Nvidia knows this really REALLY well.
Oh, absolutely. Nvidia is the modern day “nobody gets fired for buying IBM”. The reason we’re still using Unix is not because it’s the best, but because it had to much inertia to let any alternative become its successor. Similarly, C and HTML are maybe the most terrible yet extremely useful languages we have.
It may have a different connotation in the US than in Europe. In US media I’ve only heard the term being used recently, in Europe it’s been around for decades.
In Europe, “Antifascist” is a label that people groups use to describe themselves, similar to how other describe themselves as patriot or pacifist. Someone carrying an Antifa flag at a protest is maybe comparable to someone carrying a Gadsden flag at a right wing rally.
Yeah, basically this. People have been calling themselves Antifascist, but only the right-wing calls it "Antifa". Literally anyone carrying a sign at a protest that says "Reject Fascism" is an antifascist. Those are the people the US government is calling terrorists.
reply