In 5 years you have either some type of explosion which willjust make all the hardware from today affordable or you have such an AI explosion, that the today hardware is written off and not efficient enough anymore that you can buy it for cheap.
In parallel, its clear that we need more memory.
In parallel models in hardware will become a thing on mass market.
In parallel everything gets more efficient. The 30B parameter model will be for sure more intelligent in 5 years than it is today.
Arguably no one was really prepared for AI, and NVidia kind of happended to coincidentally have suitable hardware. I have the feeling that something simpler and more efficient is possible when designed for the ground up purely for AI. But I could be wrong. That would mean there is a lot of room left for improvement.
It is not at all just quantities of memory (or speed of computation)
It's in large part a problem of bandwidth, too. Mostly really. HBM memory can do up to 3TB/s vs DDR5 like 250GB/S. The latter is just too slow to process 2.5B parameter models, it simply can't move the values back and forth fast enough. It would drag to a crawl. Much smaller dense models at that speed on the NVIDIA Spark can't do more than 15tok/sec.
Real serving systems for these models involve large numbers of parallel GPUs with massive memory bandwidth, hooked up via NVlink.
It will take a long time for that level of tech to get down to consumer level.
(An ideal computing architecture built for LLMs would in fact offer some way of colocating computation with memory. If you can put matmul etc right in the DRAM and avoid going back and forth over the bus...)
Im still giving myself the illusion that tracking my sleep is doing anything relevant.
I sleep bad -> buy watch -> watch tells me I sleep bad -> win/win?
In the gym, i use the timer. But is it worth it to carry a watch all the time, keep it charged etc. for a timer? I'm considering buying a timer for the gym.
Yeah, I agree - I think smart watches haven't yet evolved out of just being a fun tech gadget. If you just want a timer, I'd recommend just getting a cheap digital watch - you can get a casio for like 20 euros/dollars/pounds/etc and it'll give you a timer and the time/date and not much else. No cool tech gadget, but it 'll stay charged for 10 years rather than 2 days.
As funny as Mechahitler was it was more of a Microsoft Tay moment with the chatbot parroting what Twitter’s users were telling him without guardrails or a safe system prompt. It had nothing to do with grok’s or Musks pro nazi views (or lack thereof)
Also, that's not what strawmanning is. I never denied that Grok didn't act bizarrely offensively over a fucking year and a half ago (so did other LLMs, btw... and so have many other experiments over the years, remember Microsoft's?), which is an eternity in this space. I know Musk is polarizing, but give me a fucking break. Don't assume malice when social incompetence serves as an exculpatory factor.
Apparently, you are unable to comprehend that your opinion of things has been tainted away from the truth by an algorithm incentivized to outrage you. That what you call your "values" are, in fact, driven by someone else's greed for eyeball attention. Do you think civilizations that become anti-Western-values over time are more driven by facts and empiricism, or by catchy slogans that twist the truth and a media that uses cherry-picked examples which immediately trigger emotions?
So i do care that Elon Musk is responsible for USAID shutdown. The richest man on the world shuts down human support so abruptly that he causes real humans to die.
Elon Musk, as the richest person on the planet, bought himself a propaganda platform he controls and started to finger around in democracy.
It disheartens me that if I want to find the most objective lowdown on the USAID and governmental meddling thing, I can get it trivially from an AI, and people will immediately dismiss it as slop, even if it provides receipts for all of its assertions:
Why don't more people do this, or trust it? Do they not want to get upset when their sacred cows are toppled? Do they not realize that getting their cows toppled actually makes them more correct? I don't know, and I'm starting to not care.
My statement to you is: I don't care what you believe or care about. I only care if you have actually faced the evidence against your beliefs and care. If they still stand after that from a good-faith point of view, then you can have them. Otherwise, they are just "partisan", and I have zero interest in partisan politics anymore, because it's way too bullshit-infused at this point, driven by a media whose profit only comes from outrage, and mired in people who are obsessed with identity politics.
(A friend once told me, "you are the most persuaded by evidence and argument of anyone I know... and that's a sad thing for humanity," so there's that.)
That said... My personal take (since I find this particular point unaddressed by the larger conversation around this) is that a lot of intangible value was lost by USAID going away, such as American goodwill. This then turns into fodder for terrorists... Which is, of course, very nearsighted. And the bill for that may come due one day.
RL can do behavior cloning, but really needs good simulations or verifiable environments to get to superhuman levels. That currently exists for math, coding, and a lot of videogames. Soon there will be good enough simulations for robotics.
There's a lot of domains where that simply isn't the case (like bio)
You get much better supervised data in bio/chem though. These data companies have people working on exactly that.
While it's not going to give you an "alphago" effect, it is still enough to work at human levels, augmented with the general knowledge of an LLM, together making it super-human.
Yes. But there is also no other choice for people in these professions. The underlying job has been automated already. What's left is automating the last leg.
If you consider a 5-year outlook, it is also a very temporary job unless you're like a specialist neurosurgeon or something, as one of the examples in that article shows:
> The on-again, off-again nature of the work is not just the result of company culture; it stems from the cadence of AI development itself. People across the industry described the pattern. A model builder, like OpenAI or Anthropic, discovers that its model is weak on chemistry, so it pays a data vendor like Mercor or Scale AI to find chemists to make data. The chemists do tasks until there is a sufficient quantity for a batch to go back to the lab, and the job is paused until the lab sees how the data affects the model. Maybe the lab moves forward, but this time, it’s asking for a slightly different type of data. When the job resumes, the vendor discovers the new instructions make the tasks take longer, which means the cost estimate the vendor gave the lab is now wrong, which means the vendor cuts pay or tries to get workers to move faster. The new batch of data is delivered, and the job is paused once more. Maybe the lab changes its data requirements again, discovers it has enough data, and ends the project or decides to go with another vendor entirely. Maybe now the lab wants only organic chemists and everyone without the relevant background gets taken off the project. Next, it’s biology data that’s in demand, or architectural sketches, or K–12 syllabus design.
Humans, however, are highly variable, which may produce really varied and interesting results if they work together.
One instance of an LLM is the same as another instance, so while you may get more out of it by stacking more of them, I strongly suspect it falls victim to diminishing returns. 100 instances of the same LLM may converge on the same result as 10.
I think using different AGENTS.md can give the same model different perspectives on the same problem. For example a model with a well-tuned AGENTS.md by an expert mathematician approaching the same problem as the same model with a well-tuned AGENTS.md by an expert biologist can grind on the same problem from different perpectives.
It's worth a shot at least, as a microservices architect I have a bias that we aren't networking these enough, a single main agent session orchestrating multiple subagents is different from multiple main agent sessions with their own subagents coordinating with each other.
Crucially, does it make capabilities infinitely scalable? My comment just said that models may have a hard cap, and maybe doing specific setups like yours can make reaching it easier, but making the 'team' 10x larger after that optimal point may bring few to no improvements.
Although I'm also not sure about just how much better models can really get with this technique. Ultimately you're still getting the same model with the same training data, which are the important parts. Asking it to pretend to be something feels like it would just put a color filter in front of the conclusion the model has already predicted, or maybe alter the path to the conclusion slightly or pick a less likely answer that it still could've provided normally.
I agree it doesn't make the capabilities infinitely scalable, wasn't arguing with that point. It's just an experiment. I'm not talking about "you are an expert mathematician, go", I'm talking about an expert encoding their heuristics into the AGENTS.md base context. Routing the model's attention to very different aspects of the same problem in the early context.
FWIW I mean if I have an AGENTS.md that encodes my software heuristics (use an interface in situations like X, here's how we name variables, etc.) it generates far cleaner code than if I don't.
Edit- mostly pointing out that stacking 10 base models vs. 10 models with sufficiently different base context isn't necessarily the same attention routing. I suppose I was thinking about tasks that don't have a concrete single answer.
> 100 instances of the same LLM may converge on the same result as 10.
Not in the highly verifiable domains. There you can take it from say 80-90% maj@x to 99% pass@n. Math, some parts of programming and cybersec are examples of highly verifiable domains. (e.g. if you're searching for a linux LPE, that's expensive to search but easy/cheap to verify - just have a token in /root and have the model retrieve that token)
Verifiability makes it easier to understand how well the LLM works, but this doesn't counter my hypothesis. If X number of instances get 99.0% on an objective, verifiable metric, is there any guarantee that 10X will get 99.9%? The fact that we are reliant on new model releases to push capability in big ways, and that people running gigantic clusters of LLMs end up beaten by new models implies that the capabilities of a given model have a hard upper limit, and that it may not even take much to reach it.
High temperature makes the LLM pick more out-of-distribution tokens, but the choices its presented with are still the same or same-ish. I'm not convinced that the more random outputs don't end up averaging to roughly the same conclusion after enough passes.
Yes, given enough time I can answer all the questions in an IQ test correctly. We measure human intelligence in a time-limited setting and score relative to the performance of other humans doing the exact same task. Problem is brains can’t be scaled. To scale humans we need organizations, but human organizations also don’t scale well with increasing headcount.
LLMs scale well in almost all dimensions. Context window (working memory) can be a bottleneck but for humans you can’t scale it at all.
Why do you deny blaming Trump? Most other republicans wouldn't have done that. He's opened up a new ability to ignore standards of behavior, but without him I really think it wouldn't have happened. And average voters have never heard of treaties like that. They voted for idiocracy realized, and they got it, but they didn't know what they voted for.
So voters are responsible (and I never convinced my dad even to this day that trump was bad, how much am I at fault), but he's an out there candidate in many ways too.
Its not about not blaming Trump, its about making it clear that its also the fault of every US American to even allow this in the first place. Not standing up, not fightning against it etc.
I think that's a fair comment. We are all responsible for our collective actions as a country, even if we are a tiny voice in the nation, even if whatever side you are on is small. We can all dare to speak our thoughts to others, share our views.
In 2016 Trump got 46.1% of the popular vote vs Hillary Clinton's 48.2%.
Expressed as fractions of the total US population in 2016:
- about 28.5% of the population was not eligible to vote (due to age or other reasons)
- about 31.6% of the population was eligible to vote but did not vote for either candidate
- about 20.4% of the population voted for Hillary Clinton
- about 19.5% of the population voted for Trump
If approx 1 in 5 is 'tiny' then support for the alternative was also 'tiny'.
LLM was invented and its just clear that this is something someone needs to build.
Why?
Because it makes just sense. You don't want an agent running on a laptop you close. You want to keep context small, you want to split up work / parallize it etc.
I'm now waiting for a while until the open source agent platform emerges and im borderline motivated to build something but i'm not doing it. He did, which is not a crime.
The whole industry is now pushing through memory.
In 5 years you have either some type of explosion which willjust make all the hardware from today affordable or you have such an AI explosion, that the today hardware is written off and not efficient enough anymore that you can buy it for cheap.
In parallel, its clear that we need more memory.
In parallel models in hardware will become a thing on mass market.
In parallel everything gets more efficient. The 30B parameter model will be for sure more intelligent in 5 years than it is today.