I've tried doing a loop of rending the SVG and then tweaking based on that, with local models (so, not nearly as strong). It wasn't very successful; it would mostly report that the image looked great and didn't need any tweaks. Maybe I should try it again, there have been some newer models since I first tried it. And yeah, maybe worth trying with bigger models. But I have found that models aren't necessarily the best at visual reasoning and review, even with a vision loop. Their lack of visual reasoning is part of why they still have trouble with things like ARC-AGI-3.
I've found much better luck giving it an audit check-list, including some steers like: are there any visual glitches or SVG bugs, are the colours consistent, etc.
You can always ask them to draw something else, as a way to avoid any possible pelican related data contamination; given how popular the pelican test is, I'm sure there's some pelican SVG drawing in the training sets of at least some of these models by now. For instance, you could ask for an SVG drawing of a cyborg bear riding a rocket powered unicycle.
It's a silly fun little benchmark, and because Simon's been doing it for so long, you have a lot of examples over the years to compare. But you can always come up with and run your own test with other drawings.
It's really frustrating, because now when I want to write something like a "not X but Y" or "you're absolutely right," I have to stop and decide if I want to self-censor to avoid sounding like a bot.
Sometimes those constructs are actually useful, but man has their overuse really killed them!
It drives us crazy because everyone is using the same 2-3 different machines. So rather than each person having their own unique speaking style, the whole world (or, everyone that publishes direct LLM output) is now speaking in the same couple of styles.
And these machines all tend to converge on very similar styles; they have huge amounts of overlap in training data (much of it being already obnoxious internet marketing), they frequently train on each others outputs, and the RLHF process has a tendency to emphasize certain kinds of "cheap win" styles of speech.
He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier models would be exempted, but once it crossed some benchmark based threshold it would be subject to this kind of oversight.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.
In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.
Yeah, Kraken SDR removed some functionality due to these concerns, if I remember correctly.
Odd, because export controls don't generally apply to published material (like open source software), but maybe they were worried that because they were also selling the hardware they could have issues due to the combo being export controlled.
As someone who works daily with export-control-adjacent hardware and software, my experience is that people tend to aggressively self-censor to a far higher standard than export control regulations actually require. The perceived headache of drawing the ire of whoever it is the enforces this stuff (which as I type this comment I'm just realizing I don't know who specifically is responsible for that) is so scary that people don't want to take any risk at all of being targeted.
But there are already benchtop or handheld signal analyzer for that purpose.
This seems more like a tool for checking across entire large assemblies like an entire building, car, aircraft, etc, for unknown sources. If you have an individual discrete device that you're already testing, just using traditional instrumentation seems reasonable, but on a large, complex assembly, I can see it being useful. Also useful for things like detecting if a particular antenna is working without actually going up there to measure near it; if you have a MIMO setup with multiple antennas, this might make it easier to check if all of them are working correctly when mounted in inconvenient areas.
I think that for a single device, this probably wouldn't help much over just having a more traditional signal analyzer, either benchtop or handheld. If you know what you're testing, just using a signal analyzer around it will give you a good first pass picture of emissions, and probably be much more informative and precise than this.
This seems more useful for finding unknown or hidden RF sources, for instance looking thorugh an entire building to find unknown RF sources, or maybe a whole complex assembly like a car or aircraft.
This is really weird. Someone creating 4 new accounts just to call this project fraudulent because it can't statically analyze every property you'd like? Does this person have a personal grudge against the author, or something?
Gemma 12B? It's unique in the Gemma family, and unique among vision models. It's a novel encoder-less model...the whole model is vision. Somehow. I don't understand it, but it blows away Gemma 4 31B and Qwen 27B in my tests. It's not even close. And, is also tiny and fast, compared to those larger models, so it's better and faster and smaller. Weird combo.
Tried it out. I'm compring against Qwen 3.5 122B-A10B, so a much larger model. It gets some correct, but Qwen 3.5 122B-A10B has done much better. Gemma 4 12B even hallucinated some species in trying to identify a plant, and the other guesses it made weren't all that close, while Qwen 3.5 122B-A10B got it right on the first try.
12B did get one right that 31B got wrong. I'd have to do a much more thorough eval to really compare, just a few anecdotal observations and it's kind of hard to really distinguish, but from the samples I've seen, Qwen 3.5 122B-A10B is doing much better at this task.
The 12B architecture definitely is interesting, and it may punch above its weight due to this (though again, would really need to do proper evals to compare). But of the models I've tried, Qwen3.5 122B-A10B really seems like the best for this kind of task.
Ah, yeah, a model ten times as large on disk will know more, for sure. My tasks are more about "what's happening in this image?" rather than "what is this thing?", which doesn't require encyclopedic knowledge, it need good reasoning about what it's seeing.
