Get the top HN stories in your inbox every day.
davidmckayv
nico
Another issue: Gemini can’t do tool calling and (forced) json output at the same time
If you want to use application/json as the specified output in the request, you can’t use tools
So if you need both, you either hope it gives you correct json when using tools (which many times it doesn’t). Or you have to do two requests, one for the tool calling, another for formatting
At least, even if annoying, this issue is pretty straightforward to get around
mattnewton
Back before structured outputs were common among model providers, I used to have a “end result” tool the model could call to get the structured response I was looking for. It worked very reliably.
It’s a bit of a hack but maybe that reliably works here?
nico
You can definitely build an agent and have it use tools like you mention. That’s the equivalent of making 2 requests to Gemini, one to get the initial answer/content, then another to get it formatted as proper json
The issue here is that Gemini has support for some internal tools (like search and web scraping), and when you ask the model to use those, you can’t also ask it to use application/json as the output (which you normally can when not using tools)
Not a huge issue, just annoying
behnamoh
Does any other provider allow that? what use cases are there for JSON + tool calling at the same time?
chrisweekly
Please correct my likely misunderstanding here, but on the surface, it seems to me that "call some tools then return JSON" has some pretty common use cases.
victorbjorklund
Let's say you wanna build an app that gives back structured data after a web search. First a tool call to a search api. Then do some reasoning/summar/etc on the data returned by the tool. And finally return JSON.
ayende
OpenAI, Ollama, DeepSeek all do that.
And wanting to programmatically work with the result + allow tool calls is super common.
shijithpk
Suppose there's a pdf with lots of tables i want to scrape. I mention the pdf url in my message and with gemini's url context tool, i now have access to the pdf.
I can ask gemini to give me the pdf's content as a json and it complies most of the time. But at times, there's an introductory line like "Here's your json:". Those introductory lines interfere with programmatically using the output. They're sometimes there, sometimes not.
If I could have structured output at the same time as tool use, I can reliably use what gemini spits out as it'll be in a json, no annoying intro lines.
wahnfrieden
OpenAI
golfer
Unfortunately Gemini isn't the only culprit here. I've had major problems with ChatGPT reliability myself.
mguerville
I only hit that problem in voice mode, it'll just stop halfway and restart. It's a jarring reminder of its lack of "real" intelligence
patrickmcnamara
I've heard a lot that voice mode uses a faster (and worse) model than regular ChatGPT. So I think this makes sense. But I haven't seen this in any official documentation.
Narciss
This is more because of VAD - voice activity detection
SilverElfin
I think what I am seeing from ChatGPT is highly varying performance. I think this must be something they are doing to manage limitations of compute or costs. With Gemini, I think what I see is slightly different - more like a lower “peak capability” than ChatGPT’s “peak capability”.
Fade_Dance
I'm fairly sure there's some sort of dynamic load balancing at work. I read an anecdote from someone had a test where they asked it to draw a little image (something like an ascii cat, but probably not exactly that since it seems a bit basic), and if the result came back poor they didn't bother using it until a different time of day.
Of course it could all be placebo, but when you intuitively think about it, somewhere on the road the the hundreds of billions in datacenter capex, one would think that there will be periods where compute and demand are out of sync. It's also perfectly understandable why now would be a time to be seeing that.
driese
Small things like this or the fact that AI studio still has issues with simple scrolling confuse me. How does such a brilliant tool still lack such basic things?
victorbjorklund
It's crazy how Google can create so many really amazing products technically but they fall short just because of basic UI/UX issues.
normie3000
I see Gemini web frequently break its own syntax highlighting.
brap
The scrolling in AI Studio is an absolute nightmare and somehow they managed to make it worse.
It’s so annoying that you have this super capable model but you interact with it using an app that is complete ass
SXX
App was likely built my same LLM...
Spooky23
Because they are moving fast and breaking shit.
Ask ChatGPT to output markdown or PDF on iOS or Mac app and the web experience. The web is often better - the apps will return nothing.
SkyPuncher
This is my perception as well.
Gemini 2.5 Pro is _amazing_ for software architecture, but I just get tired of poking it along. Sonnet does well enough.
dorianmariecom
chatgpt also has lots of reliability issues
diego_sandoval
If anyone from OpenAI is reading this, I have two complaints:
1. Using the "Projects" thing (Folder organization) makes my browser tab (on Firefox) become unusably slow after a while. I'm basically forced to use the default chats organization, even though I would like to organize my chats in folders.
2. After editing a message that you already sent,you get to select between the different branches of the chat (1/2, and so on), which is cool, but when ChatGPT fails to generate a response in this "branched conversation" context, it will continue failing forever. When your conversation is a single thread and a ChatGPT message fails with an error, re trying usually works and the chat continues normally.
porridgeraisin
And 3)
On mobile (android) opening the keyboard scrolls the chat to the bottom! I sometimes want to type referring something from the middle of the LLMs last answer.
zarmin
It would also be nice if ChatGPT could move chats between projects. My sidebar is a nightmare.
m101
I wonder if this is because a memory cap was reached at that output token. Perhaps they route conversations to different hardware depending on how long they expect it to be.
smittywerben
When this happened to me it was because, I can only guess, it was the Gemini servers were overloaded. Symptoms: Gemini model, Opaque API wrapper error, truncated responses. To be fair the Anthropic servers are overloaded a lot too but they have a clear error. I gave Gemini a few days on the bench and it fixed itself without any client side changes. YMMV.
tschillaci
Half my requests get retried because they fail, I've contributed to a ticket in June, with no fix yet.
simonw
I added support to these models to my llm-gemini plugin, so you can run them like this (using uvx so no need to install anything first):
export LLM_GEMINI_KEY='...'
uvx --isolated --with llm-gemini llm -m gemini-flash-lite-latest 'An epic poem about frogs at war with ducks'
Pelicans: https://github.com/simonw/llm-gemini/issues/104#issuecomment...
zamalek
I wonder if [good examples of] SVGs of pelicans on bikes are "being introduced" into training sets. Some of the engineers who work on this stuff are the kind to hang out here.
simonw
It's possible, but honestly I've never seen a decent vector illustration of a pelican on a bicycle myself so they'd have to work pretty hard to find one!
dimal
They could just ask a designer to do a few bespoke illustrations, then generate synthetic data from that, right? Have an image model generate a set of variations, then convert them to SVG.
But looking at these images, Google clearly hasn’t done that yet.
canadiantim
Who wins in the end? the frogs? the ducks? or the pelicans?
tclancy
I heard the dragon took the pole, but it may have been wind-aided.
nine_k
This depends on the value of your LLM_GEMINI_KEY!
herpderperator
Serious question: If it's an improved 2.5 model, why don't they call it version 2.6? Seems annoying to have to remember if you're using the old 2.5 or the new 2.5. Kind of like when Apple released the third-gen iPad many years ago and simply called it the "new iPad" without a number.
skerit
That's why people called the second version of Sonnet v3.5 simply v3.6, and Anthropic acknowledged that by naming the next version v3.7
Aeolun
Only Anthropic has a slightly understandable version scheme.
alwillis
It's pretty common to refer to models by the month and year they were released.
For example, the latest Gemini 2.5 Flash is known as "google/gemini-2.5-flash-preview-09-2025" [1].
[1]: https://openrouter.ai/google/gemini-2.5-flash-preview-09-202...
cpeterso
If they're going to include the month and year as part of the version number, they should at least use big endian dates like gemini-2.5-flash-preview-2025-09 instead of 09-2025.
herpderperator
Or, you know, just Gemini 2.6 Flash. I don't recall the 2.5 version having a date associated with it when it came out, though maybe they are using dates now. In marketing, at least, it's always known as Gemini 2.5 Flash/Pro.
kingo55
It had a date, but I also agree this is extremely confusing. Even semver 2.5.1 would be clearer IMO.
vitorgrs
It always had dates... They release multiple versions and update regularly. Not sure if this is the first 2.5 Flash update, but pretty sure Pro had a few updates as well...
This is also the case with OpenAI and their models. Pretty standard I guess.
They don't change the versioning, because I guess they don't consider it to be "a new model trained from scratch".
Thorrez
>For example, the latest Gemini 2.5 Flash is known as "google/gemini-2.5-flash-preview-09-2025" [1].
That "example" is the name used in the article under discussion. There's no need to link to openrouter.ai to find the name.
relatedtitle
I'm pretty sure Google just does that for preview models and they drop the date from the name when it's released.
someguyiguess
If only there was some of versioning nomenclature they could use. Maybe even one that is … semantic? Oh how I wish someone would introduce something like this to the software engineering field. /s
In all seriousness though, their version system is awful.
qafy
2.5 is not the version number, it's the generation of the underlying model architecture. Think of it like the trim level on a Mazda 3 hatchback. Mazda already has the Mazda 3 Sport in their lineup, then later they release the Mazda 3 Turbo which is much faster. When they release this new version of the vehicle its not called the Mazda 4... that would be an entirely different vehicle based on a new platform and powertrain etc (if it existed). The new vehicle is just a new trim level / visual refresh of the existing Mazda 3.
That's why Google names it like this, but I agree its dumb. Semver would be easier.
someguyiguess
I’d say it’s more like naming your Operating System off of the kernel version number.
pests
Gonna steal this to help explain to non tech friends when it comes up again.
JumpCrisscross
Maybe they’re signalling it’s more of a bug fix?
manquer
2.5.1 then .
semantic versioning works for most scenarios.
JumpCrisscross
Would that automatically roll over anyone pinging 2.5 via their API?
ashwindharne
Google seems to be the main foundation model provider that's really focusing on the latency/TPS/cost dimensions. Anthropic/OpenAI are really making strides in model intelligence, but underneath some critical threshold of performance, the really long thinking times make workflows feel a lot worse in collaboration-style tools, vs a much snappier but slightly less intelligent model.
It's a delicate balance, because these Gemini models sometimes feel downright lobotomized compared to claude or gpt-5.
omarspira
I would be surprised if this dichotomy you're painting holds up to scrutiny.
My understanding is Gemini is not far behind on "intelligence", certainly not in a way that leaves obvious doubt over where they will be over the next iteration/model cycles, where I would expect them to at least continue closing the gap. I'd be curious if you have some benchmarks to share that suggest otherwise.
Meanwhile, afaik something Google has done, and perhaps relates back to your point re "latency/TPS/cost dimensions" that other providers aren't doing as much is integrating their model into interesting products beyond chat, at a pace that seems surprising given how much criticism they had been taking for being "slow" to react to the LLM trend.
Besides the Google Workspace surface and Google search, which now seem obvious - there are other interesting places where Gemini will surface - https://jules.google/ for one, to say nothing of their experiments/betas in the creative space - https://labs.google/flow/about
Another I noticed today: https://www.google.com/finance/beta
I would have thought putting Gemini on a finance dashboard like this would be inviting all sorts of regulatory (and other) scrutiny... and wouldn't be in keeping with a "slow" incumbent. But given the current climate, it seems Google is plowing ahead just as much as anyone else - with a lot more resources and surface to bring to bear. Imagine Gemini integration on Youtube. At this point it just seems like counting down the days...
CuriouslyC
I do scientific and hard code a lot. Gemini is a good bit below GPT5 in those areas, though still quite good. It's also just a bad agent, it lacks autonomy and isn't RL'd to explore well. Gemini's superpower is being really smart while also having by far the best long context reasoning, use it like an oracle with bundles of your entire codebase (or a subtree if it's too big) to guide agents in implementation.
cerved
Yesterday I asked Gemini to recalculate the timestamps of tasks in a sequence of tasks, given it's duration and the previous timestamp. It proceeded to write code which gave results like this
2025-09-26T14:32:10Z
2025-09-26T14:32:10Z200s
2025-09-26T14:32:10Z200s600s
2025-09-26T14:32:10Z200s600s300s
Gemini is by far the dumbest LLM I've used
lelanthran
They're all a little dumb. I asked claude for a python function or functions that will take in markdown in a string and return a string with ansi codes for bold, italics and underline.
It gave me a 160 line parse function.
After gaping for a short while, I implemented it in a 5 line function and a lookup table.
These vibe codes who are proud that they generated thousands of lines of code makes me wonder if they are ever reading what they generate with a critical eye.
ainch
Gemini 2.5-Pro was great when it released, but o3 and GPT-5 both eclipsed it for me—the tool use/search improvements open up so many use cases that Gemini fails at.
perfmode
How’d I never hear of Jules? Cool.
Al-Khwarizmi
And yet my smart speakers with the Google assistant still default to a dumb model from the pre-LLM era (although my phone's version of the assistant does call Gemini). I wonder why that is, as it would be an obvious place to integrate Gemini. The bar is very very low as anything outside the standard setting alarms, checking the weather, etc. it gets wrong most of the time.
jjani
Can't agree with that. Gemini doesn't lead just on price/performance - ironically it's the best "normie" model most of the time, despite it's lack of popularity with them until very recent.
It's bad at agentic stuff, especially coding. Incomparably so compared to Claude and now GPT-5. But if it's just about asking it random stuff, and especially going on for very long in the same conversation - which non-tech users have a tendency to do - Gemini wins. It's still the best at long context, noticing things said long ago.
Earlier this week I was doing some debugging. For debugging especially I like to run sonnet/gpt5/2.5-pro in parallel with the same prompt/convo. Gemini was the only one that, 4 or so messages in, pointed out something very relevant in the middle of the logs in the very first message. GPT and Sonnet both failed to notice, leading them to give wrong sample code. I would've wasted more time if I hadn't used Gemini.
It's also still the best at a good number of low-resource languages. It doesn't glaze too much (Sonnet, ChatGPT) without being overly stubborn (raw GPT-5 API). It's by far the best at OCR and image recognition, which a lot of average users use quite a bit.
Google's ridiculously bad at marketing and AI UX, but they'll get there. They're already much more than just a "bang for the buck" player.
FWIW I use all 3 above mentioned on a daily basis for a wide variety of tasks, often side-by-side in parallel to compare performance.
breakingcups
My pet theory without any strong foundation is because OpenAI and Anthropic have trained their models really hard to fit the sycophantic mold of:
===============================
Got it — *compliment on the info you've shared*, *informal summary of task*. *Another compliment*, but *downside of question*.
----------
(relevant emoji) Bla bla bla
1. Aspect 1
2. Aspect 2
----------
*Actual answer*
-----------
(checkmark emoji) *Reassuring you about its answer because:*
* Summary point 1
* Summary point 2
* Summary point 3
Would you like me to *verb* a ready-made *noun* that will *something that's helpful to you 40% of the time*?
===============================
kridsdale1
I suspect this has emerged organically from the user given RLHF via thumb voting in the apps. People LIKE being treated this way so the model converges in that direction.
Same as social media converging to rage bait. The user base LIKES it subconsciously. Nobody at the companies explicitly added that to content recommendation model training. I know, for the latter, as I was there.
Twirrim
Gemini does the sycophantic thing too, so I'm not sure that holds water. I keep having to remind it to stop with the praise whenever my previous instruction slips out of context window.
porridgeraisin
Oh god I _hate_ this. Does anyone have any custom instructions to shut this thing off. The only thing that worked for me is to ask the model to be terse. But that causes the main answer part to be terse too, which sucks sometimes.
typpilol
Anthropic also injects these long conversation reminders that are paragraph upon paragraphs about safety and what not to do.
People have said it destroys the intelligence mid convo
m_mueller
Not the case with GPT-5 I’d say. Sonnet 4 feels a lot like this, but the coding and agency of it is still quite solid and overall IMO the best coder. Gemini2.5 to me is most helpful as a research assistant. It’s quite good together with google search based grounding.
lelanthran
Gemini does this too, but also adds a youtube link to every answer.
Just on the video link alone Gemini is making money on the free tier by pointing the hapless user at an ad while the other LLMs make zilch off the free tier.
dudeinhawaii
I've experienced the opposite. Gemini is actually the MOST sycophantic model.
Additionally, despite having "grounding with google search" it tends to default to old knowledge. I usually have to inform it that it's presently 2025. Even after searching and confirming, it'll respond with something along the lines of "in this hypothetical timeline" as if I just gaslit it.
Consider this conversation I just had with all Claude, Gemini, GPT-5.
<ask them to consider DDR6 vs M3 Ultra memory bandwidth>
-- follow up --
User: "Would this enable CPU inference or not? I'm trying to understand if something like a high-end Intel chip or a Ryzen with built in GPU units could theoretically leverage this memory bandwidth to perform CPU inference. Think carefully about how this might operate in reality."
<Intro for all 3 models below - no custom instructions>
GPT-5: "Short answer: more memory bandwidth absolutely helps CPU inference, but it does not magically make a central processing unit (CPU) “good at” large-model inference on its own."
Claude: "This is a fascinating question that gets to the heart of memory bandwidth limitations in AI inference. "
Gemini 2.5 Pro: "Of course. This is a fantastic and highly relevant question that gets to the heart of future PC architecture."
viraptor
Not really. Any prefix before the content you want is basically "thinking time". The text itself doesn't even have to reflect it, it happens internally. Even if you don't go for the thinking model explicitly, that task summary and other details can actually improve the quality, not reduce it.
BeetleB
I recently started using Open WebUI, which lets you run your query on multiple models simultaneously. My anecdote: For non-coding tasks, Gemini 2.5 Pro beats Sonnet 4 handily. It's a lot more common to get wrong/hallucinated content from Sonnet 4 than Gemini.
not_kurt_godel
Agreed. People talk up Claude but every time I try it I wind up coming back to Gemini fairly quickly. And it's good enough at coding to be acceptably close to Claude as well IMO.
mcintyre1994
Google also has a lot of very useful structured data from search that they’re surely going to figure out how to use at some point. Gemini is useless at finding hotels, but it says it’s using Google’s Hotel data, and I’m sure at some point it’ll get good at using it. Same with flights too. If a lot of LLM usage is going to be better search, then all the structured data Google have for search should surely be a useful advantage.
dpoloncsak
Does it still try to 'unplug' itself if it gets something wrong, or did they RL that out yet?
jjani
Not sure if you're joking or serious? Every model has "degenerate" behavior it can be coerced into. Sonnet is even more apologetic on average.
oasisbob
> because these Gemini models sometimes feel downright lobotomized compared to claude or gpt-5.
I'm using Gemini (2.5-pro) less and less these days. I used to be really impressived with its deep research capabilities and ability to cite sources reliably.
The last few weeks, it's increasingly argumentative and incapable of recognizing hallucinations around sourcing. I'm tired of arguing with it on basics like RFCs and sources it fabricates, won't validate, and refuses to budge on.
Example prompt I was arguing with it on last night:
> within a github actions workflow, is it possible to get access to the entire secrets map, or enumerate keys in this object?
As recent supply-chain attacks have shown, exfiltrating all the secrets from a Github workflow is as simple as `${{ toJSON(secrets) }}` or `echo ${{ toJSON(secrets) }} | base64` at worse. [1]
Give this prompt a shot! Gemini won't do anything except be obstinately ignorant. With me, it provided a test case workflow, and refused to believe the results. When challenged, expect it to cite unrelated community posts. Chatgpt had no problem with it.
[1] https://github.com/orgs/community/discussions/174045 https://github.com/orgs/community/discussions/47165
istjohn
You should never argue with an LLM. Adjust the original prompt and rerun it.
oasisbob
While arguing may not be productive, I have had good results challenging Gemini on hallucinated sources in the past. eg, "You cited RFC 1918, which is a mistake. Can you try carefully to cite a better source here?" which would get it to re-evaluate, maybe by using another tool, admit the mistake, and allow the research to continue.
With this example, several attempts resulted in the same thing: Gemini expressing a strong belief that Github has a security capability which is really doesn't have.
If someone is able to get Gemini to give an accurate answer to this with a similar question, I'd be very curious to hear what it is.
mips_avatar
IMO the race for Latency/TPS/cost is entirely between grok and gemini flash. No model can touch them (especially for image to text related tasks), openai/anthropic seem entirely uninterested in competing for this.
CuriouslyC
grok-4-fast is a phenomenal agentic model, and gemini flash is great for deep research leaf nodes since it's so cheap, you can segment your context a lot more than you would for pro to ensure it surfaces anything that might be valuable.
baby
why use grok? It seems like it's constantly being throttled in order to appear more right-wing
baby
Agree, Gemini is soooooo freaking fast, but I rarely use it personally because Anthropic/OpenAI model have such a better output
ta12653421
10 years ago: "before you marry someone, put the person in front of a really slow internet connection"
today: "before you marry someone, put the person in front of a slow AI model"
;-)
kanwisher
We had to drop Gemini api cause it was so unreliable in production, no matter how long you waited.
simianwords
The other day I heard gpt-5 was really an efficiency update
M4v3R
It was both efficiency and knowledge/reasoning update. GPT-5 excels at coding, it solves tasks the previous versions just could not do.
newfocogi
Non-AI Summary:
Both models have improved intelligence on Artificial Analysis index with lower end-to-end response time. Also 24% to 50% improved output token efficiency (resulting in lower cost).
Gemini 2.5 Flash-Lite improvements include better instruction following, reduced verbosity, stronger multimodal & translation capabilities. Gemini 2.5 Flash improvements include better agentic tool use and more token-efficient reasoning.
Model strings: gemini-2.5-flash-lite-preview-09-2025 and gemini-2.5-flash-preview-09-2025
Mistletoe
2.5 Flash is the first time I've felt AI has become truly useful to me. I was #1 AI hater but now find myself going to the Gemini app instead of Google search. It's just better in every way and no ads. The info it provides is usually always right and it feels like I have the whole generalized and accurate knowledge of the internet at my fingertips in the app. It's more intimate, less distractions. Just me and the Gemini app alone talking about kale's ideal germination temperature, instead of a bunch of mommy bloggers, bots, and SEO spam.
Now how long can Google keep this going and cannibalizing how they make money is another question...
yesco
It's also excellent for subjective NLP-type analysis. For example, I use it for "scouting" chapters in my translation pipeline to compile coherent glossaries that I can feed into prompts for per-chapter translation.
This involves having it identify all potential keywords and distinct entities, determine their approximate gender (important for languages with ambiguous gender pronouns), and then perform a line-by-line analysis of each chapter. For each line, it identifies the speaking entity, determines whose POV the line represents, and identifies the subject entity. While I didn't need or expect perfection, Gemini Flash 2.5 was the only model I tested that could not only follow all these instructions, but follow them well. The cheap price was a bonus.
I was thoroughly impressed, it's now my go-to for any JSON-formatted analysis reports.
indigodaddy
Google AI mode is excellent as well, which I guess is just Gemini 2.5 Flash I'd imagine as well?
kridsdale1
If you have access, try AI Mode on Google.com. It’s a different product from Gemini that tries to solve “search engine data presented in LLM format”.
Disclaimer: I recently joined this team. But I like the product!
jonplackett
I think “Non-AI summary” is going to become a thing. I already enjoyed reading it more because I knew someone had thought about the content.
paxys
As soon as it becomes a thing LLMs will start putting "Non-AI summary" at the top of their responses.
nharada
I'm stealing "Non-AI Summary"
crishoj
Any idea what "output token efficiency" refers to? Gemini Flash is billed by number of input/output tokens, which I assume is fixed for the same output, so I'm struggling to understand how it could result in lower cost. Unless of course they have changed tokenization in the new version?
Romario77
They provide the answer in less words (while still conveying what needed to be said).
Which is a good thing in my book as the models now are way too verbose (and I suspect one of the reasons is the billing by tokens).
minimaxir
The post implies that the new model are better at thinking, therefore less time/cost spent overall.
The first chart implies the gains are minimal for nonthinking models.
kaspermarstal
Models are less verbose, so produces fewer output tokens, so answers cost less.
jama211
Thank you for this, seems like an iterative improvement.
zitterbewegung
Okay this is a nitpick but why wouldn't you increment a part of the version number to signify that there is an improvement? These releases are confusing.
TIPSIO
This is also my beef...
Anthropic kind of did the same thing [1] except it back-fired recently with the cries of "nerfing".
We buy these tokens, which are very hard to do in limited tiers, they expire after only a year, and we don't even know how often the responses are changing in the background. Even a 1% improvement or reduction I would want disclosed.
Really scary foundation AI companies are building on IMO. Transparency and access is important.
Aeolun
Are your tokens at any risk of lasting longer than a year? When I buy them it’s generally because I expect to use them reasonably soonish.
Al-Khwarizmi
I wouldn't call that a nitpick, it's a major annoyance. Version numbers become useless with that kind of policy.
kridsdale1
The numbers are branding. The appear to be an indicator of a given year long training run. New “versions” are tweaks of the same base.
tempest_
Sure and that is why you can call it 2.5.<whatever>
They just don't want to be pinned down because the shifting sands are useful for the time when the LLM starts to get injected with ads or paid influence.
sally_glance
I wish they would actually explain it like that somewhere. Or publish the internal version numbers they must certainly be using to ensure a proper development process.
bl4ckneon
I would assume that it will supersede the model that they currently have. So eventually 2.5 flash will be the new and improved 2.5 Flash rather than 2.6.
Same way that openai updated their 4-o models and the like, which didn't turn out so well when it started glazing everyone and they had to revert it (maybe that was just chat and not api)
zitterbewegung
Even if it was just chat and or API I have used the API and I know that they have at minimum added the retraining date and time that they could just affix to the Gemini 2.5 Flash and Flash-Lite because when I use the API I have to verify that the upgrade of the backend system didn't break anything and pinning versions I assume is pretty common.
undefined
undefined
someguyiguess
Google has historically always made bad UX choices like this. Conway’s law definitely applies here. Too many different silos building every Google project.
hahn-kev
Most of their products are server based so there's no version really. Also they kill stuff off before it would ever be v2 anyway. Also also, they're still better than Microsoft, see Xbox and Windows.
aeon_ai
I think a Model-specific SemVer needs to be created to be clearer as to what degree of change has taken place, in the age of model weights.
Something that distinguishes between a completely new pre-training process/architecture, and standard RLHF cycles/optimizations.
minimaxir
Gemini 2.5 Flash has been the LLM I've used the most recently for a variety of domains, especially image inputs and structured outputs which beat both OpenAI and Anthropic in my opinion.
pupppet
Gemini 2.5 Flash runs circles around ChatGPT 5 for many of my tasks, I’m surprised it’s not more popular than it is.
Liwink
Gemini 2.5 Flash is an impressive model for its price. However, I don't understand why Gemini 2.0 Flash is still popular.
From OpenRouter last week:
* xAI: Grok Code Fast 1: 1.15T
* Anthropic: Claude Sonnet 4: 586B
* Google: Gemini 2.5 Flash: 325B
* Sonoma Sky Alpha: 227B
* Google: Gemini 2.0 Flash: 187B
* DeepSeek: DeepSeek V3.1 (free): 180B
* xAI: Grok 4 Fast (free): 158B
* OpenAI: GPT-4.1 Mini: 157B
* DeepSeek: DeepSeek V3 0324: 142B
simonw
My one big problem with OpenRouter is that, as far as I can tell, they don't provide any indication of how many companies are using each model.
For all I know there are a couple of enormous whales on there who, should they decide to switch from one model to another, will instantly impact those overall ratings.
I'd love to have a bit more transparency about volume so I can tell if that's what is happening or not.
minimaxir
Granted, due to OpenRouter's 5.5% surcharge, any enormous whales have a strong financial incentive to use the provider's API directly.
A "weekly active API Keys" faceted by models/app would be a useful data point to measure real-world popularity though.
eli
They kinda have that already, no? https://openrouter.ai/apps?url=https%3A%2F%2Faider.chat%2F
frde_me
I know we have a lot of workloads at my company on older models no one has bothered to upgrade yet
koakuma-chan
Hell yeah, GPT 35 Turbo
kilroy123
There are cheaper models. Could cut the bill in half or more.
tiahura
Primarily classification or something else?
mistic92
Price, 2.0 Flash is cheaper than 2.5 Flash but still very good model.
nextos
API usage of Flash 2.0 is free, at least till you hit a very generous bound. It's not simply a trial period. You don't even need to register any payment details to get an API key. This might be a reason for its popularity. AFAIK only some Mistral offerings have a similar free tier?
FergusArgyll
Yeah, that's my use case. When you want to test some program / script that utilizes an llm in the middle and you just want to make sure everything non-llm related is working. It's free! just try again and again till it "compiles" and then switch to 2.5
YetAnotherNick
Gemini 2.0 Flash is the best fast non reasoning model by quite a margin. Lot of things doesn't require any reasoning.
crazysim
Maybe the same reason why they kept the name for the 2.5 Flash update.
People are lazy at pointing to the latest name.
rohansood15
2.0 Flash is significantly cheaper than 2.5 Flash, and is/was better than 2.5-Flash-Lite before this latest update. It's a great workhorse model for basic text parsing/summary/image understanding etc. Though looks like 2.5-Flash-Lite will make it redundant.
koakuma-chan
Why is Grok so popular
minimaxir
Grok Code Fast 1 usage is driven almost entirely by Kilo Code and Cline: https://openrouter.ai/x-ai/grok-code-fast-1/apps
Both apps have offered usage for free for a limited time:
https://blog.kilocode.ai/p/grok-code-fast-get-this-frontier-...
ewoodrich
Yep Kilo (and Cline/Roo more recently) push these free trial of the week models really hard, partially as incentive to register an account with their cloud offering. I began using Cline and Roo before "cloud" features were even a thing and still haven't bothered to register, but I do play with the free Kilo models when I see them since I'm already signed in (they got me with some kind of register and spend $5 to get $X model credits deal) and hey, it's free (I really don't care about my random personal projects being used for training).
If xAI in particular is in the mood to light cash on fire promoting their new model, you'll see it everywhere during the promo period, so not surprised that heavily boosts xAI stats. The mystery codename models of the week are a bit easier to miss.
NitpickLawyer
It's pretty good and fast af. At backend stuff is ~ gpt5-mini in capabilities, writes ok code, and works good with agentic extensions like roo/kilo. My colleagues said it handles frontend creation so-so, but it's so fast that you can "roll" a couple of tries and choose the one you want.
Also cheap enough to not really matter.
SR2Z
Yeah, the speed and price are why I use it. I find that any LLM is garbage at writing code unless it gets constant high-entropy feedback (e.g. an MCP tool reporting lint errors, a test, etc.) and the quality of the final code depends a lot more on how well the LLM was guided than the quality of the model.
A bad model with good automated tooling and prompts will beat a good model without them, and if your goal is to build good tooling and prompts you need a tighter iteration loop.
coder543
I think it has been free in some editor plugins, which is probably a significant factor.
I would rather use a model that is good than a model that is free, but different people have different priorities.
YetAnotherNick
Non free has double usage than free. Free one uses your data for training.
Imustaskforhelp
I mean, I can kinda roll through a lot of iterations with this model without worrying about any AI limits.
Y'know with all these latest models, the lines are kinda blurry actually. The definition of "good" is being foggy.
So it might as well be free as the definition of money is clear as crystal.
I also used it for some time to test on something really really niche like building telegram bot in cloudflare workers and grok-4-fast was kinda decent on that for the most part actually. So that's nice.
BoredPositron
They had a lot of free promos with coding apps. It's okay and cheap so I bet some sticked with it.
davey48016
I think it's very cheap right now.
riku_iki
I think it is included for free into some coding product
keeeba
It came from nowhere to 1T tokens per week, seems… suspect.
Simon321
it was free
PetrBrzyBrzek
It’s cheaper and faster. What’s not to understand?
testycool
You can get it to be unhinged as well. It's awesome.
Hobadee
Am I using a different Gemini from everyone else? We have Google Workspace at my job, so Gemini is baked in.
It is HORRENDOUS when compared to other models.
I hear a bunch of other people talking about how great Gemini is, but I've never seen it.
The responses are usually either incorrect, way too long, (essays when I wanted summaries) or just...not...good. I will ask the exact same question to both Gemini and ChatGPT (free) and GPT will give a great answer while the Gemini answer is trash.
Am I missing something?
Twirrim
I've been finding it leaps and bounds above other models but I'm only using it via aistudio. I haven't tried any IDE integration or similar, so can't talk to that. I do still have to tell it to stop it with the effusive praise (I guess that also helps reduce context windows)
BlueGh0st
I have the same sentiment. I've never really had success using Gemini outside of translation. Although, even with that, Gemini would often refuse and I had to remind it that it does actually know other languages.
My most recent trials output single commas as responses to basic questions or it simply refuses the task on ethical grounds such as generating a photo of a backpack wearing a hoodie for some reason (it claimed harmful stereotypes and instead generated an ape).
Refusing to do perfectly ethical tasks is probably the most consist problem I've had.
ls612
I use Gemini almost exclusively for coding and 2.5 Pro is extremely good at it. It has revised hundreds of lines of academic code for me at a time and the results run correctly with only minor revision.
I will also say whatever they use for the AI search summary is good enough for me like 50% of the time I google something, but those are generally the simpler 50% of queries.
Al-Khwarizmi
It depends on what you use it for. For answering questions I tend to prefer GPT-5, but for writing (e.g. turn these informally written ideas/bullet points into a report/proposal/etc., now shorten it a bit, emphasize this idea more, etc.) it's the best by far IMHO.
mastercheif
I agree. I think it comes down OpenAI's superior post-training.
ChatGPT is better at:
A) Interpreting what I'm asking it for me needing to provide additional explicit context.
B) Formatting answers in a way that are easily digestible.
mupuff1234
> Google Workspace at my job, so Gemini is baked in.
I think the "baked in" Gemini models are different, try using Gemini through the actual Gemini site.
do_anh_tu
Maybe you are using it wrong.
fzimmermann89
The switch by Artificial Analysis from per-token-cost to per-benchmark-cost shows some effect! Its nice that labs are now trying to optimize what I actually have to pay to get an answer - It always annoys me to have to pay for all the senseless rambling of the less-capable reasoning models.
svantana
Did they? I'm looking at the Artificial Analysis leaderboard site now and I only see price as USD/1M tokens.
stephen_cagle
I still can't understand how functioning adults believe that releasing their work in two separate places is a good idea (Ai Studio and Vertex AI).
lysecret
Don’t forget they also have two versions for their genaisdk and you can also use their genaisdk through vertex great! Best part is all LLMs get horribly confused as well and mix different sdks etc.
Computer0
I wonder how Gemini subscribers feel!
Get the top HN stories in your inbox every day.
This really captures something I've been experiencing with Gemini lately. The models are genuinely capable when they work properly, but there's this persistent truncation issue that makes them unreliable in practice.
I've been running into it consistently, responses that just stop mid-sentence, not because of token limits or content filters, but what appears to be a bug in how the model signals completion. It's been documented on their GitHub and dev forums for months as a P2 issue.
The frustrating part is that when you compare a complete Gemini response to Claude or GPT-4, the quality is often quite good. But reliability matters more than peak performance. I'd rather work with a model that consistently delivers complete (if slightly less brilliant) responses than one that gives me half-thoughts I have to constantly prompt to continue.
It's a shame because Google clearly has the underlying tech. But until they fix these basic conversation flow issues, Gemini will keep feeling broken compared to the competition, regardless of how it performs on benchmarks.
https://github.com/googleapis/js-genai/issues/707
https://discuss.ai.google.dev/t/gemini-2-5-pro-incomplete-re...