the Pentagon is not on the ground - how LLMs fail to intuit
Posted: Tue Apr 08, 2025 4:08 pm
A snippet of conversation with Gemini 2.0 Flash:
LLMs struggle with the most obvious things precisely because they are obvious. Humans don't say the most obvious things because we all already understand them, and so they aren't in the training data. A lot of people assume LLMs can intuit obvious things from context, because all the context available implies these things. In some cases, they do appear to manage this, but if they were truly intuiting information, why would they be blind to something so obvious?
Another example: How many towers were in the World Trade Center? The answer is three, but there is so much written about the Twin Towers everywhere that almost no one remembers that there was a third tower at 7 World Trade Center. It was damaged by debris from the collapse of the Twin Towers, caught fire, and then collapsed hours later. There's also a 7 World Trade Center today, it's another tower. They decided to rebuild it.
LLMs follow the trends, what's popularly said. Even in a conversation where you clearly establish the existence, non-existence due to collapse, and current existence of a third tower in the World Trade Center, they will consistently forget this fact and argue that there were only two towers.
(I would appreciate more examples if you have them!)
---
Hi. This is Tangent, or Guard13007 depending on where you know me from. I accidentally didn't sync my password database, so I don't have access to my main account right now.
Don't trust someone making claims like this. Wait until the user account with my name actually confirms it, silly.
I should be clear that while the specific snippet is a little absurdist, nothing was said that should influence the ability of the model to understand whether or not a building is on the ground. I highlight this because to me it is an example of how LLMs cannot intuit information. How many places on the internet specify that buildings are on the ground? I would wager not many. It's something so obvious that no one would ever say it.User: No, the Pentagon was on the ground. It has nothing to do with the planes that hit the towers.
Gemini: The Pentagon is not on the ground. It is a five-sided building in Arlington, Virginia. The Pentagon was hit by a plane during the 9/11 attacks. Two planes hit the World Trade Center.
LLMs struggle with the most obvious things precisely because they are obvious. Humans don't say the most obvious things because we all already understand them, and so they aren't in the training data. A lot of people assume LLMs can intuit obvious things from context, because all the context available implies these things. In some cases, they do appear to manage this, but if they were truly intuiting information, why would they be blind to something so obvious?
Another example: How many towers were in the World Trade Center? The answer is three, but there is so much written about the Twin Towers everywhere that almost no one remembers that there was a third tower at 7 World Trade Center. It was damaged by debris from the collapse of the Twin Towers, caught fire, and then collapsed hours later. There's also a 7 World Trade Center today, it's another tower. They decided to rebuild it.
LLMs follow the trends, what's popularly said. Even in a conversation where you clearly establish the existence, non-existence due to collapse, and current existence of a third tower in the World Trade Center, they will consistently forget this fact and argue that there were only two towers.
(I would appreciate more examples if you have them!)
---
Hi. This is Tangent, or Guard13007 depending on where you know me from. I accidentally didn't sync my password database, so I don't have access to my main account right now.
Don't trust someone making claims like this. Wait until the user account with my name actually confirms it, silly.
