Very intelligent people are generally very intelligent all the time. They have access to more frames of reference and more ways of thinking all the time.
Socially, this means less intelligent people will not follow some-to-much of what they say. Thoughts with later points building on ones made previously may not be understood, while jokes based on situational meta-analyses may not be appreciated. Indeed, some utterances of very intelligent people may even sound insulting to less intelligent listeners.
To make matters worse, very intelligent people themselves have a hard time understanding less intelligent people. They may laugh at associations a less intelligent person does not see, thus offending. They may see the end of a less intelligent person’s line of thought when it has barely begun. They may all too easily refute a conclusion that had taken the less intelligent person a long time to reach.
To make matters even worse, since very intelligent people are by definition unique, other very intelligent people may not understand them either. Unique people are unique in many different ways, while people closer to the center of the bell curve are more similar in more ways.
Of course very intelligent people also make many mistakes. Camaraderie with less intelligent people can appear patronizing and be boring to both parties.
All of the above causes social problems for very intelligent people. And this shows how difficult it can be for intelligence to increase in any society.
Indeed, there is some evidence that the general intelligence of first-world populations has been declining, a reverse Flynn effect.
This makes sense if we consider that first-world populations are larger and dominated more by the center of the bell curve than they were in the past. The center dominates more today not only because it is larger but also because media and public education amplify it much more than in the past.
This louder and stronger center surely makes it tougher for very intelligent people to fully develop. Social forces will tend to marginalize—even bully—them more than in the past.