The "distant past" thing surely isn't hard to believe, since most of the Culture novels take place between 1300 and 2100 CE (aka AD).
Now, why would the Culture start on Golter? Because Golter lies in the middle of nowhere - literally. It's inhabited by an old race that has made several attempts at reaching space. It has many of the prototypes of later Culture culture; the Free Land zones mimic the various subcultures of the Culture and its tendency to spawn groups that will split away; there are primitive A.I.s and robots; the culture is extremely hedonistic in places, and there's a laissez-faire attitude toward just about everything. It even has a primitive version of Culture habitats, both in the Log Jam and out in space.
The Culture acts in many ways as if it wants to get to know EVERYONE. Isn't that how a race that was born on a star "millions" of light years from anywhere else would behave?
All right, so some of this could be attributed to the fact that Banks's style will tend toward certain themes, but even so...
- This book and the Culture book Use of Weapons both make reference to a weapon called a Hand Cannon. Again, this could be just a similar style, but it suggests that they are in the same universe.
- Hand Cannon is a very common term for a large caliber pistol, so common that it's a trope title itself.
- Even though the Culture novel Excession explicitly talks about an approximately Napoleonic-equivalent era battle forming a hegemony on one of the Culture's home planets, which eventually goes on into space, with none of the (extremely long) history of Golter involved? Also, in the same novel, one of the Culture's oldest inhibitants - and its age - is discussed; Phage Rock IIRC. Your theory simply doesn't hold water, I'm afraid.
- The Culture is specifically noted to have been formed by the union of several spacefaring species some considerable time after they all went interstellar. As WMGs go, this one's fine.
