Your people were banished for heresy.
It's unlikely they were banished for simple crimes, given that they are entire family units with children. Heresy makes a lot of sense.- In real life, at least among the English-speaking world, colonies in previously little-explored places often sprung up from the migration of (often unwelcome/persecuted) religious minorities—Puritans, Quakers, Catholics, Mormons, what have you—so this isn't such a wild guess.
The people were banished for incest.
It's a relatively small group, with nowhere near enough members to ensure genetic diversity, even with the occasional nomad group wandering by, right from the get go, they will intermarry, despite the fact that, give or take a few generation, everybody is everyone else's cousin.