The bandits from the first game weren't so much "bandits" as a rogue kingdom.
Think about it. What kind of bandits have an army and a fortress? Maybe the Kingpin wasn't so much a kingpin as a true king. The Pillagers are the Kingpin's sons.
They do look a little alike.