There was no gold.
- Two of the men had no prospecting experience. The other was something of a blowhard. They spent all their time collecting fool's gold or some similar mineral.
- How would the bandits NOT recognize gold dust when they saw it? How could the wind blow away something a dense as gold?
- It's gold dust, not nuggets, and they hadn't melted it into bricks or coins yet, so they probably thought all gold has to look a certain way. And while gold is very dense, in dust form it's still dust and can can be blown away.