In the Stranger's confrontation with the town's hired guns in the bar, he grabbed a bottle in order to test their reaction. He knew who would probably draw first, who was hesitant, who was scared, and how fast they were. So when he tells them, "I'm faster than you'll ever live to be," it wasn't just bravado, it was the truth based on information that was given.
Callie's failure to injure the Stranger, despite shooting multiple times at him from less than 10 feet away, can either be chalked up to extremely bad aim or the possibility that the Stranger is not human.
A more mundane answer is she's overwhelmed with emotion after having just been raped and firing blind with anger.
If the Drifter is a ghost then he's almost certainly damned himself with his murders, rape, and mental torture of his enemies—no matter how deserving.
If the Stranger is the ghost of Marshall Jim Duncan, and Mordecai finally giving the murdered Marshall a grave marker with the words "Rest in Peace" may be part of why the Stranger shimmers out of existence the way he shimmered in and it's possible that if the Townspeople had at the very least done that in the first place the Stranger might not have turned up at all.
