This trope, admittedly, has been bolstered by the idea that society's existence is contingent on that of top-down leadership, but also because of the "propaganda by the deed" violence some anarchists about a century ago perpetrated against the robber barons of the gilded age and various heads of state — and innocent bystanders and other random individuals (one notorious bombing was of a French cafe on the grounds the people there were bourgeois). While not diminishing the terrorism that occurred, anarchists reject the notion of their belief meaning or necessarily resulting in violence. Anarchists advocate opposition to oppressors and their instruments, but in nearly all cases seek to peacefully produce voluntary, bottom-up order in their stead. There are in fact whole branches of anarchism, such as Christian anarchism and anarcho-pacifism, to whom violence is utterly unacceptable, sometimes to extreme lengths. Leo Tolstoy, the famed writer and a Christian anarchist in his later days, said one should not use violence even for self-defense, which no other anarchist tendency follows.
Of course, there have been anarchists throughout history who appear to have played this trope chillingly straight. Max Stirner stood for egoist anarchism-the belief that everyone should be allowed to do whatever they want, period. However, like most egoists, Stirner believed voluntary cooperation and avoiding chaos was in the self-interest of every human being, and thus a (sane) egoist would not necessarily want chaos. Furthermore, he made it clear that "self-interest" was to be defined by the individual in question and had no problem with altruistic acts so long as they were done with the knowledge that one was doing them because it was what they truly desired and not in service to an abstract ideal that they viewed as separate from themselves. A French variation was illegalism, the idea that committing crimes was the only true expression of anarchy. And don't get started on the Russian nihilists and anarchists of the 19th century, many of whom were pacifists but more than a few of whom were willing to commit terrorism and murder in the name of combating repressive institutions, such as to bring down the absolute monarchy of Russia and other things seen as parts of the bourgeois control. These ideologies reached their climax in the 1881 assassination of the Tsar, which got most of the perpetrators hanged and helped to undermine the Russian anarchist movement generally (though it must be noted that in 1900-1910s terrorism rates climbed sharply, to the extent that each year thousands of government officials, from constables to governors, were killed or maimed; this is often forgotten). Use of political violence to make change reached an ugly, bloody conclusion in the 1917 October Revolution of the Bolsheviks, the repression of all opposition (including the anarchists) during the subsequent civil war, and Stalin's purges in The '30s and '40s.
In the USA, "anarchist" tends to call up images of Nietzsche wannabes, ultra-leftist radicals (though Anarchism and Marxist Communism have been miles away from each other since Marx and Bakunin were rivals in the First International-you'll see anarchists and communists work together in protests, however) and Bomb-Throwing Anarchists, while "Libertarian" tends to call up images of redneck Crazy Survivalists, Right Wing Militia Fanatics and Social Darwinist Corrupt Corporate Executives (and hippies who want marijuana legalized). Worldwide, "libertarian" and similar terms gravitate to the same meanings as "Anarchist" does stateside, though of the socialist variety.
Predictably, anarchists have been historical targets of brutal persecution from both capitalist and socialist authoritarians, due to them believing that both are wrong in their domineering ways. Note, however, that there are both capitalists and socialists who use the term "anarchist" for themselves, and that different branches of anarchist thought disagree with each other just as much as any other factions of such do.
It should be mentioned that anarchy in the popular lexicon refers to any lack of a formal state government, even an involuntary one. Thus most times anarchy has actually existed, it usually is dominated by a lot of violent power struggles. While referring to failed states as in a state of Anarchy isn't in line with anarchistic philosophy, most people would view the distinction as just splitting hairs. So from a simple matter of linguistics there is a justifiable reason for the association between anarchy and chaotic violence.
