It's not because a specific arrangement of syllables is inherently evil. The Bible gives no list of words to avoid; it only gives one, which is the Lord's name (more on that later).
I think people get far too caught up in avoiding specific words. They'll use replacements like crap, heck, etc. Yet the point of passages such as Ephesians 4:29 and 5:4* is not to say that some words are not okay, and other words with identical meanings are. The point is the intent of the words. Does it tear someone down? To say, "you piece of crap" is no more morally right than the alternative, because the intent is still to use language to harm someone else.
This is not to say that all swear words are okay in some circumstances. Some simply have such sexual meanings that there really is no acceptable reason to use them. Others reduce someone's humanity by definition, and to call someone made in God's image a female dog is inherently wrong. Also, the word "damn" literally means to condemn to punishment in Hell, so that's generally a no-go.
Personally, I do not particularly care if someone swears out of surprise, or general frustration. However, there is one more thing to consider. (Well, there's a lot more, but whatever.)
Romans 14:13-23, copied from biblegateway.com: "13 Therefore let us not pass judgment on one another any longer, but rather decide never to put a stumbling block or hindrance in the way of a brother. 14 I know and am persuaded in the Lord Jesus that nothing is unclean in itself, but it is unclean for anyone who thinks it unclean. 15 For if your brother is grieved by what you eat, you are no longer walking in love. By what you eat, do not destroy the one for whom Christ died. 16 So do not let what you regard as good be spoken of as evil. 17 For the kingdom of God is not a matter of eating and drinking but of righteousness and peace and joy in the Holy Spirit. 18 Whoever thus serves Christ is acceptable to God and approved by men. 19 So then let us pursue what makes for peace and for mutual upbuilding. 20 Do not, for the sake of food, destroy the work of God. Everything is indeed clean, but it is wrong for anyone to make another stumble by what he eats.21 It is good not to eat meat or drink wine or do anything that causes your brother to stumble.22 The faith that you have, keep between yourself and God. Blessed is the one who has no reason to pass judgment on himself for what he approves. 23 But whoever has doubts is condemned if he eats, because the eating is not from faith. For whatever does not proceed from faith is sin."
This passage is speaking about clean and unclean food, but I think it applies to the use of language as well. If, by our words, we cause someone else to stumble in their faith, then we have messed up.This is the best argument against swearing. Although it may not be inherently wrong to use some words, if you use them in the presence of others, and it causes them to stumble, then it is wrong.
*Eph. 4:29 "Do not let any unwholesome come out of your mouths, but only that which is helpful for building others up according to their needs, that it may benefit those who listen,"
5:4 "Nor should there be obscenity, foolish talk or coarse joking, which are out of place, but rather thanksgiving."
Song of the Day: Pieces, by Amanda Cook