Dialogue is an area where many writers struggle. This is pretty ironic when you consider that words are writers' stock in trade, and unless a given writer is mute, he or she has been plying that trade since about the age of eleven months. Yet while most of us communicate normally and without much difficulty in our everyday lives, for some reason many of us have a tendency to go all flowery, choppy, melodramatic or wooden when it comes time to put words in our characters' mouths. Avoiding the following dialogue traps will go a long way toward making your dialogue more natural and believable.
1. No two people talk exactly the same. In believable stories, as in life, each person will have his or her own rhythms of speech, pet phrases and regional or family expressions. This doesn't mean each character should broadcast his geographic or cultural background with every sentence, however. It just means that if, by about a quarter of the way through the book, a reader can't tell your characters apart merely based on their dialogue, you haven't made each character's "voice" distinctive. The important thing here is to be subtle when drawing those distinctions. If you're not sure what this means or how to go about it, here's an exercise to try: the next time you're in a crowded, public place, pay attention to the bits and pieces of conversation floating all around you. Notice how different people express the same thoughts differently.
For example, where one person might say, "I called Sally," another might say, "I phoned Sally," or, "I rang Sally." Where Joe (in his forties) says, "That whole night was a waste of time," Jake (a twentysomething) might say, "Two words: epic fail," and Steve (an ex-military man) might say, "FUBAR, all the way, man." Thinking about your characters' backgrounds, histories, and even biases and motivations when constructing their dialogue will help in making their voices distinct from one another.
2. Life is not a movie. While heated exchanges, adamant diatribes and weepy heart-to-hearts all have their bit to contribute in various stories, they should be used sparingly if you don't want your novel to read like a soap opera script. If you're prone to succumb to melodrama in your dialogue, try reading it aloud. If the words feel or sound unnatural coming out of your own mouth, they shouldn't be coming out of your characters' mouths, either. Of course there's some wiggle room here if you're writing something historical, a fantasy, sci-fi, or anything else with purposely unusual language.
3. Men and women communicate differently. This really boils down to a single, simple concept. Speaking in gross generalizations, the masculine communication style is based on utility, whereas the feminine communication style is about socialization.
In the masculine, words are used to accomplish some goal. The goal is usually imparting necessary---and that word, "necessary", is key here---information, but it can also be to quickly size up a person or situation, or to establish or reinforce the pecking order (e.g., teasing). Generally speaking, believable masculine characters talk less than feminine characters, and get to the point pretty quickly. With feminine characters, a given conversation need not have an intrinsic point: the point of the conversation may simply be for the feminine characters to hear and be heard, and feel validated by one another as a result. But having said that, I'd caution against too much mutual navel-gazing on the part of your feminine characters, lest you bore your readers.
4. In general, the words should not draw attention to themselves. Dialogue should never take your reader out of the story, for any reason. If your reader must reach for a dictionary or fire up some device that has access to Wikipedia in order to understand what the heck that character is talking about, that reader is being pulled out of the story world.
While particularly intellectual characters may employ five-dollar words at times, try to err on the side of conservatism in that area. If you can substitute a word or phrase that's better-known, though still only rarely used in everyday conversation, make the change. Similarly, if a given character wants everyone to think she's worldly and well-traveled she may pepper her speech with foreign words, and that's appropriate. Just make sure the foreign words are familiar to most readers, or that their meaning is adequately conveyed through context.
5. Dialogue that's used for exposition will sound stilted 99.99% of the time---so don't do it! As a general guideline, characters should NEVER say things to one another only for the purpose of conveying necessary information or background to the reader. If a given character might just as well open his bit of talking with, "Well, since the reader probably doesn't know anything about particle physics, let me give you a thumbnail sketch of string theory," then you're doing something wrong. Find a way to get the expository into your story in other ways: through actions, settings, and so on. Consider the following example.
Michael was physically and mentally abused for years at the hands of his mother and as a result, he has a great deal of trouble extending trust to any females. This history informs the character and actions of Michael, but is not a central focus of the story at hand. When Michael visits a new girlfriend's home for the first time, one writer might include a confrontation between the two characters in which the girlfriend voices concerns about Michael's unwillingness to open up to her and Michael responds by spilling his guts about his mother. A better writer will have Michael flinch when the girlfriend removes her belt while changing out of her work clothes, when she playfully quotes an overbearing female movie or TV character, or when she reaches for the knife block while preparing dinner, and then have the girlfriend notice this.
6. When in doubt, read it out...loud. This goes back to trap #2, but it bears repeating.