Anyone who follows this blog knows that I'm not big on rules of writing. But in my experience as an author, a reader, and an editor, I've found that the word "that" is one of the least-needed, that it is among the most overused and misused words, in all of modern literature.
Notice how the "thats" add nothing to the passage. They don't clarify, they don't improve flow, and they don't reflect any sort of stylistic choice, either. They're just taking up space and bloating word count. The word "that" is only rarely actually needed in a sentence, but for some reason, an awful lot of writers are in the (bad) habit of peppering their prose with this largely superfluous word. Consider the following, typical constructions:
He knew that I wasn't going away.
vs.
He knew I wasn't going away.
She was sure that everything would be fine.
vs.
She was sure everything would be fine.
You get the idea. When you find yourself tempted to include a "that" in a sentence immediately following a verb or adjective, try the sentence without the "that" first. In the vast majority of cases, you'll find your meaning is perfectly clear, and your prose much tighter, without it. Now look at these constructions:
None of the cars that we saw were suitable.
vs.
None of the cars we saw were suitable.
The books that I needed weren't in stock.
vs.
The books I needed weren't in stock.
Again, the "that" adds nothing but characters on the page. As with "thats" following a verb or adjective, try any sentence where a "that" follows a noun without the "that", and see if it doesn't read tighter.
So when is it appropriate to use "that"? When it's needed to clarify your meaning:
As a pronoun - That is the hotel where we stayed last time we were here.
As an adjective - I'm pretty sure that book belongs to Jimmy.
Or to improve the flow of your prose, as a stylistic choice:
As an adverb - It didn't matter all that much.
(compare this to)
It didn't matter much.
The judge that heard the case was biased.
vs.
The judge who heard the case was biased.
All the kids that came to the party had a good time.
vs.
All the kids who came to the party had a good time.
Presumably, the judge is a person, not a thing. Kids are people, too. People are "whos", not "thats".
However, both of these examples illustrate the case where a relative clause requires an object to restrict an antecedent, which is just a fancy-pants grammarian way of saying a "that" or "who" really is needed to clarify the meaning of the sentence and make it grammatically correct. Just be sure to use "who" in reference to people, and "that" in reference to things.
One more thing: are you mixing up your "whiches" with your "thats"? Consider:
The dog that was found in her yard was a stray.
vs.
The dog, which was found in her yard, was a stray.
Note the difference in meaning. In the first sentence, the fact that (<-- see, even I think they're necessary sometimes!) the dog was found in her yard is an important detail. The use of "that" makes the subject, "dog", more specific. We're not talking about the dog she saw in the park, or the dog on the corner. In this type of usage, the phrase, "that was found in her yard" is called a "restrictive phrase".
In the second sentence, the fact that the dog was found in her yard is incidental. The sentence could be shortened to, "The dog was a stray," with no loss of meaning or clarity. In this type of usage, the phrase, "which was in her yard," is called a "nonrestrictive phrase". The phrase does not exist to better specify a particular dog, it's there to provide additional information about the dog.
It can be a tricky distinction. Let the comma be your guide when choosing between the two words. In general, if the desired meaning or emphasis of a given sentence is best conveyed when a descriptive phrase within it is offset by commas, you're looking at a nonrestrictive phrase and "which" is the right way to go. Conversely, if such a phrase is not surrounded by commas, you're looking at a restrictive use and can safely go with "that".
Also note, what's grammatically correct isn't always a match with the most commonly-accepted usage:
The puzzle pieces which we couldn't find this morning turned up under the cushions.
Gah! This sentence is like fingernails on a chalkboard to a grammarian's ears because it uses "which" as part of a restrictive phrase. Yet the sentence will seem correct to most readers because most of them think the rule used to divide the "thats" from the "whiches" is based on whether or not the noun in the sentence is plural. Look at the grammatically correct version of the same sentence:
The puzzle pieces that we couldn't find this morning turned up under the cushions.
It sorta kinda doesn't "sound" right, does it? It's because the incorrect usage has become more ubiquitous than the correct one. You can moan and complain, stamp your little grammarian feet, and even threaten to pull out your Chicago Manual of Style, but about 99 times out of a hundred you will lose a bar bet on this. It's usually fine to go with the more common, incorrect usage in such a case, but better still to avoid the whole kerfuffle and dispense with both words whenever possible. Remember, most often, you'll find a given sentence is just fine without either one:
The puzzle pieces we couldn't find this morning turned up under the cushions.
But if you should find yourself in a bar with nothing better than grammar to bet on, whip out your web-enabled phone and bask in the victory round - Grammar Girl's got your back.