Ben, a guilty pleasure is something one enjoys and considers pleasurable despite feeling guilt for enjoying it. The "guilt" involved is sometimes simply fear of others discovering one's lowbrow or otherwise embarrassing tastes. Fashion, video games, music, movies, and junk food can be examples of guilty pleasures.
For example, one might say that Nickelback is their guilty pleasure...because even though Nickelback sucks, they still enjoy listening to their music.
NOTE: I do not listen to Nickelback.
My guilty pleasure is Christian metal, because I grew up listening to it but am no longer Christian, and my taste has shifted away from metal. It's still fun to listen to every once in a while for old times' sake.