Is it rude to stare at someone while talking?

Not really. He's just paying attention to you. If he were looking around everywhere else, you might think he wasn't really listening.