It's not constant. It's only now and then, but when it happens it's pretty frustrating. Sometimes he thinks he heard me say one thing, when I said something else, and he'll react to what he thought I said rather than what I actually said. This can cause a problem until we figure out where the breakdown in communication happened.
Complicate matters with the fact that Asperger's is a possibility. He definitely has alexithymia, which means he can't put a name to feelings. Ask him how he felt when such-and-such happened, and he'll answer instead with what action he took when such-and-such happened. He honestly thinks he's answering the question. It's just that the word "feel" doesn't compute. Also, he is completely unable to detect tone of voice, whether his own or anybody else's. He can hear loud from soft or high from low, but he can't pick out little cues in the voice that would indicate boredom, sarcasm, impatience, etc.