From what I understand, many men are intimidated by overzealous HR dept. policies that exist in some office environments regarding sexual harassment. Clearly such harassment is wrong, but some companies take it too far and as a result some men feel like they would rather avoid any and all situations where they could be called out.
That's entirely possible.
If people are concerned that sitting next to you when there's a seat that isn't next to you could be seen as some kind of sexual advance they could be concerned about the implications of upsetting you, your husband, or HR.
I remember talking with a lawyer at a company I worked for some years ago, in a presentation about freedom from harassment. A question was asked about whether it was acceptable to ask someone out for a date, and even the lawyer couldn't give a straight answer as to when a simple request for a date crossed the line into harassment. It's easy to see why, from a man's perspective, it's easier to just avoid any contact that isn't strictly professional.