It may be, but if I do and I think there is no offense in my reason, I'll tell them why. My neighbors offered me some food a few years ago during the holidays. They didn't know I was Vegan. This is a family with children and seem like decent people so in order to stop them buying food for me I will not eat, I explained that I could not accept their particular offering because it contained animal products. No one was offended and it was fine.
I suppose context matters. I live in suburbia. If I lived in a city apartment and some stranger just wanted to offer me food for no apparent reason I probably wouldn't outright refuse but I'd wonder why and likely not eat it. I guess the difference between this and the suburban neighbor is...one, they had a family, two, they are likely to be living next to me for some years. There's therefore not a reason to suspect anything nefarious.