I don't know if I could say something to him in a negative way because I would not want to hurt his feelings unless a larger group of the congregation was there to complain as well. There is safety in numbers?
Saying NEGATIVE - CORRECTIVE things to a pastor...
The Bible spells out exactly how to proceed when there is hurt, offense, sin.... when something "negative" needs to be said. See Matthew 18:15-22. But note that all this embraces an ATTITUDE (often the HOW and WHY of our words matter more the words themselves). We need to come with HUMILITY.... we need to come with LOVE ("speak the truth but ALWAYS with love" "treat one another as you yourselves desire to be treated"). We need to come with a deep, genuine awareness that the fault may be on MY part, not his. We need to come with the ONLY hope being reconciliation, forgiveness, restoration (not to "rub his nose in it"). ATTITUDE is critical.
We are to come PRIVATELY. No, there is not "safety in numbers." Jesus STRESSES this is to be private, one-on-one. Make an appointment for a PRIVATE conservation. Let him know in advance that you have a concern that involves him, so that he won't be broadsided. Don't tell your wife. Don't tell ANY other church member or any other soul on Earth. Meet with HIM. Check your ATTITUDE. Check your FACTS (because you may have your facts all wrong). Speak with LOVE. Share your perspective - but with the right attitude.
You will get MUCH further with the biblical approach than by publicly spreading gossip and negativity - and organizing an army against him.