You definately need to talk to your vet and tell them that you will not be getting him neutered until you have had some behavioural advise. Tell him it is his behaviour around people you are concerned about, as his lack of confidence seems to be growing.
The incident when he went for you, he probably panicked by you reaching over him - and if he didn't actually bite, then he didn't intend to, so he showed good restraint there (I know it is hard to believe when you are in the situation, but if a dog intends to bite, the don't miss!!). Has the vet checked his eyesight??
Also, yes, if he has a tender tummy, he is more likely to react to stressors...
You really need to make sure no-one approaches him head on or pats his head - these are 2 of the most intimidating things we as humans do to dogs... better to approach in a curve, and just drop a treat by him as they pass.... (or throw it towards him as they pass further away, if he isn't comfortable with them close) once he is comfortable with this, they can progress to letting him sniff the hand with the treat in, before giving it.... then getting him to sniff a hand, then produce a treat from a pocket.... don't let strangers fuss him at all until he has shown a huge improvement in his confidence....
If I have to remove undergrowth from Honey I always ask her to wait, first, so I don't startle her - this may help Alfie...