I think a lot of it is down to the temperament of the dog.
Goldie is very submissive and I can take away anything from her....food included and she never growls.
Lottie however, who is much more dominant, used to growl when I took away her food bowl.
I just made a habit of taking things from her and if I took her food bowl, then I replaced it with something extra tasty in the bowl. That way she learns that she gets a treat when I take things away from her.
She is much better with the growling now, although I have just started giving her pigs ears and she tried it again...lol. So back to working on it with her
I think the growling is a normal reaction from some dogs, just let him see that it is ok to trust you to give it back.