I have been on both sides of this situation
Coco has more or less grown out of it but she will still nick a ball if she sees a chance to swap it for a tasty treat.
She has also done far worse.
I just give my ball to the other owner if they don't want to wait for me to get it off her - most dog owners are fine about it. This guy sounds a bit uptight to me so I really would not worry about it
If he gets so angry about something like that, I dread to think what he is like to live with
To be honest, if that really is Pepper's worst "issue", you really have very little to worry about
You can just carry on training him to swap his treasures for a high value treat or toy - he will get there eventually but there is no point you stressing about it as it will just make him more wound up too