You can stop doing it whenever, if your pup will eat it that way. You may find that when he's teething he struggles with it being dry, if he does soak it again. Sometimes soaking it is useful if the pup loses interest as it brings out the flavour a bit more. Again its trial and error I'm afraid!
Edited to add: Sorry I've said much the same as Gill!