Try not to worry too much - whats done is done; obviously, it is better if puppies are not bought in these circumstances, but the most important thing now is make sure you do a good a job as you possibly can with your pup
It sounds like he has already been to the vets; did he get a clean bill of health? Having been brought up in close proximity to a lot of other dogs, then he may have picked up some parasites - ear mites and worms are not uncommon but are easily treated
I would suggest that you socialise and habiutalise him as much as possible - it may sound extreme, but I made a list of all the different experiences that Molo needed based on the list in the back of Gwen Baileys book "The Perfect Puppy" and ticked them off - my aim was to try and make sure he met a wide range of people (old, young, beards, walking sticks, different hair etc) and went to as many different places (farm, town, country, station (bus/train), ferry (we live on an island) etc etc). Puppy classes are essential imo - many vet surgeries run puppy parties, which helps them learn how to interact with other dogs, and other people. There are also classes that focus on how to train your dog basic obedience/commands
The important thing is that you know now that this kennels is less than ideal, and so ots probably best not to recommend them, and even advice people not to go there if they ask where you got your pup
I think your suspicions are probably correct; good breeders would allow you to meet the puppies dame, and are unlikely to have x-breed litters