Millie suffers with this, and always has; every now and then it was like she'd forgotten what you looked like, or just couldn't see you - we never worried about it; until she started losing sight in half light, and reacted badly to being approached in half light / dark. However she was subsequently diagnosed with PRA about 6 years ago, so now we feel that her sight has never been 100% hence these little 'blips'.
Never had it was Jennie, nor Hazel (so far!)
As it's only happened once perhaps it's just a one off and Toby was genuinely just a bit confused. But the next time you go to the vets it's probably worth a check over, I possibly wouldn't be rushing there myself though, unless it was consistently happening, or his eyes looked sore / weepy etc.