 And the text doesn't say that, but that's a reasonable thing to assume. And if that is the case, if they did say, ah, you're a spy, come to report back to our father, it would explain why. When Joseph thinks back to that day, the first thing he says as he speaks to them is, you are spies. You see, he's reenacting the scene.