You can say 'How was I to know?', or in British English 'I wasn't to know', to mean that you cannot be blamed or criticized for something you did, because you did not have enough information to realize that it was wrong.
How was I to know you'd return so suddenly?
You weren't to know–in fact, I think I'm the only person who does know.