Bolt says he was counting on taking baby daughter Olympia to Tokyo