read 10000 chars from file, split by "!!!", then tokenize each part using tokenizer(text) except the last, add the last to the next first.   Do it until reaching million tokens, then yield return the tokenized array of texts, continue until the end of file