Batch Tansform and accents in the json file
Hi.I used the notebooklab2_batch_transform.ipynbof@philschmidto launch batch for inferences. My dataset is composed of texts in Portuguese (ie, with accents).When I download the json file created and open it withSublime Textornote, I see that all letters with accents were converted as following:(...)
{"inputs": "forma\u00e7\u00e3o ...}
(...)In this example,forma\u00e7\u00e3oisformação.What do you think?I can use my json file or I need to solve this problem in order to get (real) letters with accents in my json file? Thanks.
2021-12-14T08:32:13Z
You can encode your json file correctly usingwith open('keys.json', encoding='utf-8') as fh:
    data = json.load(fh)
2021-12-14T13:17:22Z
Thanks@philschmid. I tested your code withencoding='utf-8'but it did not change the content of my json file with strange letters instead of letters with accents.However, I just found the complementary code in thispostthat solves my problem:ensure_ascii=Falseas an argument ofjson.dump().Here the code I use now(taken from notebooklab2_batch_transform.ipynband modified with the 2 cited arguments):with open(dataset_csv_file, "r+") as infile, open(dataset_jsonl_file, "w+", encoding='utf-8') as outfile:
    reader = csv.DictReader(infile)
    for row in reader:
        json.dump(row, outfile, ensure_ascii=False)
        outfile.write('\n')