Q:

BOM character copied into JSON in Python 3

Inside my application, user can upload the file (text file), and I need to read it and construct json object for another API call. 
I open file with 
f = open(file, encoding="utf-8")

get the first word and construct Json object,...
My problem is that some files (especially from Microsoft environment) that have BOM object at the beginning. Problem is that my Json now have this character inside
{
   "word":"\\ufeffMyWord"
}

and of course, the API is not working from this point on. 
I obviously miss something, because, shouldn't utf-8 remove BOM objects? (Because it is not utf-8-sig). 
How to overcome this?

A:

No, the UTF-8 standard does not define a BOM character. That's because UTF-8 has no byte order ambiguity issue like UTF-16 and UTF-32 do. The Unicode consortium doesn't recommend using U+FEFF at the start of a UTF-8 encoded file, while the IETF actively discourages it if alternatives to specify the codec exist. From the Wikipedia article on BOM usage in UTF-8:

The Unicode Standard permits the BOM in UTF-8, but does not require or recommend its use.
[...]
The IETF recommends that if a protocol either (a) always uses UTF-8, or (b) has some other way to indicate what encoding is being used, then it "SHOULD forbid use of U+FEFF as a signature."

The Unicode standard only 'permits' the BOM because it is a regular character, just like any other; it's a zero-width non-breaking space character. As a result, the Unicode consortium recommends it is not removed when decoding, to preserve information (in case it had a different meaning or you wanted to retain compatibility with tools that have come to rely on it).
You have two options:

Strip the string first, U+FEFF is considered whitespace so removed with str.strip(). Or explicitly just strip the BOM:
text = text.lstrip('\ufeff')  # remove the BOM if present

(technically that'll remove any number of zero-width non-breaking space characters, but that is probably what you'd want anyway).
Open the file with the utf-8-sig codec instead. That codec was added to handle such files, explicitly removing the UTF-8 BOM bytesequence from the start if present, before decoding. It'll work on files without those bytes.

