Handle misencoded license text files graceful.

[schlenk]

This handles #868

If a license file is not decodeable as UTF-8, it uses the fallback to io2str using chardet to determine the encoding will be tried.

It also adds some tests to exercise the code:

• A UTF16 encoded file. chardet fails to detect it as UTF-16, so it gets skipped.
• A RTF file. This is technically text/rtf (or application/rtf, but misdetected as application/msword...). It is technically UTF-8 and gets properly encoded.
• A plain text UTF-8 file. This gets properly encoded (but maybe the XML encoding should have xml:space="preserve" set, to properly protect newlines).

[jkowalleck]

the current implementation seams utterly complex.
please refactor it.

[schlenk]

the current implementation seems utterly complex. please refactor it.

A good chunk of the complexity stems from the original code in pep639.py, which i shifted around a little.
I could change some of those, e.g. kick out the base64 encoding path and the looking into multiple locations loop, as only 'licenses' is mentioned in the PEP, if that would help.

[jkowalleck]

@schlenk, I've rearranged the tests via d95dc73

[schlenk]

@schlenk, I've rearranged the tests via d95dc73

Thanks. Looks much cleaner. I found it somewhat hard to figure out how to add good tests in the first place. Is there some documentation about the test concept/how to add good tests? As there is already quite a bit of scaffolding and infrastructure present, but not entirely obvious at first.

[jkowalleck]

@schlenk , after revieweing, I propose the following simplification: schlenk#1

If you agree, just merge the pullrequest, and it will apply upstream

[jkowalleck]

Something is off on windows.
not gonna lie, i've expected such a thing, since windows often auto-converts line endings - which causes unexpected situations

[jkowalleck]

Something is off on windows. not gonna lie, i've expected such a thing, since windows often auto-converts line endings - which causes unexpected situations

will try to fix this