Shrinksafe fails on files with UTF-8 BOM

[dmagliola@…]

When uploading a file that starts with the UTF-8 signature (EF BB BF), ShrinkSafe? returns a file that only contains the string "null" inside.

I understand that this is probably not a common occurrence, I found this out because Visual Studio 2005 adds this signature to files. When replacing the first 3 bytes with spaces (20 20 20) using an hex editor, it works perfectly.

[Adam Peller]

To clarify, I think you're saying it's the byte order mark, not the actual encoding which breaks shrinksafe? The Dojo build suffers from the same problem (#1660)

[dmagliola@…]

Yes, that's exactly what i'm saying. After the byte order mark (the first 3 bytes), the file is pure plain-text (it's a normal JS file, without any unicode characters) And when you take an hex editor and wipe out the first 3 bytes (replacing the with spaces (20 20 20)), ShrinkSafe? works perfectly.

[Adam Peller]

you can probably apply a similar patch to what fixed #1660. Note that there's a JDK bug so that you must identify the BOM as 0xfeff instead of 0xef 0xbb 0xbf

[alex]

(In [10922]) ensure that we're UTF-8 end-to-end. Fixes #2440. Fixes #4712. Fixes #2331