EOSERV Bug Tracker > Bug #461: Incorrect Apostrophe Encoding in Map Names

Bug #461: Incorrect Apostrophe Encoding in Map Names

Incorrect Apostrophe Encoding in Map Names
ID #461
Submitter Cirras
Product EOMap
Severity Normal
Status OPEN, NEW
Submitted 24th Sep 2018
Updated 24th Sep 2018
Cirras Submitter 5 years, 35 weeks ago

For some reason, apostrophes are being saved into map names as byte 0x92. which isn't valid ASCII or UTF-8. This is part of the ASCII-extension windows encoding 1252, as far as I can tell, for "right single quotation mark". It's represented in the client like an odd backwards backtick instead of the ' that one would expect with 0x27.

My assumption is this is some weird bullcrap with windows smart quotes and the mapper form not sanitizing inputs properly for ASCII. It's ideal to stick with ASCII since UTF-8 is a superset and it makes serializing map information into JSON a pain in the ass if you're dealing with annoying windows encodings. (Outside the scope of this bug, I'm just noting it.)

Problems this has caused:

-Estimating text length of map names based off of the existing (partial) lookup table in EOSERV

-Problems when serializing map names into JSON due to 0x92 not being a valid UTF-8 byte.

How to fix:

- Add a check to EOMapper string decoding to fix existing names when they're opened/saved with the mapper. (https://pastebin.com/WMgMtRZr)

- Sanitize inputs on the map name form, somehow ensuring only valid ASCII is entered. I leave this one to your imagination, since I obviously don't have the source code for this and any such changes would probably just go to the EOMap "next gen" project anyway.

tl;dr: encoding bugs suck


Cirras Submitter 5 years, 35 weeks ago

Note: Checked the lookup table in util.cpp of EOSERV and found that it actually was not incomplete as stated previously - Looks like the value for 0x92 is just incorrect. (Unsure of the correct value)

Cirras Submitter 2 years, 9 weeks ago

In retrospect, I don't believe this is actually a bug. Map names are ansi strings (and the client reads them as such), so this is basically expected behavior.

Add Comment

Please don't post unless you have something relevant to the bug to say.
Do not comment to say "thanks" or "fix this please".

Please log in to add comments. EOSERV Bug Tracker > Bug #461: Incorrect Apostrophe Encoding in Map Names