UnicodeDecodeError if message cannot be decoded as UTF-8 and using debug log

[fxzjshm]

At client.py

katcp-python/katcp/client.py

Line 548 in 7a94c7d

.format(self.bind_address_string, repr(str(msg))))

a str(msg) is called if logger is set to debug level, but in core.py

katcp-python/katcp/core.py

Line 384 in 7a94c7d

return byte_str.decode('utf-8')

if using python3 and the message is just some bytes which cannot be decoded as UTF-8, an UnicodeDecodeError is thrown.

A workaround is just set logger level to info, but I'm wondering is it considered a bug, and is there any better options?

Thanks in advance.

[lanceWilliams]

@fxzjshm thank you for raising this issue. If possible, could you provide a code sample that lead you to this problem?

[ludwigschwardt]

In katdal and katsdptelstate we use the surrogateescape error handler which avoids all UnicodeDecodeErrors. It maps undecodable bytes to alternative unused code points. Your resulting Unicode string will be valid but messy, and the user can then decide if that's an error or not:

return byte_str.decode('utf-8', errors='surrogateescape')

I see you only call .decode in Python 3, so this handler is an option then (not available on Python 2). It does affect all Message conversions though, so I'd think about it carefully.

[bmerry]

You could potentially also replace repr(str(msg)) with repr(bytes(msg)) in the logging call. That would cause all the log messages to be prefixed with a b and any non-ASCII bytes in the message to be escaped, but you'd have a byte string you can copy and paste to exactly reproduce the encoded message.