canonical encoding for term_to_binary/binary_to_term ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

canonical encoding for term_to_binary/binary_to_term ?

Benoit Chesneau-2
Following the recent discussion on maps ordering, I’m curious if someone has already worked on an implementation of a canonical term_to_binary/binary_to_term to be able to sign them and make sure we get the same data across the wire.

Is there  any lib that does that around? For now i’m just using a canonical version of JSON but it’s pretty inefficient …

- benoit
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: canonical encoding for term_to_binary/binary_to_term ?

Guilherme Andrade
The ETF format[1] is fairly straight forward so, performance considerations excluded, is shouldn't be too hard to code a custom encoder which creates a bijective mapping between terms and binaries.

A while back I worked a bit on an ETF decoder with customizable restrictions on which types to accept to decode (and which to reject); I'm sharing it[2] in case someone finds it enlightening. I found the trickier cases to be pids, ports, references and anonymous functions.

[1]: http://erlang.org/doc/apps/erts/erl_ext_dist.html
[2]: https://github.com/g-andrade/restricted_etf/blob/master/src/restricted_etf.erl

On 7 November 2017 at 20:07, Benoit Chesneau <[hidden email]> wrote:
Following the recent discussion on maps ordering, I’m curious if someone has already worked on an implementation of a canonical term_to_binary/binary_to_term to be able to sign them and make sure we get the same data across the wire.

Is there  any lib that does that around? For now i’m just using a canonical version of JSON but it’s pretty inefficient …

- benoit
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



--
Guilherme

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: canonical encoding for term_to_binary/binary_to_term ?

zxq9-2
In reply to this post by Benoit Chesneau-2
On 2017年11月07日 火曜日 21:07:45 Benoit Chesneau wrote:
> Following the recent discussion on maps ordering, I’m curious if someone has already worked on an implementation of a canonical term_to_binary/binary_to_term to be able to sign them and make sure we get the same data across the wire.
>
> Is there  any lib that does that around? For now i’m just using a canonical version of JSON but it’s pretty inefficient …

So far I've had success with converting everything to a canonical basic form of tuples and sorted lists. The trick there is that every message in a protocol (or document system, whatever) requires a known schema so that you can identify whether a list of things is a list that needs to be sorted, or is a text string that must never be sorted. Tagging elements with atoms works just fine for that going into the format and back out again, but my point is there is always a requirement for prior knowledge of the data.

This hasn't really been much extra annoyance in practice because the system where I do this already has well defined schemas, so writing an importer/exporter isn't much work. It is a bit less trouble than defining an ASN.1 schema to serialize the data as DER, which is what we did before, and that'S guaranteed to work everywhere. This may be "inefficient" in some sense but its never been a bottleneck.

Even with a BIF, the burden is always going to be on the programmer to canonicalize and mark strings prior to sending anything to a pre-sign canonical serialization function because there is no way a serializer can know the difference between the meanings of certain nested structures.

-Craig
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions