Passing data between Erlang and C port while preserving padding and alignment?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Passing data between Erlang and C port while preserving padding and alignment?

Frank Muller
Hi guys

Let assume the following C structure (I have a bunch of them, all different):

struct foo {
 int32 i32;
 char c;
 int64 i64;
 char * p;
 float f;
 char * str[16]; 
}; 

When it gets compiled, GCC will add some padding to align data types (i.e self-alignment). More info at:

http://www.catb.org/esr/structure-packing/

On the Erlang side, it is very easy to generate a binary which matches this C structure field by field.
But it won’t work because of the padding :-/

Question: how can I derive the GCC's padding, and automatically add it to my Erlang binary?

I am open to any generic/flexible solutions even if I have to auto-generate these C structures (i.e meta-programming).

Thank you.
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Vance Shipley
On Fri, Dec 7, 2018 at 9:51 AM Frank Muller <[hidden email]> wrote:
> Question: how can I derive the GCC's padding, and automatically add it to my Erlang binary?

Frank,

I used to write linked-in drivers for embedded systems which provided
C library APIs. Back in those days I was building for 32 & 64 bit, x86
& SPARC. I used GNU autotools to figure out the target environment and
generate macros in my Erlang header files. It all worked quite
smoothly in the end.  I'd be happy to send you an example project if
you think it'd helpful.

To your specific question I think the autoconf macro AC_CHECK_ALIGNOF
does what you want:
     https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/Generic-Compiler-Characteristics.html

--
     -Vance
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Frank Muller
Hello Vance

Fantastic!!!

It would be great if you can share some code.
I’m totally lost with this tricky problem and no one was able to help me.

Thanks in advance...

/Frank
On Fri, Dec 7, 2018 at 9:51 AM Frank Muller <[hidden email]> wrote:
> Question: how can I derive the GCC's padding, and automatically add it to my Erlang binary?

Frank,

I used to write linked-in drivers for embedded systems which provided
C library APIs. Back in those days I was building for 32 & 64 bit, x86
& SPARC. I used GNU autotools to figure out the target environment and
generate macros in my Erlang header files. It all worked quite
smoothly in the end.  I'd be happy to send you an example project if
you think it'd helpful.

To your specific question I think the autoconf macro AC_CHECK_ALIGNOF
does what you want:
     https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/Generic-Compiler-Characteristics.html

--
     -Vance

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Mikael Pettersson-5
You'll need to use C's offsetof() and sizeof() to discover (a) any
internal padding between end of a previous field and the start of a
new field, basically offsetof(struct, f2) - (offsetof(struct, f1) +
sizeof(struct.f1)) gives that internal padding, and (2) any trailing
padding, given by sizeof(struct) - (offsetof(struct, lastField) +
sizeof(struct.lastField)).  Beware that the contents of padding is
unspecified, so might not be all-bits-zero.

You'll need to derive these values via the C compiler.  In theory you
could derive them in pure Erlang given the types involved, but that'd
require not only a C type parser but also a complete description of
the machine's C ABI.  I would't go that way.
On Fri, Dec 7, 2018 at 9:49 AM Frank Muller <[hidden email]> wrote:

>
> Hello Vance
>
> Fantastic!!!
>
> It would be great if you can share some code.
> I’m totally lost with this tricky problem and no one was able to help me.
>
> Thanks in advance...
>
> /Frank
>>
>> On Fri, Dec 7, 2018 at 9:51 AM Frank Muller <[hidden email]> wrote:
>> > Question: how can I derive the GCC's padding, and automatically add it to my Erlang binary?
>>
>> Frank,
>>
>> I used to write linked-in drivers for embedded systems which provided
>> C library APIs. Back in those days I was building for 32 & 64 bit, x86
>> & SPARC. I used GNU autotools to figure out the target environment and
>> generate macros in my Erlang header files. It all worked quite
>> smoothly in the end.  I'd be happy to send you an example project if
>> you think it'd helpful.
>>
>> To your specific question I think the autoconf macro AC_CHECK_ALIGNOF
>> does what you want:
>>      https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/Generic-Compiler-Characteristics.html
>>
>> --
>>      -Vance
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Frank Muller
Thanks Mikael.
Any others thoughts ?

/Frank

You'll need to use C's offsetof() and sizeof() to discover (a) any
internal padding between end of a previous field and the start of a
new field, basically offsetof(struct, f2) - (offsetof(struct, f1) +
sizeof(struct.f1)) gives that internal padding, and (2) any trailing
padding, given by sizeof(struct) - (offsetof(struct, lastField) +
sizeof(struct.lastField)).  Beware that the contents of padding is
unspecified, so might not be all-bits-zero.

You'll need to derive these values via the C compiler.  In theory you
could derive them in pure Erlang given the types involved, but that'd
require not only a C type parser but also a complete description of
the machine's C ABI.  I would't go that way.
On Fri, Dec 7, 2018 at 9:49 AM Frank Muller <[hidden email]> wrote:
>
> Hello Vance
>
> Fantastic!!!
>
> It would be great if you can share some code.
> I’m totally lost with this tricky problem and no one was able to help me.
>
> Thanks in advance...
>
> /Frank
>>
>> On Fri, Dec 7, 2018 at 9:51 AM Frank Muller <[hidden email]> wrote:
>> > Question: how can I derive the GCC's padding, and automatically add it to my Erlang binary?
>>
>> Frank,
>>
>> I used to write linked-in drivers for embedded systems which provided
>> C library APIs. Back in those days I was building for 32 & 64 bit, x86
>> & SPARC. I used GNU autotools to figure out the target environment and
>> generate macros in my Erlang header files. It all worked quite
>> smoothly in the end.  I'd be happy to send you an example project if
>> you think it'd helpful.
>>
>> To your specific question I think the autoconf macro AC_CHECK_ALIGNOF
>> does what you want:
>>      https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/Generic-Compiler-Characteristics.html
>>
>> --
>>      -Vance
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Roger Lipscombe-2
On Fri, 7 Dec 2018 at 18:02, Frank Muller <[hidden email]> wrote:
Thanks Mikael.
Any others thoughts ?

I'm assuming that you've considered serializing and deserializing the data to an unpadded (or defined padding) format, but that you've presumably discounted it for performance reasons?

Just putting it out there in case someone's facing a similar issue, but has different constraints.

We once used XDR for this (with some kind of IDL-to-C converter, if I recall correctly). Some brief googling suggests that XDR is not dead yet.

Cheers,
Roger.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Sölvi Páll Ásgeirsson
Can you use #pragma pack on the C structs?
You can then easily deconstruct them in a binary match

Regards
Sölvi

Sent from my iPhone

On 7 Dec 2018, at 21:02, Roger Lipscombe <[hidden email]> wrote:

On Fri, 7 Dec 2018 at 18:02, Frank Muller <[hidden email]> wrote:
Thanks Mikael.
Any others thoughts ?

I'm assuming that you've considered serializing and deserializing the data to an unpadded (or defined padding) format, but that you've presumably discounted it for performance reasons?

Just putting it out there in case someone's facing a similar issue, but has different constraints.

We once used XDR for this (with some kind of IDL-to-C converter, if I recall correctly). Some brief googling suggests that XDR is not dead yet.

Cheers,
Roger.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Jesper Louis Andersen-2
There is also an attribute, packed, which can be useful here:


If you are not too concerned about the efficiency, I would just pack the struct and send it over the wire. The problem with this on a modern CPU:

* While it packs tighter in your L1D cache, and this gives a speedup.
* It requires more instructions on-CPU to get alignment back in order.
* If multiple CPU threads access the data, you will end up bouncing between the L1D caches of the cores, and this is usually bad. With alignment, this can be avoided.

My attack on the problem would be to pack first, then start measuring and see how much more I can get out of a solution as suggested by Mikael in this thread.

On Sat, Dec 8, 2018 at 12:11 AM Sölvi Páll Ásgeirsson <[hidden email]> wrote:
Can you use #pragma pack on the C structs?
You can then easily deconstruct them in a binary match

Regards
Sölvi

Sent from my iPhone

On 7 Dec 2018, at 21:02, Roger Lipscombe <[hidden email]> wrote:

On Fri, 7 Dec 2018 at 18:02, Frank Muller <[hidden email]> wrote:
Thanks Mikael.
Any others thoughts ?

I'm assuming that you've considered serializing and deserializing the data to an unpadded (or defined padding) format, but that you've presumably discounted it for performance reasons?

Just putting it out there in case someone's facing a similar issue, but has different constraints.

We once used XDR for this (with some kind of IDL-to-C converter, if I recall correctly). Some brief googling suggests that XDR is not dead yet.

Cheers,
Roger.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
J.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Frank Muller
Hello Jesper,

My binaries are an infinite stream of at least ~7500B each (include lot of floats).

Packing the associated C struct will negatively  affect the performances I’m afraid.

This is why I would like to align every field to its natural alignement if possible.

/Frank

There is also an attribute, packed, which can be useful here:


If you are not too concerned about the efficiency, I would just pack the struct and send it over the wire. The problem with this on a modern CPU:

* While it packs tighter in your L1D cache, and this gives a speedup.
* It requires more instructions on-CPU to get alignment back in order.
* If multiple CPU threads access the data, you will end up bouncing between the L1D caches of the cores, and this is usually bad. With alignment, this can be avoided.

My attack on the problem would be to pack first, then start measuring and see how much more I can get out of a solution as suggested by Mikael in this thread.

On Sat, Dec 8, 2018 at 12:11 AM Sölvi Páll Ásgeirsson <[hidden email]> wrote:
Can you use #pragma pack on the C structs?
You can then easily deconstruct them in a binary match

Regards
Sölvi

Sent from my iPhone

On 7 Dec 2018, at 21:02, Roger Lipscombe <[hidden email]> wrote:

On Fri, 7 Dec 2018 at 18:02, Frank Muller <[hidden email]> wrote:
Thanks Mikael.
Any others thoughts ?

I'm assuming that you've considered serializing and deserializing the data to an unpadded (or defined padding) format, but that you've presumably discounted it for performance reasons?

Just putting it out there in case someone's facing a similar issue, but has different constraints.

We once used XDR for this (with some kind of IDL-to-C converter, if I recall correctly). Some brief googling suggests that XDR is not dead yet.

Cheers,
Roger.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
J.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Passing data between Erlang and C port while preserving padding and alignment?

Richard O'Keefe
In reply to this post by Frank Muller
The alignment/padding in a C struct depends on the hardware, the ABI, the compiler, and the options that were given to the compiler at build time.  In the past I ran into serious problems trying to exchange binary data between C programs running on the same machine that were compiled by different compilers.  (I have three C compilers on my el-cheapo Linux box, and could easily add a fourth if I wanted, all free.  The last time I tried the fifth free compiler, it did not work, but I live in hope.)

Me, I'd use an existing C compiler to do the parsing, such as GCC-XML or CastXML, extract the struct declaration from that, and generate both C and Erlang code for pickling and unpickling.  (Or if you are fluent in SML, you could give the ckit library a go.  There are also C lexers/parsers/verifiers in OCaml.)

More likely, I'd just invent a little tabular DSL for describing the structs I wanted to communicate between C and Erlang, and generate the C declaration and other code and the Erlang code from that, to save having to learn yet another tool.


On Fri, 7 Dec 2018 at 17:21, Frank Muller <[hidden email]> wrote:
Hi guys

Let assume the following C structure (I have a bunch of them, all different):

struct foo {
 int32 i32;
 char c;
 int64 i64;
 char * p;
 float f;
 char * str[16]; 
}; 

When it gets compiled, GCC will add some padding to align data types (i.e self-alignment). More info at:

http://www.catb.org/esr/structure-packing/

On the Erlang side, it is very easy to generate a binary which matches this C structure field by field.
But it won’t work because of the padding :-/

Question: how can I derive the GCC's padding, and automatically add it to my Erlang binary?

I am open to any generic/flexible solutions even if I have to auto-generate these C structures (i.e meta-programming).

Thank you.
/Frank
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions