Small and large (>=64B) binaries? How do they differ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Small and large (>=64B) binaries? How do they differ?

Frank Muller
Hi guys

Newbie here…

Can someone point me please to the C data structure representing small and large binaries inside the VM?
Do they’ve the same C representation (or not)?
How the VM distinguishes between them internally?

Thank you.
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Small and large (>=64B) binaries? How do they differ?

Dmytro Lytovchenko
Look for binary, procbin, refbin (probably), and sub binary. Use this as keywords

On 19 Sep 2017 21:50, "Frank Muller" <[hidden email]> wrote:
Hi guys

Newbie here…

Can someone point me please to the C data structure representing small and large binaries inside the VM?
Do they’ve the same C representation (or not)?
How the VM distinguishes between them internally?

Thank you.
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Small and large (>=64B) binaries? How do they differ?

Frank Muller
Hi Dmytro,

Thanks for the hint. I found a lot of references to these keywords ... bit lost.

How about the rest of the questions?

/Frank

Tue. Sept. 19 2017 à 22:13, Dmytro Lytovchenko <[hidden email]> wrote :
Look for binary, procbin, refbin (probably), and sub binary. Use this as keywords

On 19 Sep 2017 21:50, "Frank Muller" <[hidden email]> wrote:
Hi guys

Newbie here…

Can someone point me please to the C data structure representing small and large binaries inside the VM?
Do they’ve the same C representation (or not)?
How the VM distinguishes between them internally?

Thank you.
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Small and large (>=64B) binaries? How do they differ?

Dmytro Lytovchenko
From what I remember (some people who recently have worked with binaries might know much more)

Binaries in binary heap (large bins) have own structure, i think the master storage structure is called Binary. This is confusing even to me so have a spoon of salt with this advice :)


Binaries stored outside processes (proc-bins ProcBin), smaller than 64b (proc-bins, structure ErlHeapBin i think) and sub-bins (ErlSubBin) have own tag bit combinations on the process heap. So the GC and VM always know what they have found. They might share part of C structure pointing to the external binary, you can observe it as a macro included in the respective structures, think is called ErtsMagicBinary. Macros is_not_binarybinary_size, binary_bytes and ERTS_GET_REAL_BIN might become useful.

Also ref-bins all contain 'next' pointer, as a part of long linked list which begins somewhere in process header (one such list belongs to every process) and during GC this list is traversed to find out which ref-bins are about to die and refcount is decremented.

2017-09-20 6:08 GMT+02:00 Frank Muller <[hidden email]>:
Hi Dmytro,

Thanks for the hint. I found a lot of references to these keywords ... bit lost.

How about the rest of the questions?

/Frank

Tue. Sept. 19 2017 à 22:13, Dmytro Lytovchenko <[hidden email]> wrote :
Look for binary, procbin, refbin (probably), and sub binary. Use this as keywords

On 19 Sep 2017 21:50, "Frank Muller" <[hidden email]> wrote:
Hi guys

Newbie here…

Can someone point me please to the C data structure representing small and large binaries inside the VM?
Do they’ve the same C representation (or not)?
How the VM distinguishes between them internally?

Thank you.
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Small and large (>=64B) binaries? How do they differ?

Frank Muller
Thanks again Dmytro for your time.

Let assume I’m inside my NIF and would like to increment the refcount of a binary (hope this is the right call: enif_keep_resource).

Can i increment refcount for let say 1 million binaries (or even more)?

What are the consequences of doing that?
I assume I'll release them at the end.

/Frank

From what I remember (some people who recently have worked with binaries might know much more)

Binaries in binary heap (large bins) have own structure, i think the master storage structure is called Binary. This is confusing even to me so have a spoon of salt with this advice :)


Binaries stored outside processes (proc-bins ProcBin), smaller than 64b (proc-bins, structure ErlHeapBin i think) and sub-bins (ErlSubBin) have own tag bit combinations on the process heap. So the GC and VM always know what they have found. They might share part of C structure pointing to the external binary, you can observe it as a macro included in the respective structures, think is called ErtsMagicBinary. Macros is_not_binarybinary_size, binary_bytes and ERTS_GET_REAL_BIN might become useful.

Also ref-bins all contain 'next' pointer, as a part of long linked list which begins somewhere in process header (one such list belongs to every process) and during GC this list is traversed to find out which ref-bins are about to die and refcount is decremented.

2017-09-20 6:08 GMT+02:00 Frank Muller <[hidden email]>:
Hi Dmytro,

Thanks for the hint. I found a lot of references to these keywords ... bit lost.

How about the rest of the questions?

/Frank

Tue. Sept. 19 2017 à 22:13, Dmytro Lytovchenko <[hidden email]> wrote :
Look for binary, procbin, refbin (probably), and sub binary. Use this as keywords

On 19 Sep 2017 21:50, "Frank Muller" <[hidden email]> wrote:
Hi guys

Newbie here…

Can someone point me please to the C data structure representing small and large binaries inside the VM?
Do they’ve the same C representation (or not)?
How the VM distinguishes between them internally?

Thank you.
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Small and large (>=64B) binaries? How do they differ?

Sverker Eriksson-4


On 09/20/2017 06:39 PM, Frank Muller wrote:
> Thanks again Dmytro for your time.
>
> Let assume I’m inside my NIF and would like to increment the refcount of a
> binary (hope this is the right call: enif_keep_resource).
No, enif_keep_resource does not work on binaries.
"Resources" are a concept of their own and binaries are not resources.

Unfortunately there is no enif_keep_binary. The only use for such a function
is if you want to keep binaries alive even no processes are referring to
them.
And you would need some native data structure with pointers to those
binaries,
in order to find them without a binary term.

/Sverker


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Small and large (>=64B) binaries? How do they differ?

Frank Muller
Sverker,

Okay thanks for the clarifications.

How about the opposite: an Erlang process which holds pointers to a large number of RefCounted 
Bins ?

Is there any performance penalty (ex. affected by GC) of doing that?

/Frank





On 09/20/2017 06:39 PM, Frank Muller wrote:
> Thanks again Dmytro for your time.
>
> Let assume I’m inside my NIF and would like to increment the refcount of a
> binary (hope this is the right call: enif_keep_resource).
No, enif_keep_resource does not work on binaries.
"Resources" are a concept of their own and binaries are not resources.

Unfortunately there is no enif_keep_binary. The only use for such a function
is if you want to keep binaries alive even no processes are referring to
them.
And you would need some native data structure with pointers to those
binaries,
in order to find them without a binary term.

/Sverker



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions