erl_nif: persist binary across calls

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

erl_nif: persist binary across calls

Harris, Robert
Hi,

I am writing a NIF library in which I would like to create persistent binary terms
in my load() callback.  The purpose of these terms is to act as ready-made keys
for frequent calls to enif_get_map_value() in successive calls to other NIFs in the
same library.

My test case is

> #include <string.h>
> #include <erl_nif.h>
>
> ERL_NIF_TERM persistent;
>
> static ERL_NIF_TERM
> create_binary(ErlNifEnv *env)
> {
>         ErlNifBinary binary;
>         char *data = "abcdefghijklmnopqrstuvwxyz";
>         size_t size = strlen(data);
>
>         (void) enif_alloc_binary(size, &binary);
>         memcpy(binary.data, data, size);
>         binary.size = size;
>         return (enif_make_binary(env, &binary));
> }
>
> int
> load(ErlNifEnv *env, void **priv_data, ERL_NIF_TERM load_info)
> {
>         persistent = create_binary(env);
>         return (0);
> }
>
> static ERL_NIF_TERM
> world_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
> {
>          return (persistent);
> }
>
> static ERL_NIF_TERM
> hello_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
> {
>          return (create_binary(env));
> }
>
> static ErlNifFunc nif_funcs[] = {
>         {"hello", 0, hello_nif},
>         {"world", 0, world_nif},
> };
>
> ERL_NIF_INIT(hello, nif_funcs, &load, NULL, NULL, NULL)
An example of the behaviour I see is

> Erlang/OTP 20 [erts-9.3.3.3] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:10] [hipe] [kernel-poll:false]
>
> Eshell V9.3.3.3  (abort with ^G)
> 1>
> 1> hello:hello().
> <<"abcdefghijklmnopqrstuvwxyz">>
> 2> hello:world().
> eheap_alloc: Cannot allocate 2305843009213695256 bytes of memory (of type "heap_frag").
>
> Crash dump is being written to: erl_crash.dump...done
> make: *** [shell] Error 1
I'm guessing that what I'm trying is wrong but I'd appreciate a pointer as to
why.  I'm motivated in part by comments like the following extract from
enif_make_new_binary():

> The drawbacks are that the binary cannot be kept between NIF calls and it cannot
> be reallocated.

I inferred from this that a binary generated by enif_make_binary() *can* be kept
between NIF calls.  I'm a newcomer to Erlang, if that's not obvious.

Regards,

Robert
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: erl_nif: persist binary across calls

Sverker Eriksson-5
Hi Robert,

The 'env' you get in 'load' is temporary. Terms created in it will disappear
when 'load' returns.
Use enif_alloc_env() to create terms that will survive between NIF calls:

ErlNifEnv* persistent_env;

int load(ErlNifEnv *env, void **priv_data, ERL_NIF_TERM load_info)
{
    persistent_env = enif_alloc_env();
    persistent = create_binary(persistent_env);
    return 0;
}


Use enif_make_copy() to copy terms between ErlNifEnv's:

static ERL_NIF_TERM world_nif(ErlNifEnv* env, ...)
{
    return enif_make_copy(persistent);
}

If the persistent binary is large (> 64 bytes) then only a small term header
will be copied pointing to the binary data.


/Sverker

On fre, 2019-02-08 at 14:04 +0000, Harris, Robert wrote:

> Hi,
>
> I am writing a NIF library in which I would like to create persistent binary
> terms
> in my load() callback.  The purpose of these terms is to act as ready-made
> keys
> for frequent calls to enif_get_map_value() in successive calls to other NIFs
> in the
> same library.
>
> My test case is
>
> >
> > #include <string.h>
> > #include <erl_nif.h>
> >
> > ERL_NIF_TERM persistent;
> >
> > static ERL_NIF_TERM
> > create_binary(ErlNifEnv *env)
> > {
> >         ErlNifBinary binary;
> >         char *data = "abcdefghijklmnopqrstuvwxyz";
> >         size_t size = strlen(data);
> >
> >         (void) enif_alloc_binary(size, &binary);
> >         memcpy(binary.data, data, size);
> >         binary.size = size;
> >         return (enif_make_binary(env, &binary));
> > }
> >
> > int
> > load(ErlNifEnv *env, void **priv_data, ERL_NIF_TERM load_info)
> > {
> >         persistent = create_binary(env);
> >         return (0);
> > }
> >
> > static ERL_NIF_TERM
> > world_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
> > {
> >          return (persistent);
> > }
> >
> > static ERL_NIF_TERM
> > hello_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
> > {
> >          return (create_binary(env));
> > }
> >
> > static ErlNifFunc nif_funcs[] = {
> >         {"hello", 0, hello_nif},
> >         {"world", 0, world_nif},
> > };
> >
> > ERL_NIF_INIT(hello, nif_funcs, &load, NULL, NULL, NULL)
> An example of the behaviour I see is
>
> >
> > Erlang/OTP 20 [erts-9.3.3.3] [source] [64-bit] [smp:12:12] [ds:12:12:10]
> > [async-threads:10] [hipe] [kernel-poll:false]
> >
> > Eshell V9.3.3.3  (abort with ^G)
> > 1>
> > 1> hello:hello().
> > <<"abcdefghijklmnopqrstuvwxyz">>
> > 2> hello:world().
> > eheap_alloc: Cannot allocate 2305843009213695256 bytes of memory (of type
> > "heap_frag").
> >
> > Crash dump is being written to: erl_crash.dump...done
> > make: *** [shell] Error 1
> I'm guessing that what I'm trying is wrong but I'd appreciate a pointer as to
> why.  I'm motivated in part by comments like the following extract from
> enif_make_new_binary():
>
> >
> > The drawbacks are that the binary cannot be kept between NIF calls and it
> > cannot
> > be reallocated.
> I inferred from this that a binary generated by enif_make_binary() *can* be
> kept
> between NIF calls.  I'm a newcomer to Erlang, if that's not obvious.
>
> Regards,
>
> Robert
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: erl_nif: persist binary across calls

Harris, Robert
Hi Sverker,

That worked perfectly.

Many thanks,

Robert


> On 8 Feb 2019, at 15:49, Sverker Eriksson <[hidden email]> wrote:
>
> Hi Robert,
>
> The 'env' you get in 'load' is temporary. Terms created in it will disappear
> when 'load' returns.
> Use enif_alloc_env() to create terms that will survive between NIF calls:
>
> ErlNifEnv* persistent_env;
>
> int load(ErlNifEnv *env, void **priv_data, ERL_NIF_TERM load_info)
> {
>     persistent_env = enif_alloc_env();
>     persistent = create_binary(persistent_env);
>     return 0;
> }
>
>
> Use enif_make_copy() to copy terms between ErlNifEnv's:
>
> static ERL_NIF_TERM world_nif(ErlNifEnv* env, ...)
> {
>     return enif_make_copy(persistent);
> }
>
> If the persistent binary is large (> 64 bytes) then only a small term header
> will be copied pointing to the binary data.
>
>
> /Sverker
>
> On fre, 2019-02-08 at 14:04 +0000, Harris, Robert wrote:
>> Hi,
>>
>> I am writing a NIF library in which I would like to create persistent binary
>> terms
>> in my load() callback.  The purpose of these terms is to act as ready-made
>> keys
>> for frequent calls to enif_get_map_value() in successive calls to other NIFs
>> in the
>> same library.
>>
>> My test case is
>>
>>>
>>> #include <string.h>
>>> #include <erl_nif.h>
>>>
>>> ERL_NIF_TERM persistent;
>>>
>>> static ERL_NIF_TERM
>>> create_binary(ErlNifEnv *env)
>>> {
>>>         ErlNifBinary binary;
>>>         char *data = "abcdefghijklmnopqrstuvwxyz";
>>>         size_t size = strlen(data);
>>>
>>>         (void) enif_alloc_binary(size, &binary);
>>>         memcpy(binary.data, data, size);
>>>         binary.size = size;
>>>         return (enif_make_binary(env, &binary));
>>> }
>>>
>>> int
>>> load(ErlNifEnv *env, void **priv_data, ERL_NIF_TERM load_info)
>>> {
>>>         persistent = create_binary(env);
>>>         return (0);
>>> }
>>>
>>> static ERL_NIF_TERM
>>> world_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
>>> {
>>>          return (persistent);
>>> }
>>>
>>> static ERL_NIF_TERM
>>> hello_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
>>> {
>>>          return (create_binary(env));
>>> }
>>>
>>> static ErlNifFunc nif_funcs[] = {
>>>         {"hello", 0, hello_nif},
>>>         {"world", 0, world_nif},
>>> };
>>>
>>> ERL_NIF_INIT(hello, nif_funcs, &load, NULL, NULL, NULL)
>> An example of the behaviour I see is
>>
>>>
>>> Erlang/OTP 20 [erts-9.3.3.3] [source] [64-bit] [smp:12:12] [ds:12:12:10]
>>> [async-threads:10] [hipe] [kernel-poll:false]
>>>
>>> Eshell V9.3.3.3  (abort with ^G)
>>> 1>
>>> 1> hello:hello().
>>> <<"abcdefghijklmnopqrstuvwxyz">>
>>> 2> hello:world().
>>> eheap_alloc: Cannot allocate 2305843009213695256 bytes of memory (of type
>>> "heap_frag").
>>>
>>> Crash dump is being written to: erl_crash.dump...done
>>> make: *** [shell] Error 1
>> I'm guessing that what I'm trying is wrong but I'd appreciate a pointer as to
>> why.  I'm motivated in part by comments like the following extract from
>> enif_make_new_binary():
>>
>>>
>>> The drawbacks are that the binary cannot be kept between NIF calls and it
>>> cannot
>>> be reallocated.
>> I inferred from this that a binary generated by enif_make_binary() *can* be
>> kept
>> between NIF calls.  I'm a newcomer to Erlang, if that's not obvious.
>>
>> Regards,
>>
>> Robert
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__erlang.org_mailman_listinfo_erlang-2Dquestions&d=DwIGaQ&c=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY&r=rgvWXjU_2tA1ZUQSslrg0TYOEP4F7UfuySlE3gZm1mQ&m=KpVT9ckjVGBzTRcLi8KyIRtHYE40kNnBV6KUJC9f2cc&s=FGyBxvciQtMGXgbPAmz4GbykgMAb2i_uqS1EuBTNLA8&e=
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> https://urldefense.proofpoint.com/v2/url?u=http-3A__erlang.org_mailman_listinfo_erlang-2Dquestions&d=DwIGaQ&c=L_h2OePR2UWWefmqrezxOsP9Uqw55rRfX5bRtw9S4KY&r=rgvWXjU_2tA1ZUQSslrg0TYOEP4F7UfuySlE3gZm1mQ&m=KpVT9ckjVGBzTRcLi8KyIRtHYE40kNnBV6KUJC9f2cc&s=FGyBxvciQtMGXgbPAmz4GbykgMAb2i_uqS1EuBTNLA8&e=

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

smime.p7s (4K) Download Attachment