Quantcast

Processing huge binaries in NIFs

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Processing huge binaries in NIFs

Konstantin Sorokin
Hello!

I want to process huge binaries in NIFs and in order not to screw up VM's scheduler I want to move time consuming jobs into a separate thread. Do I really need to make an internal copy of the binary before passing it to a thread for processing or it'll be OK just to set pointers to raw data in a task structure and then submit that task to the worker thread ?

--
Konstantin Sorokin

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Processing huge binaries in NIFs

Sverker Eriksson
Konstantin Sorokin wrote:
> Hello!
>
> I want to process huge binaries in NIFs and in order not to screw up VM's
> scheduler I want to move time consuming jobs into a separate thread. Do I
> really need to make an internal copy of the binary before passing it to a
> thread for processing or it'll be OK just to set pointers to raw data in a
> task structure and then submit that task to the worker thread ?
>  

You don't have to copy the binary.

Solution 1.

Use enif_alloc_env() and enif_make_copy() to create a process
independent binary term that you then can pass to your worker thread.
enif_make_copy() will not copy the data if the binary is big, but rather
increase a reference counter. Then use enif_free_env() to release your
logical copy of the binary.

Solution 2.

Implement enif_keep_binary() which is on the ToDo list for the NIF API.
enif_keep_binary() would just increase the reference counter of the
ErlNifBinary without the hassle of having to create environment and
Erlang term.

/Sverker, Erlang/OTP


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...