The "ei" API and passing *shared* binaries / parts

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

The "ei" API and passing *shared* binaries / parts

Rick van Rein
Hello,

I have looked at the ei module, both documentation and code.  It looks
quite efficient, but I am missing something, and I wonder if it is
somewhere else.

I started a discussion in the Apache Qpid user forum about integration
their AMQP 1.0 API with Erlang; there does not seem to be a way of
getting to AMQP 1.0 from Erlang, and given that both are designed for
rock-solid software construction that would be a useful marriage.  And
the Go port has a variant API named "electron" that is aimed at
"synchronous" communication under the assumption of many small processes.

Having to pass a (very) large message into Erlang, and possibly out at
the other end, requires cloning the entire binary over ei, AFAIK.
Although Erlang is rumoured to internally use malloc() for binaries,
with refcounting and parts pointing there (no doubt charging the
refcount) this functionality (shared memory allocation, sharing parts,
unsharing) does not seem to be available externally.  Nor did I find a
way of exchanging shared memory and parts over ei.  Did I miss it?  It
would improve efficiency in applications where Erlang is on the sending
or receiving end of AMQP, as well as an intermediate in the pathway.

There is one more use of such an ei facility -- and that is where an
existing parser in C is asked to decipher a block of text (like SIP,
XML, ...) or binary (AMQP, DNS, OpenPGP, ...) and return parts thereof
as components to work on.  It's a pitty to have to copy in both
directions, rather than using an existing malloc() and simply pointing
into it.

Are such facilities available but have I missed them?

Cheers,
 -Rick

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: The "ei" API and passing *shared* binaries / parts

Vance Shipley
You want to use a linked in driver:
http://erlang.org/doc/tutorial/c_portdriver.html

On Thu, Jun 22, 2017 at 12:18 PM, Rick van Rein <[hidden email]> wrote:

> Hello,
>
> I have looked at the ei module, both documentation and code.  It looks
> quite efficient, but I am missing something, and I wonder if it is
> somewhere else.
>
> I started a discussion in the Apache Qpid user forum about integration
> their AMQP 1.0 API with Erlang; there does not seem to be a way of
> getting to AMQP 1.0 from Erlang, and given that both are designed for
> rock-solid software construction that would be a useful marriage.  And
> the Go port has a variant API named "electron" that is aimed at
> "synchronous" communication under the assumption of many small processes.
>
> Having to pass a (very) large message into Erlang, and possibly out at
> the other end, requires cloning the entire binary over ei, AFAIK.
> Although Erlang is rumoured to internally use malloc() for binaries,
> with refcounting and parts pointing there (no doubt charging the
> refcount) this functionality (shared memory allocation, sharing parts,
> unsharing) does not seem to be available externally.  Nor did I find a
> way of exchanging shared memory and parts over ei.  Did I miss it?  It
> would improve efficiency in applications where Erlang is on the sending
> or receiving end of AMQP, as well as an intermediate in the pathway.
>
> There is one more use of such an ei facility -- and that is where an
> existing parser in C is asked to decipher a block of text (like SIP,
> XML, ...) or binary (AMQP, DNS, OpenPGP, ...) and return parts thereof
> as components to work on.  It's a pitty to have to copy in both
> directions, rather than using an existing malloc() and simply pointing
> into it.
>
> Are such facilities available but have I missed them?
>
> Cheers,
>  -Rick
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions



--
     -Vance
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: The "ei" API and passing *shared* binaries / parts

Karl Nilsson-2
In reply to this post by Rick van Rein
Hi,

I don't know anything about `ei` but we recently wrote an AMQP 1.0 client in erlang [1]. It isn't 100% complete and is unlikely to provide the kind of performance you may be expecting given you are looking at the internals of the runtime itself but it works and it covers the subset of functionality required for RabbitMQ projects. We have done basic testing against QPid.


Cheers
Karl

On Thu, 22 Jun 2017 at 08:33 Rick van Rein <[hidden email]> wrote:
Hello,

I have looked at the ei module, both documentation and code.  It looks
quite efficient, but I am missing something, and I wonder if it is
somewhere else.

I started a discussion in the Apache Qpid user forum about integration
their AMQP 1.0 API with Erlang; there does not seem to be a way of
getting to AMQP 1.0 from Erlang, and given that both are designed for
rock-solid software construction that would be a useful marriage.  And
the Go port has a variant API named "electron" that is aimed at
"synchronous" communication under the assumption of many small processes.

Having to pass a (very) large message into Erlang, and possibly out at
the other end, requires cloning the entire binary over ei, AFAIK.
Although Erlang is rumoured to internally use malloc() for binaries,
with refcounting and parts pointing there (no doubt charging the
refcount) this functionality (shared memory allocation, sharing parts,
unsharing) does not seem to be available externally.  Nor did I find a
way of exchanging shared memory and parts over ei.  Did I miss it?  It
would improve efficiency in applications where Erlang is on the sending
or receiving end of AMQP, as well as an intermediate in the pathway.

There is one more use of such an ei facility -- and that is where an
existing parser in C is asked to decipher a block of text (like SIP,
XML, ...) or binary (AMQP, DNS, OpenPGP, ...) and return parts thereof
as components to work on.  It's a pitty to have to copy in both
directions, rather than using an existing malloc() and simply pointing
into it.

Are such facilities available but have I missed them?

Cheers,
 -Rick

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: The "ei" API and passing *shared* binaries / parts

Rick van Rein
Hi Karl,

Yes, thanks for that.  I was also pointed to this work in the Qpid list
discussion, I'm really happy that this work has started.

My comment in this thread and here was indeed commenting on the
efficiency of the ei interface, which is orthogonal to the code wish
answered by this client -- which I'm keen on giving a spin sometime soon.

Thanks,
 -Rick

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: The "ei" API and passing *shared* binaries / parts

Rick van Rein
In reply to this post by Vance Shipley
Ah,

Vance Shipley wrote:
> You want to use a linked in driver:
> http://erlang.org/doc/tutorial/c_portdriver.html

Yes, that helps!  (Didn't see your response before somehow.)

Plus, following up on my own question, what seems to be needed to share
blobs would apparently be

 0. Have a blob uint8_t ptr [siz]
 1. Create an ETERM for it with erl_mk_binary (ptr, siz)
 2. Send with ei_encode_term
 3. Update refctr / free if needed (docs are not clear)

Sounds like a plan I to try and test.

Thanks,
-Rick


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: The "ei" API and passing *shared* binaries / parts

Vance Shipley
On Fri, Jun 30, 2017 at 12:51 PM, Rick van Rein <[hidden email]> wrote:
> what seems to be needed to share blobs would apparently be
>
>  0. Have a blob uint8_t ptr [siz]
>  1. Create an ETERM for it with erl_mk_binary (ptr, siz)
>  2. Send with ei_encode_term
>  3. Update refctr / free if needed (docs are not clear)

The most efficient and flexible way to integrate C libraries with
Erlang is to write a linked in driver which allows sharing processing
between C and Erlang, passing the data back and forth as required.
I've written a bunch of drivers over the years. If you need help
figuring any of this out just ask but the following should get you on
the right track.

The erl_driver C library API is described here:
   http://erlang.org/doc/man/erl_driver.html
   http://erlang.org/doc/man/driver_entry.html

A driver creates a binary erlang term with driver_alloc_binary():
   http://erlang.org/doc/man/erl_driver.html#driver_alloc_binary

A driver may send a binary to an erlang port with driver_output_binary():
   http://erlang.org/doc/man/erl_driver.html#driver_output_binary

A driver should use erl_drv_output_term() to send erlang terms to the
port owner:
   http://erlang.org/doc/man/erl_driver.html#erl_drv_output_term

An erlang process may use port_command/2 to send iodata() which a
driver receives in it's outputv() calback:
   http://erlang.org/doc/man/erlang.html#port_command-2
   http://erlang.org/doc/man/driver_entry.html#outputv

The data received in outputv() is an ErlIOVec structure:
   http://erlang.org/doc/man/erl_driver.html#ErlIOVec

An erlang process may use port_call/3 to send arbitrary terms which a
driver receives in it's call() callback:
   http://erlang.org/doc/man/erlang.html#port_call-3
   http://erlang.org/doc/man/driver_entry.html#call

The data received in call() is Erlang external term format which can
be decoded with a series of calls to ei_decode_*():
   http://erlang.org/doc/man/ei.html

If you only need to call C library functions from Erlang and get an
immediate result you may write a NIF instead of a full linked in
driver:
   http://erlang.org/doc/tutorial/nif.html
   http://erlang.org/doc/man/erl_nif.html




--
     -Vance
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...