Multithreaded Drivers

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Multithreaded Drivers

Sean Hinde-2
All,

I'm digging through the existing drivers trying to figure out how they
work.. Particularly with regard to multithreaded drivers.

So far I have figured out from studying efile_drv that:

The start function should return a pointer to the struct which holds all the
state for the thread. This is so that each call to open_port will generate
it's own state which is passed to whichever thread happens to be used to
process the call. This pointer is cast to a long to keep the type system
happy :)

Q. Can I use the new erl_driver.h functions with multithreaded drivers (I
notice that not all the functions in driver.h are included in the new one,
particularly driver_async)?

Q. I presume if so I can just cast the pointer to my state into a
ErlDrvData?

The start function in efile_drv uses the function sys_alloc_from(200,
sizeof(file_descriptor_state_struct)). I can't figure out the point of the
200, or how to decide what it should be in my case. As far as I can make out
it only has an effect in an instrumented system (from looking at the def in
sys.h).

Q. What is the approved way to assign and clear memory in this case and in
drivers in general?

Q. Under what circumstances is the free function passed to driver_async
called (I couldn't find any from a quick look)?

Q. What is the key parameter intended for and do I ever need to worry about
it?

I've got the hang of the idea that the main function driven by
port_command/2 should just schedule the work to be done in a thread sometime
later with the driver_async function, and that when the thread has done it's
work for this time it puts itself into the queue to call the async_ready
callback which sends the result back to erlang. This looks fine..

Q. What happens if I send another port_command/2 to the port before the
result of the possibly time consuming last operation has returned?

Q. I wonder if I can use this to call the driver_cancel function sometime
later to abort the operation?

Q. What is the best way to timeout the operation which is being carried out
in a thread so I can return the resource to useful state (in case it is in
an endless loop, or the far end of some link has died leaving the thread
waiting forever..)?

My head is gradually turning to jelly here.. All comments on my
understanding (or answer to my questions) are welcome!

Cheers,
Sean





NOTICE AND DISCLAIMER:
This email (including attachments) is confidential.  If you have received
this email in error please notify the sender immediately and delete this
email from your system without copying or disseminating it or placing any
reliance upon its contents.  We cannot accept liability for any breaches of
confidence arising through use of email.  Any opinions expressed in this
email (including attachments) are those of the author and do not necessarily
reflect our opinions.  We will not accept responsibility for any commitments
made by our employees outside the scope of our business.  We do not warrant
the accuracy or completeness of such information.



Reply | Threaded
Open this post in threaded view
|

Multithreaded Drivers

Raimo Niskanen-3
I managed to send this message to Sean only, missed the list, and
finally now fixes that mistake...

/ Raimo Niskanen, Erlang/OTP, Ericsson UAB.

------------------------------------------------------------------------------------------




Sorry for the delay, Sean, I just came back from my vacation. Answers
inserted below...



Sean Hinde wrote:

>
> All,
>
> I'm digging through the existing drivers trying to figure out how they
> work.. Particularly with regard to multithreaded drivers.
>
> So far I have figured out from studying efile_drv that:
>
> The start function should return a pointer to the struct which holds all the
> state for the thread. This is so that each call to open_port will generate
> it's own state which is passed to whichever thread happens to be used to
> process the call. This pointer is cast to a long to keep the type system
> happy :)
>

The start function and all driver callback functions are called in the
same thread context, i.e the emulator main thread. The emulator is using
only one thread, the main thread, and emulates all erlang processes in
that thread. This is to avoid platform dependant and time consuming
thread locking between erlang processes.

The multithread interface that efile_drv uses is the only way to execute
in other threads from within the emulator (expect for some obscure
support threads), and the only code that executes in other threads are
the callback (worker) functions given to driver_async(). Therefore all
thread specific data must be accessible through the data pointer also
given to driver_async().

> Q. Can I use the new erl_driver.h functions with multithreaded drivers (I
> notice that not all the functions in driver.h are included in the new one,
> particularly driver_async)?
>

We have tried for a while to get rid of driver.h, and for R8 it finally
seems to happen; erl_driver.h will then (probably) contain all functions
from driver.h.

> Q. I presume if so I can just cast the pointer to my state into a
> ErlDrvData?
>

Yes.

> The start function in efile_drv uses the function sys_alloc_from(200,
> sizeof(file_descriptor_state_struct)). I can't figure out the point of the
> 200, or how to decide what it should be in my case. As far as I can make out
> it only has an effect in an instrumented system (from looking at the def in
> sys.h).
>

Correct, it is used for an instrumented system. You can look in the
source code for instrument.erl in application 'tools' to see the values
that are defined today.

> Q. What is the approved way to assign and clear memory in this case and in
> drivers in general?
>

To allocate and free memory please use driver_alloc(), driver_realloc()
and driver_free() in erl_driver.h. Also, driver_alloc_binary(),
driver_realloc_binary() and driver_free_binary() may be useful.

> Q. Under what circumstances is the free function passed to driver_async
> called (I couldn't find any from a quick look)?
>

When an async request is cancelled with driver_async_cancel(), and the
request has not started to run yet. Either the free function or the
driver callback async_ready is called.

When a port is closing while having unfinished async requests. The
driver callbacks cannot be called since the port is regarded as closed.

> Q. What is the key parameter intended for and do I ever need to worry about
> it?
>

The key parameter is used to select thread. If it is NULL, as for
efile_drv, the requests are round robin scheduled on the threads in the
async thread pool. Otherwise it must be a pointer to an integer that is
used to select thread through a simple hash function (modulo number of
threads).

E.g efile_drv may have to worry about this when (if) it becomes possible
to access a file from different erlang processes. Requests to one
specific file must then be handled by the same thread, or else a read
from one erlang process could run ahead of a write from another erlang
process by using a different thread. Therefore efile_drv must use maybe
the file descriptor or perhaps the port number as key, it must be an
integer unique for the file.

> I've got the hang of the idea that the main function driven by
> port_command/2 should just schedule the work to be done in a thread sometime
> later with the driver_async function, and that when the thread has done it's
> work for this time it puts itself into the queue to call the async_ready
> callback which sends the result back to erlang. This looks fine..
>
> Q. What happens if I send another port_command/2 to the port before the
> result of the possibly time consuming last operation has returned?
>

This is entirely up to your interface between the erlang process and the
driver. You decide if the erlang process should wait for a message from
the port after each port_command/2 or not.

Messages are sent asynchronously from the erlang process to the port
with port_command/2, and the other way (also asynchronously) with
driver_output*(). Note that set_busy_port() can be used to suspend
future port_command/2. Synchronous calls from erlang processes to the
port are made with port_control/3.


> Q. I wonder if I can use this to call the driver_cancel function sometime
> later to abort the operation?
>

Yes you can, until the request has started to run on a thread in the
thread pool. The function driver_async_cancel() returns 1 if the request
was cancelled, and 0 if the request could not be found in the queues
(might be running, or has alread run, or invalid async_id).

> Q. What is the best way to timeout the operation which is being carried out
> in a thread so I can return the resource to useful state (in case it is in
> an endless loop, or the far end of some link has died leaving the thread
> waiting forever..)?
>

While the thread is running it cannot be cancelled. This also blocks all
async requests that may be scheduled on the same thread later. All async
requests must be reasonably short, or else they will slow down e.g
efile_drv. It is best if the worker function itself can limit its
execution time.

It might be possible to use the erts_mutex*() and erst_cond*() functions
in some way, but the water is getting deep here...

>
>
>

Hope this helps you forward.

/ Raimo Niskanen, Erlang/OTP, Ericsson UAB.


Reply | Threaded
Open this post in threaded view
|

Multithreaded Drivers

Raimo Niskanen-3
And here is some more info on the same subject that I think is of public
interest...

/ Raimo Niskanen, Erlang/OTP, Ericsson UAB.

------------------------------------------------------------------------------------------

Sean Hinde wrote:
>
> Raimo,
>
> Another question.. Has anyone ever experimented with avoiding the copying of
> data between the erlang thread and the worker thread? Perhaps by using a
> mutex type arrangement, or even by simply ensuring that the erlang side
> doesn't send any more data to overwrite the buffer until it is given
> permission to do so.
>

Yes, I have. I am rewriting efile_drv to store file data in the port
queue. Then data is accessed both from the main thread and from the
worker threads. Access is protected by a mutex in the port data
structure. So far it seems to work fine.

Beware of the danger the port being closed at any time, meaning that the
port data structure and hence the mutex can be freed and perhaps
overwritten at any time, so a worker thread might get the mutex
corrupted at any time.

I happen to avoid this since the port will not be closed while there is
data in the port queue. A workaround for other cases might be to store a
byte in the port queue while there are unfinished async requests.

/ Raimo