Synchronous communication with prim_file port

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Synchronous communication with prim_file port

Salikhov Dinislam
Hello.

I have an erlang process that receives logs from other processes and
writes them into the file.
The logs are written to the file via file:write() which in turn boils
down to erlang:port_command/2 call followed by getting response from
driver via "receive {Port, {data, Data}} -> ... end" (see
./erts/preloaded/src/prim_file.erl, write/2 and follow the calls).
And here comes a trouble: if log rate is too high then while driver does
its job, a lot of messages are waiting in the process's message queue
and the receive has to walk through all of them to find the driver's answer.

Is there any rationale for write operation being done this way?
I'd like to propose an optimization here: if we know in advance that
standard drv_get_response/1 will be used to get driver's response, then
we can use synchronous erlang:port_control/3 instead to avoid the whole
message queue traverse.

Salikhov Dinislam
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Synchronous communication with prim_file port

Max Lapshin-2
This is a common problem.

Perhaps you can buffer these log records in another process that will accumulate it in memory.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Synchronous communication with prim_file port

Lukas Larsson-8
In reply to this post by Salikhov Dinislam
Hello,

On Mon, Oct 23, 2017 at 2:35 PM, Salikhov Dinislam <[hidden email]> wrote:
Is there any rationale for write operation being done this way?
I'd like to propose an optimization here: if we know in advance that standard drv_get_response/1 will be used to get driver's response, then we can use synchronous erlang:port_control/3 instead to avoid the whole message queue traverse.

Using erlang:port_control/3 will not help as the file I/O job is scheduled on the async thread poll which means that it has to ack by sending a message. 

We are in the process of re-writing the efile driver for OTP 21 using a nif, this re-write will solve this problem as no message passing is needed when ack:ing writes done on dirty schedulers.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Synchronous communication with prim_file port

Max Lapshin-2
But it will not save from overflowing message queue.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions