Flush "stdout" buffer

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Flush "stdout" buffer

Frank Muller
Hi guys,

I'd like to know if there's an equivalent function to C fflush(stdout) ?

I've a small Erlang program which prints data to stdout. After every io:format call, I'd like to ensure output is flushed to screen.

Thanks in adavance. 
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Frank Muller
Can someone help on this please?

Thanks
/Frank

Le mer. 30 août 2017 at 07:21, Frank Muller <[hidden email]> wrote :
Hi Prakash,

Thanks, but I doubt it's that easy.

In C, you've to call fflush or set the stdout to unbuffer mode. 

/Frank

Le mer. 30 août 2017 à 06:51, Prakash Parmar <[hidden email]> wrote :

Hi Frank,

 

As per my experience, Like in C, keeping newline "~n" in io:format will flush it to screen.

 

/Prakash

 

-----Original Message-----
From: "Frank Muller" <[hidden email]>
Sent: Wednesday, August 30, 2017 9:58am
To: "Erlang-Questions Questions" <[hidden email]>
Subject: [erlang-questions] Flush "stdout" buffer

Hi guys,
I'd like to know if there's an equivalent function to C fflush(stdout) ?
I've a small Erlang program which prints data to stdout. After every io:format call, I'd like to ensure output is flushed to screen.
Thanks in adavance. 
/Frank



******* DISCLAIMER: This email and any files transmitted with it are privileged and confidential information and intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Torry Harris Business Solutions has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. The recipient should check this email and any attachments for the presence of viruses. THBS reserves the right to monitor and review the content of all messages sent to or from this e-mail address********
******* DISCLAIMER: This email and any files transmitted with it are privileged and confidential information and intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Torry Harris Business Solutions has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail.  The recipient should check this email and any attachments for the presence of viruses. THBS reserves the right to monitor and review the content of all messages sent to or from this e-mail address.********

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Per Hedeland
On 2017-09-01 13:58, Frank Muller wrote:
> Can someone help on this please?

Well, what is the problem that you want to solve? Erlang doesn't really
have an equivalent to C stdio, and anything you write with io:format/1,2,3
to "standard output" is sent immediately to the OS via a write(2) (or
typically writev(2)) call. Are you seeing something else?  Try this:

erl -noshell -eval 'io:format("foo "),timer:sleep(2000),io:format("bar~n"),timer:sleep(2000),erlang:halt(0)'

--Per

> Thanks
> /Frank
>
> Le mer. 30 août 2017 at 07:21, Frank Muller <[hidden email] <mailto:[hidden email]>> wrote :
>
>     Hi Prakash,
>
>     Thanks, but I doubt it's that easy.
>
>     In C, you've to call fflush or set the stdout to unbuffer mode.
>
>     /Frank
>
>     Le mer. 30 août 2017 à 06:51, Prakash Parmar <[hidden email] <mailto:[hidden email]>> wrote :
>
>         Hi Frank,
>
>         As per my experience, Like in C, keeping newline "~n" in io:format will flush it to screen.
>
>         /Prakash
>
>         -----Original Message-----
>         From: "Frank Muller" <[hidden email] <mailto:[hidden email]>>
>         Sent: Wednesday, August 30, 2017 9:58am
>         To: "Erlang-Questions Questions" <[hidden email] <mailto:[hidden email]>>
>         Subject: [erlang-questions] Flush "stdout" buffer
>
>         Hi guys,
>         I'd like to know if there's an equivalent function to C fflush(stdout) ?
>         I've a small Erlang program which prints data to stdout. After every io:format call, I'd like to ensure output is flushed to screen.
>         Thanks in adavance.
>         /Frank
>
>
>
>         ******* DISCLAIMER: This email and any files transmitted with it are privileged and confidential information and intended solely for the use of the individual or entity to which they are
>         addressed. If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its
>         contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Torry Harris Business Solutions has taken every reasonable precaution to minimize this risk,
>         but is not liable for any damage you may sustain as a result of any virus in this e-mail. The recipient should check this email and any attachments for the presence of viruses. THBS reserves
>         the right to monitor and review the content of all messages sent to or from this e-mail address********
>
>         ******* DISCLAIMER: This email and any files transmitted with it are privileged and confidential information and intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Torry Harris Business Solutions has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail.  The recipient should check this email and any attachments for the presence of viruses. THBS reserves the right to monitor and review the content of all messages sent to or from this e-mail address.********
>
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Frank Muller
Hi Per,

Hmmm ... So If i understand it well, we've no control on that after all.
 
The data can still be buffered on the kernel side. 

Thanks 
/Frank

Le ven. 1 sept. 2017 à 14:58, Per Hedeland <[hidden email]> a écrit :
On 2017-09-01 13:58, Frank Muller wrote:
> Can someone help on this please?

Well, what is the problem that you want to solve? Erlang doesn't really
have an equivalent to C stdio, and anything you write with io:format/1,2,3
to "standard output" is sent immediately to the OS via a write(2) (or
typically writev(2)) call. Are you seeing something else?  Try this:

erl -noshell -eval 'io:format("foo "),timer:sleep(2000),io:format("bar~n"),timer:sleep(2000),erlang:halt(0)'

--Per

> Thanks
> /Frank
>
> Le mer. 30 août 2017 at 07:21, Frank Muller <[hidden email] <mailto:[hidden email]>> wrote :
>
>     Hi Prakash,
>
>     Thanks, but I doubt it's that easy.
>
>     In C, you've to call fflush or set the stdout to unbuffer mode.
>
>     /Frank
>
>     Le mer. 30 août 2017 à 06:51, Prakash Parmar <[hidden email] <mailto:[hidden email]>> wrote :
>
>         Hi Frank,
>
>         As per my experience, Like in C, keeping newline "~n" in io:format will flush it to screen.
>
>         /Prakash
>
>         -----Original Message-----
>         From: "Frank Muller" <[hidden email] <mailto:[hidden email]>>
>         Sent: Wednesday, August 30, 2017 9:58am
>         To: "Erlang-Questions Questions" <[hidden email] <mailto:[hidden email]>>
>         Subject: [erlang-questions] Flush "stdout" buffer
>
>         Hi guys,
>         I'd like to know if there's an equivalent function to C fflush(stdout) ?
>         I've a small Erlang program which prints data to stdout. After every io:format call, I'd like to ensure output is flushed to screen.
>         Thanks in adavance.
>         /Frank
>
>
>
>         ******* DISCLAIMER: This email and any files transmitted with it are privileged and confidential information and intended solely for the use of the individual or entity to which they are
>         addressed. If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its
>         contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Torry Harris Business Solutions has taken every reasonable precaution to minimize this risk,
>         but is not liable for any damage you may sustain as a result of any virus in this e-mail. The recipient should check this email and any attachments for the presence of viruses. THBS reserves
>         the right to monitor and review the content of all messages sent to or from this e-mail address********
>
>         ******* DISCLAIMER: This email and any files transmitted with it are privileged and confidential information and intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Torry Harris Business Solutions has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail.  The recipient should check this email and any attachments for the presence of viruses. THBS reserves the right to monitor and review the content of all messages sent to or from this e-mail address.********
>
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

zxq9-2
On 2017年09月02日 土曜日 06:42:43 Frank Muller wrote:
> Hi Per,
>
> Hmmm ... So If i understand it well, we've no control on that after all.
>
> The data can still be buffered on the kernel side.

Right, WE don't have control over it, but the runtime already does, and as far as I can tell, fflush() is called every place it would be meaningful to do so already.

https://github.com/erlang/otp/search?utf8=%E2%9C%93&q=fflush

This is how things go when working with a MUCH higher level runtime than you seem to be accustimed to.

-Craig
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

zxq9-2
On 2017年09月02日 土曜日 15:54:51 zxq9 wrote:
> This is how things go when working with a MUCH higher level runtime than you seem to be accustimed to.

Tone doesn't transmit well in email.
I'd like to emphasize that this is not an indictment or condescending remark.
Letting go of the details really does take a lot of getting used to.

-Craig
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Frank Muller
Thanks Craig.

/Frank

Le sam. 2 sept. 2017 à 08:59, zxq9 <[hidden email]> a écrit :
On 2017年09月02日 土曜日 15:54:51 zxq9 wrote:
> This is how things go when working with a MUCH higher level runtime than you seem to be accustimed to.

Tone doesn't transmit well in email.
I'd like to emphasize that this is not an indictment or condescending remark.
Letting go of the details really does take a lot of getting used to.

-Craig
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Michael Truog
In reply to this post by Frank Muller
On 09/01/2017 04:58 AM, Frank Muller wrote:
> Can someone help on this please?
>
If you use stdout as an Erlang port like:

STDOUT =  erlang:open_port({fd, 0, 1}, [out, {line, 256}]),
erlang:port_command(STDOUT, "unbuffered").

Then you don't need to flush stdout and you could probably flush stdout by just using erlang:port_command(STDOUT, "").  Doing output this way bypasses the io server and is useful for doing things like progress bars in Erlang source code.

Best Regards,
Michael
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Per Hedeland
In reply to this post by Frank Muller
On 2017-09-02 08:42, Frank Muller wrote:
> Hi Per,
>
> Hmmm ... So If i understand it well, we've no control on that after all.
> The data can still be buffered on the kernel side.

Yes, just like it can when you call fflush() in C. C stdio implements a
user-level buffer where the result of your fprintf() etc call goes -
fflush() does a write(2)/writev(2) call with the contents of that
buffer. That's as far as you get with *any* language executing outside
the kernel, and for non-disk I/O it's really all that you need - for
disk I/O you may want to consider fsync(2) in rare cases, available in
Erlang as file:sync/1, but irrelevant for non-disk I/O.

You still haven't said what observed problem you want to solve...

--Per
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Per Hedeland
In reply to this post by zxq9-2
On 2017-09-02 08:54, zxq9 wrote:
> On 2017t0902å Üå 06:42:43 Frank Muller wrote:
>> Hi Per,
>>
>> Hmmm ... So If i understand it well, we've no control on that after all.
>>
>> The data can still be buffered on the kernel side.
>
> Right, WE don't have control over it, but the runtime already does, and as far as I can tell, fflush() is called every place it would be meaningful to do so already.

Well, as stated, that is probably correct, but as I at least implied,
the VM doesn't even *use* stdio for the I/O operations requested by
Erlang code. Basically all your "fflush" hits are in test / debug /
port-program code that doesn't even run in the VM.

--Per
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Éric Pailleau
In reply to this post by Per Hedeland
Hi,

 From Erlang perspective, you can get the message queue length of io
server handling your write.
io servers are erlang processes consuming an io messages queue.

To do so, you need to get the group leader pid of the writing process.

_______________________________________________
group_leader() -> pid()

Returns the process identifier of the group leader for the process
evaluating the function.

Every process is a member of some process group and all groups have a
group leader. All I/O from the group is channeled to the group leader.
When a new process is spawned, it gets the same group leader as the
spawning process. Initially, at system startup, init is both its own
group leader and the group leader of all processes.
_______________________________________________

then you can ask the number of messages still in queue of group leader,
using

  erlang:process_info/2 + message_queue_len


note that it is not however a guarantee that all data was flushed if
message queue = 0, because last one message maybe currently being
treated, and as other mentionned, you don't have control on what
underlaying OS is doing low level.



>>
>> Hmmm ... So If i understand it well, we've no control on that after all.
>> The data can still be buffered on the kernel side.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Per Hedeland
In reply to this post by Michael Truog
On 2017-09-02 10:07, Michael Truog wrote:
> On 09/01/2017 04:58 AM, Frank Muller wrote:
>> Can someone help on this please?
>>
> If you use stdout as an Erlang port like:
>
> STDOUT =  erlang:open_port({fd, 0, 1}, [out, {line, 256}]),
> erlang:port_command(STDOUT, "unbuffered").

Can you actually observe a difference *in buffering* between doing that
and just

io:format("unbuffered").

?

> Then you don't need to flush stdout

You also don't need to flush after a call to io:format/1 as above.

> and you could probably flush stdout by just using erlang:port_command(STDOUT, "").

I can't believe that it would have any effect whatsoever.

> Doing output this way bypasses the io server

True. But the io server doesn't *buffer*, does it? Obviously there is
some amount of message passing and processing that you avoid by going
directly to a file descriptor port, but I really don't think you can
observe that difference in interactive use. If you are creating an
application for non-interactive use that will write lots of data on its
"stdout", your method is probably preferable though.

> and is useful for doing things like progress bars in Erlang source code.

I think the very trivial -eval snippet I posted in another reply
(io:format/1 interleaved with timer:sleep/1) clearly shows that you can
do progress bars just fine with io:format/1 too. But besides the
obvious, such that your progress bar will be printed on the remote node
instead of locally when running a remote shell, and that you lose the
functionality of io:format/2, I noticed a difference that I hadn't
foreseen:

If you are at a (e.g.) shell prompt, your method will just print on the
current line *after* the prompt, while io:format/1 will result in
"printing before" the prompt, i.e. the line is redrawn with the prompt
at the end. Both are pretty ugly but in different ways:-) though, and I
guess a progress bar is typically printed when you have given a
"command" and are waiting for the result - i.e. *not* at a prompt - in
which case there is again no observable difference.

--Per
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Michael Truog
On 09/02/2017 04:08 AM, Per Hedeland wrote:

> On 2017-09-02 10:07, Michael Truog wrote:
>> On 09/01/2017 04:58 AM, Frank Muller wrote:
>>> Can someone help on this please?
>>>
>> If you use stdout as an Erlang port like:
>>
>> STDOUT =  erlang:open_port({fd, 0, 1}, [out, {line, 256}]),
>> erlang:port_command(STDOUT, "unbuffered").
>
> Can you actually observe a difference *in buffering* between doing that
> and just
>
> io:format("unbuffered").

Yes, it really does matter.  You will notice that the io module doesn't have any flush function, so you are unable to force any buffering in the Erlang VM to flush.  However, if you really care about stdout in unbuffered mode, you need to use the Erlang port as described above.  That is similar to using fcntl to set file descriptor 1 with O_NONBLOCK.  Programming languages normally have a way of setting stdout (and stderr, though stderr is really suppose to always be in unbuffered mode) in an unbuffered mode so all the output is available as quickly as possible.  An example for C++ is:

setbuf(stdout, NULL);
std::cout.setf(std::ios::unitbuf);

If you have pipes setup to handle stdout/stderr from a forked OS process that is executing a programming language, you want to have stdout and stderr in unbuffered mode to ensure the output is available without any extra delays (e.g., for logging).  Otherwise, it is problematic if the OS process crashes, because you may never receive a stacktrace (if one is provided) or any other output if it is available.

That is why each external (running in an OS process, not in the Erlang VM) CloudI API implementation (https://github.com/CloudI/CloudI/tree/develop/src/api) puts stdout and stderr into unbuffered mode.  In Erlang source code we can't really put the io server into an unbuffered mode, and there is no reason to really, because it is simpler to bypass the io server by making an Erlang port, as shown above.

Best Regards,
Michael
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Frank Muller
Ok, thanks for the clarification.

/Frank

Le sam. 2 sept. 2017 à 19:15, Michael Truog <[hidden email]> a wrote :
On 09/02/2017 04:08 AM, Per Hedeland wrote:
> On 2017-09-02 10:07, Michael Truog wrote:
>> On 09/01/2017 04:58 AM, Frank Muller wrote:
>>> Can someone help on this please?
>>>
>> If you use stdout as an Erlang port like:
>>
>> STDOUT =  erlang:open_port({fd, 0, 1}, [out, {line, 256}]),
>> erlang:port_command(STDOUT, "unbuffered").
>
> Can you actually observe a difference *in buffering* between doing that
> and just
>
> io:format("unbuffered").

Yes, it really does matter.  You will notice that the io module doesn't have any flush function, so you are unable to force any buffering in the Erlang VM to flush.  However, if you really care about stdout in unbuffered mode, you need to use the Erlang port as described above.  That is similar to using fcntl to set file descriptor 1 with O_NONBLOCK.  Programming languages normally have a way of setting stdout (and stderr, though stderr is really suppose to always be in unbuffered mode) in an unbuffered mode so all the output is available as quickly as possible.  An example for C++ is:

setbuf(stdout, NULL);
std::cout.setf(std::ios::unitbuf);

If you have pipes setup to handle stdout/stderr from a forked OS process that is executing a programming language, you want to have stdout and stderr in unbuffered mode to ensure the output is available without any extra delays (e.g., for logging).  Otherwise, it is problematic if the OS process crashes, because you may never receive a stacktrace (if one is provided) or any other output if it is available.

That is why each external (running in an OS process, not in the Erlang VM) CloudI API implementation (https://github.com/CloudI/CloudI/tree/develop/src/api) puts stdout and stderr into unbuffered mode.  In Erlang source code we can't really put the io server into an unbuffered mode, and there is no reason to really, because it is simpler to bypass the io server by making an Erlang port, as shown above.

Best Regards,
Michael

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Per Hedeland
In reply to this post by Michael Truog
On 2017-09-02 19:15, Michael Truog wrote:

> On 09/02/2017 04:08 AM, Per Hedeland wrote:
>> On 2017-09-02 10:07, Michael Truog wrote:
>>> On 09/01/2017 04:58 AM, Frank Muller wrote:
>>>> Can someone help on this please?
>>>>
>>> If you use stdout as an Erlang port like:
>>>
>>> STDOUT =  erlang:open_port({fd, 0, 1}, [out, {line, 256}]),
>>> erlang:port_command(STDOUT, "unbuffered").
>>
>> Can you actually observe a difference *in buffering* between doing that
>> and just
>>
>> io:format("unbuffered").
>
> Yes, it really does matter.

Unfortunately you didn't answer my question.

> You will notice that the io module doesn't have any flush function,

Yes, of course it doesn't have any flush function, since it doesn't do
any buffering.

> so you are unable to force any buffering in the Erlang VM to flush.

This gets us into the realm of philosophy - is it possible to be unable
to flush buffering that doesn't exist?

> However, if you really care about
> stdout in unbuffered mode, you need to use the Erlang port as
> described above.

Please provide *some* justification for that claim. At least a way to
observe that buffering is actually happening - ideally a pointer to the
specific piece of code that implements it.

I don't care much what you do in your code, but I strongly oppose that
you lead "newbies" to believe that they have to resort to ugly low-level
stuff like the above in order to achieve functionality that is actually
the default.

> That is similar to using fcntl to set file descriptor 1 with O_NONBLOCK.

No, neither that code nor making C stdout unbuffered is even remotely
similar to making file descriptor 1 non-blocking.

> Programming languages normally have a way of
> setting stdout (and stderr, though stderr is really suppose to always be in unbuffered mode) in an unbuffered mode so all the output is available as quickly as possible.

Yes, programming languages that have C stdio or a corresponding
buffering mechanism normally have a way of turning it off. Erlang
doesn't have such a mechanism (for non-disk I/O).

[snip irrelevant elaboration on C/C++ programming]

> In Erlang source code we can't really put the io server into an unbuffered mode, and there is no reason to really, because it is simpler to bypass the io server by making an Erlang port, as
> shown above.

No - there is no reason to do it, because it isn't needed.

Btw, this doesn't mean that Erlang "standard_io" is somehow "higher
level" than C stdio, it's rather the opposite - but it's simpler to use,
since you don't need to care about tweaking the buffering with setvbuf()
and/or fflush() like you do in C.

And finally, yet another demonstration of the lack of buffering in
Erlang "standard_io" - a progress bar!:-)

Eshell V8.3.5.1  (abort with ^G)
1> F = fun (0,_) -> ok; (N,G) -> io:format("*"),timer:sleep(1000),G(N-1,G) end.
#Fun<erl_eval.12.118419387>
2> F(20,F).

--Per
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Michael Truog
On 09/02/2017 09:22 PM, Per Hedeland wrote:

> On 2017-09-02 19:15, Michael Truog wrote:
>> On 09/02/2017 04:08 AM, Per Hedeland wrote:
>>> On 2017-09-02 10:07, Michael Truog wrote:
>>>> On 09/01/2017 04:58 AM, Frank Muller wrote:
>>>>> Can someone help on this please?
>>>>>
>>>> If you use stdout as an Erlang port like:
>>>>
>>>> STDOUT =  erlang:open_port({fd, 0, 1}, [out, {line, 256}]),
>>>> erlang:port_command(STDOUT, "unbuffered").
>>>
>>> Can you actually observe a difference *in buffering* between doing that
>>> and just
>>>
>>> io:format("unbuffered").
>>
>> Yes, it really does matter.
>
> Unfortunately you didn't answer my question.
>
>> You will notice that the io module doesn't have any flush function,
>
> Yes, of course it doesn't have any flush function, since it doesn't do
> any buffering.

When you call a function in the io module, what happens?  An io:format/1 function call sends the data in an Erlang message and that Erlang message goes into the group leader's Erlang process message queue, to get received when the Erlang process is ready for it.  That use of the Erlang process message queue is a form of buffering, ignoring everything that happens after that in the group leader Erlang process.

>
>> so you are unable to force any buffering in the Erlang VM to flush.
>
> This gets us into the realm of philosophy - is it possible to be unable
> to flush buffering that doesn't exist?

Refer to the explanation above.

>
>> However, if you really care about stdout in unbuffered mode, you need to use the Erlang port as
>> described above.
>
> Please provide *some* justification for that claim. At least a way to
> observe that buffering is actually happening - ideally a pointer to the
> specific piece of code that implements it.
>
> I don't care much what you do in your code, but I strongly oppose that
> you lead "newbies" to believe that they have to resort to ugly low-level
> stuff like the above in order to achieve functionality that is actually
> the default.

To allow output to be unbuffered, you need to bypass buffers.  If the approach isn't simple enough, that means there is room for improvement in the Erlang programming language.

>
>> That is similar to using fcntl to set file descriptor 1 with O_NONBLOCK.
>
> No, neither that code nor making C stdout unbuffered is even remotely
> similar to making file descriptor 1 non-blocking.
>
>> Programming languages normally have a way of setting stdout (and stderr, though stderr is really suppose to always be in unbuffered mode) in an unbuffered mode so all the output is available as quickly as possible.
>
> Yes, programming languages that have C stdio or a corresponding
> buffering mechanism normally have a way of turning it off. Erlang
> doesn't have such a mechanism (for non-disk I/O).
>
> [snip irrelevant elaboration on C/C++ programming]
>
>> In Erlang source code we can't really put the io server into an unbuffered mode, and there is no reason to really, because it is simpler to bypass the io server by making an Erlang port, as shown above.
>
> No - there is no reason to do it, because it isn't needed.
>
> Btw, this doesn't mean that Erlang "standard_io" is somehow "higher
> level" than C stdio, it's rather the opposite - but it's simpler to use,
> since you don't need to care about tweaking the buffering with setvbuf()
> and/or fflush() like you do in C.
>
> And finally, yet another demonstration of the lack of buffering in
> Erlang "standard_io" - a progress bar!:-)
>
> Eshell V8.3.5.1  (abort with ^G)
> 1> F = fun (0,_) -> ok; (N,G) -> io:format("*"),timer:sleep(1000),G(N-1,G) end.
> #Fun<erl_eval.12.118419387>
> 2> F(20,F).
>
To have a fast responsive progress bar, you need unbuffered output that won't get stuck on other use of io, allowing unimpeded use of the \b character that goes directly to the terminal without special terminal settings.  If you don't understand this, I have an example at https://github.com/okeuday/reltool_util/blob/b80cf8ecc423a108402529272ffe00f018bcf5ca/scope#L873-L944 .

Best Regards,
Michael

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Jesper Louis Andersen-2
On Sun, Sep 3, 2017 at 8:30 AM Michael Truog <[hidden email]> wrote:

When you call a function in the io module, what happens?  An io:format/1 function call sends the data in an Erlang message and that Erlang message goes into the group leader's Erlang process message queue, to get received when the Erlang process is ready for it.  That use of the Erlang process message queue is a form of buffering, ignoring everything that happens after that in the group leader Erlang process.


It is definitely asynchronous communication with latency.

I don't think it is buffering, however, albeit that is a bit philosophical. As soon as the message arrives, it is sent on. In no place do you delay the write in the hope that more data arrives shortly. If anything it acts like a queue, not a buffer.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Per Hedeland
On 2017-09-03 12:33, Jesper Louis Andersen wrote:

> On Sun, Sep 3, 2017 at 8:30 AM Michael Truog <[hidden email] <mailto:[hidden email]>> wrote:
>
>
>     When you call a function in the io module, what happens?  An io:format/1 function call sends the data in an Erlang message and that Erlang message goes into the group leader's Erlang process
>     message queue, to get received when the Erlang process is ready for it.  That use of the Erlang process message queue is a form of buffering, ignoring everything that happens after that in the
>     group leader Erlang process.
>
>
> It is definitely asynchronous communication with latency.
>
> I don't think it is buffering, however, albeit that is a bit philosophical. As soon as the message arrives, it is sent on. In no place do you delay the write in the hope that more data arrives
> shortly. If anything it acts like a queue, not a buffer.

Exactly.

So I guess we just have a bit of miscommunication - personally I haven't
previously encountered the use of the term "buffering" for "asynchronous
communication with latency", but I don't know if there is an formal
definition somewhere. However the thread started with the question of
how to do the equivalent of fflush() or "set to unbuffered" for C stdio,
which deals with the stdio buffering that *is* precisely "delay the
write in the hope that more data arrives shortly" - and will delay the
write *forever* in some cases unless you know what you're doing. And
this does not exist in Erlang "standard_io".

But for the question of how to do output with "maximal synchronicity"
and minimal latency, opening a file descriptor port and using
erlang:port_command/2,3 is probably a fine answer.

--Per
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Flush "stdout" buffer

Richard A. O'Keefe-2
In reply to this post by zxq9-2


On 2/09/17 6:54 PM, zxq9 wrote:
> On 2017年09月02日 土曜日 06:42:43 Frank Muller wrote:
>> Hi Per,
>>
>> Hmmm ... So If i understand it well, we've no control on that after all.
>>
>> The data can still be buffered on the kernel side.
>
> Right, WE don't have control over it, but the runtime already does, and as far as I can tell, fflush() is called every place it would be meaningful to do so already.

For kernel buffering there is fsync(fileno(...)).  Of course this
does nothing about *device* buffering...

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions