Port process exits with status einval while external program still runs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Port process exits with status einval while external program still runs

Khitai Pang
I have observed a situation where the port process exits with reason
'einval' but the external program still runs.  The connected process of
the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
process exit with 'einval' while the external program still runs?


Thanks
Khitai
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Khitai Pang
The port is opened with {packet, 2}, I'm guessing that the data sent to
the port is too big and it's size can't fit in a two-byte integer.

Thanks
Khitai

On 2017/5/27 23:31, Khitai Pang wrote:

> I have observed a situation where the port process exits with reason
> 'einval' but the external program still runs.  The connected process of
> the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
> process exit with 'einval' while the external program still runs?
>
>
> Thanks
> Khitai
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Khitai Pang
No it can't be the case, port_command will return error and the data
will not be sent to the port.  This is really weird, this erlang app has
been running for a month and millions and millions of messages have been
processed by the ports, and this is first time I see port process exit
with einval and external prog still runs.

Thanks
Khitai

On 2017/5/27 23:42, Khitai Pang wrote:

> The port is opened with {packet, 2}, I'm guessing that the data sent to
> the port is too big and it's size can't fit in a two-byte integer.
>
> Thanks
> Khitai
>
> On 2017/5/27 23:31, Khitai Pang wrote:
>> I have observed a situation where the port process exits with reason
>> 'einval' but the external program still runs.  The connected process of
>> the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
>> process exit with 'einval' while the external program still runs?
>>
>>
>> Thanks
>> Khitai
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Michael Truog
In reply to this post by Khitai Pang
  On 2017/5/27 23:31, Khitai Pang wrote:
>> I have observed a situation where the port process exits with reason
>> 'einval' but the external program still runs.  The connected process of
>> the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
>> process exit with 'einval' while the external program still runs?

Normally that is due to the port source code not handling a HUP on the output file descriptor, from poll (POLLHUP) or the equivalent using a different function.

Best Regards,
Michael
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Khitai Pang
On 2017/5/28 00:06, Michael Truog wrote:

 On 2017/5/27 23:31, Khitai Pang wrote:
I have observed a situation where the port process exits with reason
'einval' but the external program still runs.  The connected process of
the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
process exit with 'einval' while the external program still runs?

Normally that is due to the port source code

Do you mean the source code of the external program?

not handling a HUP on the output file descriptor,from poll (POLLHUP) or the equivalent using a different function.

Could you be more specific about this situation?  In what occasions will this happen?

My external program is a python script that uses erlport to communicate with erlang.


Thanks
Khitai

Best Regards,
Michael


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Michael Truog
On 05/27/2017 09:21 AM, Khitai Pang wrote:
On 2017/5/28 00:06, Michael Truog wrote:

 On 2017/5/27 23:31, Khitai Pang wrote:
I have observed a situation where the port process exits with reason
'einval' but the external program still runs.  The connected process of
the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
process exit with 'einval' while the external program still runs?

Normally that is due to the port source code

Do you mean the source code of the external program?

Yes


not handling a HUP on the output file descriptor,from poll (POLLHUP) or the equivalent using a different function.

Could you be more specific about this situation?  In what occasions will this happen?

My external program is a python script that uses erlport to communicate with erlang.

erlport should be getting EPIPE in the source code https://github.com/hdima/erlport/blob/master/priv/python2/erlport/erlproto.py#L106-L110 .  However, that depends on the write function getting called to discover the Erlang VM disappeared.

Best Regards,
Michael

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Khitai Pang
On 2017/5/28 01:45, Michael Truog wrote:
On 05/27/2017 09:21 AM, Khitai Pang wrote:
On 2017/5/28 00:06, Michael Truog wrote:

 On 2017/5/27 23:31, Khitai Pang wrote:
I have observed a situation where the port process exits with reason
'einval' but the external program still runs.  The connected process of
the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
process exit with 'einval' while the external program still runs?

Normally that is due to the port source code

Do you mean the source code of the external program?

Yes


not handling a HUP on the output file descriptor,from poll (POLLHUP) or the equivalent using a different function.

Could you be more specific about this situation?  In what occasions will this happen?

My external program is a python script that uses erlport to communicate with erlang.

erlport should be getting EPIPE in the source code https://github.com/hdima/erlport/blob/master/priv/python2/erlport/erlproto.py#L106-L110 .  However, that depends on the write function getting called to discover the Erlang VM disappeared.

Thank you for your explanation, but I don't really understand, if the cause is that the Erlang VM certainly disappeared, how come the connected process got the {EXIT,#Port<0....>,einval} message?

Anyway, what to do to avoid this issue?

Thanks
Khitai

Best Regards,
Michael


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Michael Truog
On 05/27/2017 12:06 PM, Khitai Pang wrote:
On 2017/5/28 01:45, Michael Truog wrote:
On 05/27/2017 09:21 AM, Khitai Pang wrote:
On 2017/5/28 00:06, Michael Truog wrote:

 On 2017/5/27 23:31, Khitai Pang wrote:
I have observed a situation where the port process exits with reason
'einval' but the external program still runs.  The connected process of
the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
process exit with 'einval' while the external program still runs?

Normally that is due to the port source code

Do you mean the source code of the external program?

Yes


not handling a HUP on the output file descriptor,from poll (POLLHUP) or the equivalent using a different function.

Could you be more specific about this situation?  In what occasions will this happen?

My external program is a python script that uses erlport to communicate with erlang.

erlport should be getting EPIPE in the source code https://github.com/hdima/erlport/blob/master/priv/python2/erlport/erlproto.py#L106-L110 .  However, that depends on the write function getting called to discover the Erlang VM disappeared.

Thank you for your explanation, but I don't really understand, if the cause is that the Erlang VM certainly disappeared, how come the connected process got the {EXIT,#Port<0....>,einval} message?
That message is the Erlang message inside the Erlang VM related to the Erlang port type used for the pipes that were used for the erlport executable.  So, that means the Erlang port type died with a posix einval error, but that doesn't mean that the erlport executable is aware of a problem.  It is possible the error is only seen by the Erlang VM if the erlport has no reason to call the write function, mentioned above.

Anyway, what to do to avoid this issue?
Not sure, it may require changing erlport or taking a different approach.  To avoid the problem, you could attempt to always make the erlport executable exit by executing "sys.exit(0)" or something similar, but that isn't dependable when you are unable to send erlport something to exit (i.e., if a sudden error makes the Erlang port type unusable in the Erlang VM, as appears to have happened here).

Best Regards,
Michael

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Khitai Pang
On 2017/5/28 03:28, Michael Truog wrote:
On 05/27/2017 12:06 PM, Khitai Pang wrote:
On 2017/5/28 01:45, Michael Truog wrote:
On 05/27/2017 09:21 AM, Khitai Pang wrote:
On 2017/5/28 00:06, Michael Truog wrote:

 On 2017/5/27 23:31, Khitai Pang wrote:
I have observed a situation where the port process exits with reason
'einval' but the external program still runs.  The connected process of
the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
process exit with 'einval' while the external program still runs?

Normally that is due to the port source code

Do you mean the source code of the external program?

Yes


not handling a HUP on the output file descriptor,from poll (POLLHUP) or the equivalent using a different function.

Could you be more specific about this situation?  In what occasions will this happen?

My external program is a python script that uses erlport to communicate with erlang.

erlport should be getting EPIPE in the source code https://github.com/hdima/erlport/blob/master/priv/python2/erlport/erlproto.py#L106-L110 .  However, that depends on the write function getting called to discover the Erlang VM disappeared.

Thank you for your explanation, but I don't really understand, if the cause is that the Erlang VM certainly disappeared, how come the connected process got the {EXIT,#Port<0....>,einval} message?
That message is the Erlang message inside the Erlang VM related to the Erlang port type used for the pipes that were used for the erlport executable.  So, that means the Erlang port type died with a posix einval error, but that doesn't mean that the erlport executable is aware of a problem.  It is possible the error is only seen by the Erlang VM if the erlport has no reason to call the write function, mentioned above.

Hi Michael,

OK I see what you mean, the external program process doesn't know that its Erlang port process has exited, because it hasn't written anything to the output file descriptor after that.  I think you are right about this.

Anyway, what to do to avoid this issue?
Not sure, it may require changing erlport or taking a different approach.  To avoid the problem, you could attempt to always make the erlport executable exit by executing "sys.exit(0)" or something similar, but that isn't dependable when you are unable to send erlport something to exit (i.e., if a sudden error makes the Erlang port type unusable in the Erlang VM, as appears to have happened here).

When the Erlang port process is gone, I don't think the Erlang VM can send anything to the external program process.  However, I think we still have a way to kill the external program process: after creating a port, use erlang:port_info(Port, os_pid) to get the OS pid of the external program process and save it somewhere, e.g. in gen_server state.  When the Erlang port process dies, kill the external program process by its OS pid.  I haven't tested it, though.

Actually, in my case, it's not a serious problem that the external program process is not aware of the exiting of its Erlang port process and keeps running after that.  What I'm really concerned with is why the Erlang port process died with einval error, this is a serious problem and I need to find a way to fix it.  Any idea?


Thanks
Khitai

Best Regards,
Michael

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Michael Truog
On 05/27/2017 01:35 PM, Khitai Pang wrote:
On 2017/5/28 03:28, Michael Truog wrote:
On 05/27/2017 12:06 PM, Khitai Pang wrote:
On 2017/5/28 01:45, Michael Truog wrote:
On 05/27/2017 09:21 AM, Khitai Pang wrote:
On 2017/5/28 00:06, Michael Truog wrote:

 On 2017/5/27 23:31, Khitai Pang wrote:
I have observed a situation where the port process exits with reason
'einval' but the external program still runs.  The connected process of
the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
process exit with 'einval' while the external program still runs?

Normally that is due to the port source code

Do you mean the source code of the external program?

Yes


not handling a HUP on the output file descriptor,from poll (POLLHUP) or the equivalent using a different function.

Could you be more specific about this situation?  In what occasions will this happen?

My external program is a python script that uses erlport to communicate with erlang.

erlport should be getting EPIPE in the source code https://github.com/hdima/erlport/blob/master/priv/python2/erlport/erlproto.py#L106-L110 .  However, that depends on the write function getting called to discover the Erlang VM disappeared.

Thank you for your explanation, but I don't really understand, if the cause is that the Erlang VM certainly disappeared, how come the connected process got the {EXIT,#Port<0....>,einval} message?
That message is the Erlang message inside the Erlang VM related to the Erlang port type used for the pipes that were used for the erlport executable.  So, that means the Erlang port type died with a posix einval error, but that doesn't mean that the erlport executable is aware of a problem.  It is possible the error is only seen by the Erlang VM if the erlport has no reason to call the write function, mentioned above.

Hi Michael,

OK I see what you mean, the external program process doesn't know that its Erlang port process has exited, because it hasn't written anything to the output file descriptor after that.  I think you are right about this.

Anyway, what to do to avoid this issue?
Not sure, it may require changing erlport or taking a different approach.  To avoid the problem, you could attempt to always make the erlport executable exit by executing "sys.exit(0)" or something similar, but that isn't dependable when you are unable to send erlport something to exit (i.e., if a sudden error makes the Erlang port type unusable in the Erlang VM, as appears to have happened here).

When the Erlang port process is gone, I don't think the Erlang VM can send anything to the external program process.  However, I think we still have a way to kill the external program process: after creating a port, use erlang:port_info(Port, os_pid) to get the OS pid of the external program process and save it somewhere, e.g. in gen_server state.  When the Erlang port process dies, kill the external program process by its OS pid.  I haven't tested it, though.

Actually, in my case, it's not a serious problem that the external program process is not aware of the exiting of its Erlang port process and keeps running after that.  What I'm really concerned with is why the Erlang port process died with einval error, this is a serious problem and I need to find a way to fix it.  Any idea?

Based on http://erlang.org/doc/man/erlang.html#open_port-2 documentation "einval if no POSIX code is appropriate", einval is not the normal POSIX error from exec.  If the port executable is still running, then perhaps it unintentionally closed the pipes the Erlang VM is trying to use (since that doesn't appear to be mentioned in any of the other POSIX errors).  That may have happened after a python exception possibly, but this is something you would need to explore.

Best Regards,
Michael

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Port process exits with status einval while external program still runs

Khitai Pang
On 2017/5/28 06:00, Michael Truog wrote:
On 05/27/2017 01:35 PM, Khitai Pang wrote:
On 2017/5/28 03:28, Michael Truog wrote:
On 05/27/2017 12:06 PM, Khitai Pang wrote:
On 2017/5/28 01:45, Michael Truog wrote:
On 05/27/2017 09:21 AM, Khitai Pang wrote:
On 2017/5/28 00:06, Michael Truog wrote:

 On 2017/5/27 23:31, Khitai Pang wrote:
I have observed a situation where the port process exits with reason
'einval' but the external program still runs.  The connected process of
the port receives {EXIT,#Port<0.3069817>,einval}.  What makes a port
process exit with 'einval' while the external program still runs?

Normally that is due to the port source code

Do you mean the source code of the external program?

Yes


not handling a HUP on the output file descriptor,from poll (POLLHUP) or the equivalent using a different function.

Could you be more specific about this situation?  In what occasions will this happen?

My external program is a python script that uses erlport to communicate with erlang.

erlport should be getting EPIPE in the source code https://github.com/hdima/erlport/blob/master/priv/python2/erlport/erlproto.py#L106-L110 .  However, that depends on the write function getting called to discover the Erlang VM disappeared.

Thank you for your explanation, but I don't really understand, if the cause is that the Erlang VM certainly disappeared, how come the connected process got the {EXIT,#Port<0....>,einval} message?
That message is the Erlang message inside the Erlang VM related to the Erlang port type used for the pipes that were used for the erlport executable.  So, that means the Erlang port type died with a posix einval error, but that doesn't mean that the erlport executable is aware of a problem.  It is possible the error is only seen by the Erlang VM if the erlport has no reason to call the write function, mentioned above.

Hi Michael,

OK I see what you mean, the external program process doesn't know that its Erlang port process has exited, because it hasn't written anything to the output file descriptor after that.  I think you are right about this.

Anyway, what to do to avoid this issue?
Not sure, it may require changing erlport or taking a different approach.  To avoid the problem, you could attempt to always make the erlport executable exit by executing "sys.exit(0)" or something similar, but that isn't dependable when you are unable to send erlport something to exit (i.e., if a sudden error makes the Erlang port type unusable in the Erlang VM, as appears to have happened here).

When the Erlang port process is gone, I don't think the Erlang VM can send anything to the external program process.  However, I think we still have a way to kill the external program process: after creating a port, use erlang:port_info(Port, os_pid) to get the OS pid of the external program process and save it somewhere, e.g. in gen_server state.  When the Erlang port process dies, kill the external program process by its OS pid.  I haven't tested it, though.

Actually, in my case, it's not a serious problem that the external program process is not aware of the exiting of its Erlang port process and keeps running after that.  What I'm really concerned with is why the Erlang port process died with einval error, this is a serious problem and I need to find a way to fix it.  Any idea?

Based on http://erlang.org/doc/man/erlang.html#open_port-2 documentation "einval if no POSIX code is appropriate", einval is not the normal POSIX error from exec.  If the port executable is still running, then perhaps it unintentionally closed the pipes the Erlang VM is trying to use (since that doesn't appear to be mentioned in any of the other POSIX errors).  That may have happened after a python exception possibly, but this is something you would need to explore.

I also doubt that the external program (python script using erlport-0.6) did something funny and confused the port process, but no error/exception was caught in the external program log, and the issue can't be reproduced now.  Anyway, I'll keep looking into it.  Thanks!

Best Regards,
Khitai

Best Regards,
Michael

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions