Erlang SSH client hangs after some thousand requests

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Erlang SSH client hangs after some thousand requests

Mario Santoro-2

Hi all,

I wrote an Erlang ssh client module for running commands on a remote host over an ssh connection.

The main calls to Erlang ssh library functions to setup the connection are the following:

ssh:connect/4

ssh_connection:session_channel/2

ssh_connection:ptty_alloc/3

ssh_connection:shell/2

 

afterwards I’m issuing a command to remote host via:

ssh_connection:send/3

 

and receiving related reply in a msg loop, with the following format:

{ssh_cm, ConnectionRef, {data, ChannelId, Type, Data}}

 

Everything works fine and I’m able to issue commands and receive responses correctly.

 

What I’m experiencing is a strange behavior on long lasting ssh sessions: when I start a long loop of

ssh send-receive requests, it seems like the client connection hangs after a certain number or positive command requests (several thousands),

waiting indefinitely for a reply, wherefrom it exits with timeout.

The only way to recover from this status is to disconnect and reconnect to the server.

 

I tried with both Erlang OTP/17.x and OTP/18.x and towards remote hosts running Linux and/or  Solaris OS, but the result is the same.

 

This behavior seems to be deterministic: given a command, it happens after the same number of issued requests (e.g. for a trivial “pwd” command,

the client always hangs after 2510 tries!!).

This make me think as there’s a limited receive buffer size somewhere into the client ssh library or something similar (maybe something that I should set at connection setup?);

since I’m quite newbie to Erlang, I’d very much appreciate if anyone could help me to find out the reason for this behaviour.

 

Thanks in advance for your help!

 

Regards,

/Mario

 

 

 

Ericsson

MARIO SANTORO
Service Project Manager
Region Mediterranean
RMED ITC CU SCM SL AIB DMO


Ericsson
Via Anagnina, 203
00118, ROME, ITALY
Mobile +39 335 699 5215
Office +39 06 7258 8404
[hidden email]
www.ericsson.com



http://www.ericsson.com/current_campaign

 

----------

Legal entity: Ericsson Telecomunicazioni SpA, registered office in Rome.
This Communication is Confidential. We only send and receive email on the basis of the terms set out at www.ericsson.com/email_disclaimer


_______________________________________________
erlang-bugs mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-bugs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Erlang SSH client hangs after some thousand requests

Hans Nilsson R (AL/EAB)
Do you call ssh_connection:adjust_window/3?

-Hans

On 12/12/2016 06:01 PM, Mario Santoro wrote:

> Hi all,
>
> I wrote an Erlang ssh client module for running commands on a remote
> host over an ssh connection.
>
> The main calls to Erlang ssh library functions to setup the connection
> are the following:
>
> ssh:connect/4
>
> ssh_connection:session_channel/2
>
> ssh_connection:ptty_alloc/3
>
> ssh_connection:shell/2
>
>  
>
> afterwards I’m issuing a command to remote host via:
>
> ssh_connection:send/3
>
>  
>
> and receiving related reply in a msg loop, with the following format:
>
> {ssh_cm, ConnectionRef, {data, ChannelId, Type, Data}}
>
>  
>
> Everything works fine and I’m able to issue commands and receive
> responses correctly.
>
>  
>
> What I’m experiencing is a strange behavior on long lasting ssh
> sessions: when I start a long loop of
>
> ssh send-receive requests, it seems like the client connection hangs
> after a certain number or positive command requests (several thousands),
>
> waiting indefinitely for a reply, wherefrom it exits with timeout.
>
> The only way to recover from this status is to disconnect and reconnect
> to the server.
>
>  
>
> I tried with both Erlang OTP/17.x and OTP/18.x and towards remote hosts
> running Linux and/or  Solaris OS, but the result is the same.
>
>  
>
> This behavior seems to be deterministic: given a command, it happens
> after the same number of issued requests (e.g. for a trivial “pwd” command,
>
> the client always hangs after 2510 tries!!).
>
> This make me think as there’s a limited receive buffer size somewhere
> into the client ssh library or something similar (maybe something that I
> should set at connection setup?);
>
> since I’m quite newbie to Erlang, I’d very much appreciate if anyone
> could help me to find out the reason for this behaviour.
>
>  
>
> Thanks in advance for your help!
>
>  
>
> Regards,
>
> /Mario
>
>  
>
>  
>
>  
>
> Ericsson <http://www.ericsson.com/>
>
> *MARIO SANTORO *
> Service Project Manager
> Region Mediterranean
> RMED ITC CU SCM SL AIB DMO
>
>
> *Ericsson*
> Via Anagnina, 203
> 00118, ROME, ITALY
> Mobile +39 335 699 5215
> Office +39 06 7258 8404
> [hidden email]
> www.ericsson.com
>
>
>
> http://www.ericsson.com/current_campaign
> <http://www.ericsson.com/current_campaign>
>
>  
>
> ----------
>
> Legal entity: Ericsson Telecomunicazioni SpA, registered office in Rome.
> This Communication is Confidential. We only send and receive email on
> the basis of the terms set out at www.ericsson.com/email_disclaimer
> <http://www.ericsson.com/email_disclaimer>
>
>
>
> _______________________________________________
> erlang-bugs mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-bugs
>
_______________________________________________
erlang-bugs mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-bugs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Erlang SSH client hangs after some thousand requests

Mario Santoro-2
No, I don't.
Should I?

-Mario

-----Original Message-----
From: Hans Nilsson R
Sent: 12 December 2016 18:23
To: Mario Santoro; [hidden email]
Subject: Re: [erlang-bugs] Erlang SSH client hangs after some thousand requests

Do you call ssh_connection:adjust_window/3?

-Hans

On 12/12/2016 06:01 PM, Mario Santoro wrote:

> Hi all,
>
> I wrote an Erlang ssh client module for running commands on a remote
> host over an ssh connection.
>
> The main calls to Erlang ssh library functions to setup the connection
> are the following:
>
> ssh:connect/4
>
> ssh_connection:session_channel/2
>
> ssh_connection:ptty_alloc/3
>
> ssh_connection:shell/2
>
>  
>
> afterwards I'm issuing a command to remote host via:
>
> ssh_connection:send/3
>
>  
>
> and receiving related reply in a msg loop, with the following format:
>
> {ssh_cm, ConnectionRef, {data, ChannelId, Type, Data}}
>
>  
>
> Everything works fine and I'm able to issue commands and receive
> responses correctly.
>
>  
>
> What I'm experiencing is a strange behavior on long lasting ssh
> sessions: when I start a long loop of
>
> ssh send-receive requests, it seems like the client connection hangs
> after a certain number or positive command requests (several
> thousands),
>
> waiting indefinitely for a reply, wherefrom it exits with timeout.
>
> The only way to recover from this status is to disconnect and
> reconnect to the server.
>
>  
>
> I tried with both Erlang OTP/17.x and OTP/18.x and towards remote
> hosts running Linux and/or  Solaris OS, but the result is the same.
>
>  
>
> This behavior seems to be deterministic: given a command, it happens
> after the same number of issued requests (e.g. for a trivial "pwd"
> command,
>
> the client always hangs after 2510 tries!!).
>
> This make me think as there's a limited receive buffer size somewhere
> into the client ssh library or something similar (maybe something that
> I should set at connection setup?);
>
> since I'm quite newbie to Erlang, I'd very much appreciate if anyone
> could help me to find out the reason for this behaviour.
>
>  
>
> Thanks in advance for your help!
>
>  
>
> Regards,
>
> /Mario
>
>  
>
>  
>
>  
>
> Ericsson <http://www.ericsson.com/>
>
> *MARIO SANTORO *
> Service Project Manager
> Region Mediterranean
> RMED ITC CU SCM SL AIB DMO
>
>
> *Ericsson*
> Via Anagnina, 203
> 00118, ROME, ITALY
> Mobile +39 335 699 5215
> Office +39 06 7258 8404
> [hidden email]
> www.ericsson.com
>
>
>
> http://www.ericsson.com/current_campaign
> <http://www.ericsson.com/current_campaign>
>
>  
>
> ----------
>
> Legal entity: Ericsson Telecomunicazioni SpA, registered office in Rome.
> This Communication is Confidential. We only send and receive email on
> the basis of the terms set out at www.ericsson.com/email_disclaimer
> <http://www.ericsson.com/email_disclaimer>
>
>
>
> _______________________________________________
> erlang-bugs mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-bugs
>
_______________________________________________
erlang-bugs mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-bugs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Erlang SSH client hangs after some thousand requests

Mario Santoro-2
Hi Hans,
calling ssh_connection:adjust_window/3 with a large enough bytesize seems to overtake the problem.
Thanks a lot for your useful hint! :)
-Mario

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Mario Santoro
Sent: 12 December 2016 18:25
To: Hans Nilsson R; [hidden email]
Subject: Re: [erlang-bugs] Erlang SSH client hangs after some thousand requests

No, I don't.
Should I?

-Mario

-----Original Message-----
From: Hans Nilsson R
Sent: 12 December 2016 18:23
To: Mario Santoro; [hidden email]
Subject: Re: [erlang-bugs] Erlang SSH client hangs after some thousand requests

Do you call ssh_connection:adjust_window/3?

-Hans

On 12/12/2016 06:01 PM, Mario Santoro wrote:

> Hi all,
>
> I wrote an Erlang ssh client module for running commands on a remote
> host over an ssh connection.
>
> The main calls to Erlang ssh library functions to setup the connection
> are the following:
>
> ssh:connect/4
>
> ssh_connection:session_channel/2
>
> ssh_connection:ptty_alloc/3
>
> ssh_connection:shell/2
>
>  
>
> afterwards I'm issuing a command to remote host via:
>
> ssh_connection:send/3
>
>  
>
> and receiving related reply in a msg loop, with the following format:
>
> {ssh_cm, ConnectionRef, {data, ChannelId, Type, Data}}
>
>  
>
> Everything works fine and I'm able to issue commands and receive
> responses correctly.
>
>  
>
> What I'm experiencing is a strange behavior on long lasting ssh
> sessions: when I start a long loop of
>
> ssh send-receive requests, it seems like the client connection hangs
> after a certain number or positive command requests (several
> thousands),
>
> waiting indefinitely for a reply, wherefrom it exits with timeout.
>
> The only way to recover from this status is to disconnect and
> reconnect to the server.
>
>  
>
> I tried with both Erlang OTP/17.x and OTP/18.x and towards remote
> hosts running Linux and/or  Solaris OS, but the result is the same.
>
>  
>
> This behavior seems to be deterministic: given a command, it happens
> after the same number of issued requests (e.g. for a trivial "pwd"
> command,
>
> the client always hangs after 2510 tries!!).
>
> This make me think as there's a limited receive buffer size somewhere
> into the client ssh library or something similar (maybe something that
> I should set at connection setup?);
>
> since I'm quite newbie to Erlang, I'd very much appreciate if anyone
> could help me to find out the reason for this behaviour.
>
>  
>
> Thanks in advance for your help!
>
>  
>
> Regards,
>
> /Mario
>
>  
>
>  
>
>  
>
> Ericsson <http://www.ericsson.com/>
>
> *MARIO SANTORO *
> Service Project Manager
> Region Mediterranean
> RMED ITC CU SCM SL AIB DMO
>
>
> *Ericsson*
> Via Anagnina, 203
> 00118, ROME, ITALY
> Mobile +39 335 699 5215
> Office +39 06 7258 8404
> [hidden email]
> www.ericsson.com
>
>
>
> http://www.ericsson.com/current_campaign
> <http://www.ericsson.com/current_campaign>
>
>  
>
> ----------
>
> Legal entity: Ericsson Telecomunicazioni SpA, registered office in Rome.
> This Communication is Confidential. We only send and receive email on
> the basis of the terms set out at www.ericsson.com/email_disclaimer
> <http://www.ericsson.com/email_disclaimer>
>
>
>
> _______________________________________________
> erlang-bugs mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-bugs
>
_______________________________________________
erlang-bugs mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-bugs
_______________________________________________
erlang-bugs mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-bugs
Loading...