Lots of epmd in pod

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Lots of epmd in pod

bengt e
Greetings,

There is an Erlang release(*) running in a Kubernetes pod. When I use the start script (even if it is only to see the arguments) I get more epmd processes(**).
Is that just me? A known problem that can be ignored? Worth investigation?

Best Wishes,
bengt

(*)
The Docker file has:
erlang:21.3.8.5-alpine and alpine:3.9 (one for build and one for container)

Build command:
RUN rebar3 as prod release

The rebar.config:
{relx, [{release, { 'ercdf', "1.3.7" }, [ercdf, recon, observer_cli]},
{dev_mode, true},
{include_erts, false},
{extended_start_script, true}]
}.

(**)
Example after having kubectl exec'd:
 # ps | grep epmd | wc
       46       186      1326
 #  /opt/ercdf/bin/ercdf remote_console
Erlang/OTP 21 [erts-10.3.5.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V10.3.5.4  (abort with ^G)
User switch command
 --> q
 # ps | grep epmd | wc
       58       234      1662


Sample of the set of processes that grows
 2136 root      0:00 [erl_child_setup]
 2166 root      0:00 [inet_gethost]
 2226 root      0:00 [epmd]
 2227 root      0:00 [epmd]

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lots of epmd in pod

Bryan Hunt
I bet it’s zombie processes - the epmd process is not being reaped.


The epmd will be cleaned up when the pod terminates. 

Docker fixed this by adding an —init option which runs the tini init process


Kubernetes does not start docker containers with this option 

You can use tini as a tiny init process to solve this issue - it’s already an alpine package.






Code Sync & Erlang Solutions Conferences

Code BEAM Lite BD - Budapest: 20 September 2019

Code BEAM Lite NYC - NYC: 01 October 2019

Code BEAM Lite - Berlin: 11 October 2019

RabbitMQ Summit - London: 4 November 2019

Code Mesh LDN - London: 7-8 November 2019

Code BEAM Lite India - Bangalore: 14 November 2019

Code BEAM Lite AMS - Amsterdam: 28 November 2019

Lambda Days - Kraków: 13-14 February 2020

Code BEAM SF - San Francisco: 5-6 March 2020


Erlang Solutions cares about your data and privacy; please find all details about the basis for communicating with you and the way we process your data in our Privacy Policy.You can update your email preferences or opt-out from receiving Marketing emails here.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lots of epmd in pod

Tristan Sloughter-4
Based on the output this is likely the issue but why are the epmd processes going down in the first place? Usually the first one keeps running and no new ones are created, so this issue of reaping doesn't come up.

Another option is to not use epmd when running in kubernetes. There aren't port issues in k8s so it isn't needed.


The only catch is you need to set the env var `EPMDLESS_REMSH_PORT` to the port used for distribution so that remote_console works the same as before.

I'll get something written up about using this in kubernetes soon.

Tristan

On Fri, Aug 9, 2019, at 04:35, Bryan Hunt wrote:
I bet it’s zombie processes - the epmd process is not being reaped.


The epmd will be cleaned up when the pod terminates. 

Docker fixed this by adding an —init option which runs the tini init process


Kubernetes does not start docker containers with this option 

You can use tini as a tiny init process to solve this issue - it’s already an alpine package.







Code Sync & Erlang Solutions Conferences

Code BEAM Lite BD - Budapest: 20 September 2019

Code BEAM Lite NYC - NYC: 01 October 2019

Code BEAM Lite - Berlin: 11 October 2019

RabbitMQ Summit - London: 4 November 2019

Code Mesh LDN - London: 7-8 November 2019

Code BEAM Lite India - Bangalore: 14 November 2019

Code BEAM Lite AMS - Amsterdam: 28 November 2019

Lambda Days - Kraków: 13-14 February 2020

Code BEAM SF - San Francisco: 5-6 March 2020


Erlang Solutions cares about your data and privacy; please find all details about the basis for communicating with you and the way we process your data in our Privacy Policy.You can update your email preferences or opt-out from receiving Marketing emails here.

_______________________________________________
erlang-questions mailing list
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions