Multiple BEAM nodes on one Linux node — a thought experiment

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple BEAM nodes on one Linux node — a thought experiment

Lloyd R. Prentice-2
Hello,

My Mickey-the-Dunce ignorance and curiosity has led me to explore Linux containers. Very cool. Then, a recent conversation with Oliver Korpilla tweaked my wonder nerve:

Is it possible/practical to run multiple BEAM nodes on a single Linux node?

Assume each BEAM node supports an identical web application with a Mnesia backend serving a set of CRUD services— each for a different client.

Has anyone ever done anything along this line?

Best wishes,

LRP

Sent from my iPad
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Multiple BEAM nodes on one Linux node — a thought experiment

Vincent A Okonkwo
That is pretty much what erlang release is.

I think its one of the reason docker/containers didn't catch on in the erlang community. The major problem you may face running it especially in distributed erlang mode may be name collusion in empd

On Tue, May 14, 2019 at 9:13 PM Lloyd R. Prentice <[hidden email]> wrote:
Hello,

My Mickey-the-Dunce ignorance and curiosity has led me to explore Linux containers. Very cool. Then, a recent conversation with Oliver Korpilla tweaked my wonder nerve:

Is it possible/practical to run multiple BEAM nodes on a single Linux node?

Assume each BEAM node supports an identical web application with a Mnesia backend serving a set of CRUD services— each for a different client.

Has anyone ever done anything along this line?

Best wishes,

LRP

Sent from my iPad
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Multiple BEAM nodes on one Linux node — a thought experiment

scott ribe
> On May 14, 2019, at 3:36 PM, Chinedu Okonkwo <[hidden email]> wrote:
>
> Assume each BEAM node supports an identical web application with a Mnesia backend serving a set of CRUD services— each for a different client.

Perfectly possible, but also pointless. You'd normally just have one BEAM node with one erlang process per client.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Multiple BEAM nodes on one Linux node — a thought experiment

Lloyd R. Prentice-2
Hi Scott,

Can you elaborate please?

Thanks,

Lloyd

Sent from my iPad

On May 14, 2019, at 5:45 PM, Scott Ribe <[hidden email]> wrote:

>> On May 14, 2019, at 3:36 PM, Chinedu Okonkwo <[hidden email]> wrote:
>>
>> Assume each BEAM node supports an identical web application with a Mnesia backend serving a set of CRUD services— each for a different client.
>
> Perfectly possible, but also pointless. You'd normally just have one BEAM node with one erlang process per client.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Multiple BEAM nodes on one Linux node — a thought experiment

Lloyd R. Prentice-2
In reply to this post by Vincent A Okonkwo
Hi Chinedu,

So operationally, would one install a release for each user in separate directories?

My musings, uninformed by experience, have progressed more or less along this line:

— Data loss is totally unacceptable to the users I have in mind.

— There could be many users, each with his/her own corpus of data; no intersection.

— Traffic per user would be quite low. 

— Data corpus per user would be fairly small. 

— Users come and go. When they go, they want to take their data. Some may want to return and pickup where they left off.

— The traditional solution would be a capacious, maybe even distributed, database behind a single, well maybe load balanced, front end. 

— But this means that there could potentially be an increasing volume of “dead,” e.g. unused data, or one would need a way to archive data when a user leaves. In this second case, archiving and restoring data over many users could tie up a great deal of db’s bandwidth.

— Fanciful solution I: Give each user his/her own independent website. 

— Linux containers might make this possible.

— Transactions are not an issue.

— A small Mnesia db or even dets db per user would suffice.

— Linux containers could be deployed over multiple hardware or VPS nodes, lowering hardware failure risk and making it relatively easy to scale from just a few users to many many.

LXD provides tools that would make it relatively easy to induct new users and replicate user sites.

— But would it be economically feasible?

— Fanciful solution II:  Multiple BEAM nodes per Linux node.

— Hmmm... might have to functionally replicate tools that LXD already provides.

— But it might be more resource efficient.

— Can it even work?

And so, my head spins for lack of experience.

And so, I’d much appreciate insight and guidance from experienced hands.

All the best,

Lloyd

Sent from my iPad

On May 14, 2019, at 5:36 PM, Chinedu Okonkwo <[hidden email]> wrote:

That is pretty much what erlang release is.

I think its one of the reason docker/containers didn't catch on in the erlang community. The major problem you may face running it especially in distributed erlang mode may be name collusion in empd

On Tue, May 14, 2019 at 9:13 PM Lloyd R. Prentice <[hidden email]> wrote:
Hello,

My Mickey-the-Dunce ignorance and curiosity has led me to explore Linux containers. Very cool. Then, a recent conversation with Oliver Korpilla tweaked my wonder nerve:

Is it possible/practical to run multiple BEAM nodes on a single Linux node?

Assume each BEAM node supports an identical web application with a Mnesia backend serving a set of CRUD services— each for a different client.

Has anyone ever done anything along this line?

Best wishes,

LRP

Sent from my iPad
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Multiple BEAM nodes on one Linux node — a thought experiment

Vincent A Okonkwo

Hi Lloyd,

 

Unless each user will directly have access to the erlang node (via remsh) or there is a compliance requirement at play, I will like to re-iterate Scott’s point. The is approach is pointless.

 

Erlang release is portable (per OS; so a release created on Linux only contain VM for linux), like every portable software all you need is extract and run the script.

 

But like every application you hit the same limitation, if this is a network/web app (which I assume it is), then they will each have to be configured on different ports,

because multiple applications can’t listen on same port (there are some exceptions to this), you may want to jail the application into certain folders so they don’t file:read_file(‘/etc/passwd’) on you. There will also be the backup/restore systems you will have to develop, the gateway to handle routing a user to the right instance of the  app or distribute a host:port url for each user, a lot of other things you will need just to make this operational. As I said earlier this approach is almost pointless.

 

My take will be

  1. You are better off implementing this segregation in the application layer, even if there are different apps you can route the user to the said app quite easily. Segregation can be handled in the app, you can have one dets per user or one table per user in mnesia in a single vm, you will also have all the programmatic tools to handle user idleness and shutdown unused resources.

From your description, I think the traditional solution is also a good solution. If anything you will likely underutilize your resources.

 

  1. If necessary, just use a container then workout the infrastructure to route the requests, this will need some service discovery or a manual management of the route map (eg: nginx config). But the benefit of using this instead of a raw deployment is that a lot of the issues listed have solutions that have been tested.

 

From: [hidden email]
Sent: Tuesday, 14 May 2019 11:59 PM
To: [hidden email]
Cc: [hidden email]
Subject: Re: [erlang-questions] Multiple BEAM nodes on one Linux node — a thought experiment

 

Hi Chinedu,

 

So operationally, would one install a release for each user in separate directories?

 

My musings, uninformed by experience, have progressed more or less along this line:

 

— Data loss is totally unacceptable to the users I have in mind.

 

— There could be many users, each with his/her own corpus of data; no intersection.

 

— Traffic per user would be quite low. 

 

— Data corpus per user would be fairly small. 

 

— Users come and go. When they go, they want to take their data. Some may want to return and pickup where they left off.

 

— The traditional solution would be a capacious, maybe even distributed, database behind a single, well maybe load balanced, front end. 

 

— But this means that there could potentially be an increasing volume of “dead,” e.g. unused data, or one would need a way to archive data when a user leaves. In this second case, archiving and restoring data over many users could tie up a great deal of db’s bandwidth.

 

— Fanciful solution I: Give each user his/her own independent website. 

 

— Linux containers might make this possible.

 

— Transactions are not an issue.

 

— A small Mnesia db or even dets db per user would suffice.

 

— Linux containers could be deployed over multiple hardware or VPS nodes, lowering hardware failure risk and making it relatively easy to scale from just a few users to many many.

 

— LXD provides tools that would make it relatively easy to induct new users and replicate user sites.

 

— But would it be economically feasible?

 

— Fanciful solution II:  Multiple BEAM nodes per Linux node.

 

— Hmmm... might have to functionally replicate tools that LXD already provides.

 

— But it might be more resource efficient.

 

— Can it even work?

 

And so, my head spins for lack of experience.

 

And so, I’d much appreciate insight and guidance from experienced hands.

 

All the best,

 

Lloyd

 

Sent from my iPad


On May 14, 2019, at 5:36 PM, Chinedu Okonkwo <[hidden email]> wrote:

That is pretty much what erlang release is.

 

I think its one of the reason docker/containers didn't catch on in the erlang community. The major problem you may face running it especially in distributed erlang mode may be name collusion in empd

 

On Tue, May 14, 2019 at 9:13 PM Lloyd R. Prentice <[hidden email]> wrote:

Hello,

My Mickey-the-Dunce ignorance and curiosity has led me to explore Linux containers. Very cool. Then, a recent conversation with Oliver Korpilla tweaked my wonder nerve:

Is it possible/practical to run multiple BEAM nodes on a single Linux node?

Assume each BEAM node supports an identical web application with a Mnesia backend serving a set of CRUD services— each for a different client.

Has anyone ever done anything along this line?

Best wishes,

LRP

Sent from my iPad
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

 


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Multiple BEAM nodes on one Linux node — a thought experiment

Lloyd R. Prentice-2
Hi Chinedu,

> “...one dets per user or one table per user in mnesia...”

This is very interesting. How would this work for archiving and restoring user data?

Can’t tell you how much I appreciate your input.

Best,

Lloyd

Sent from my iPad

> On May 14, 2019, at 8:50 PM, Chinedu Okonkwo <[hidden email]> wrote:
>
> one table per user in mnesia

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Multiple BEAM nodes on one Linux node — a thought experiment

Lloyd R. Prentice-2
In reply to this post by Vincent A Okonkwo
Ah, reading the Mnesia docs, I see how this can work. 

You’ve helped clarify many things.

Many thanks,

Lloyd

Sent from my iPad

On May 14, 2019, at 8:50 PM, Chinedu Okonkwo <[hidden email]> wrote:

Hi Lloyd,

 

Unless each user will directly have access to the erlang node (via remsh) or there is a compliance requirement at play, I will like to re-iterate Scott’s point. The is approach is pointless.

 

Erlang release is portable (per OS; so a release created on Linux only contain VM for linux), like every portable software all you need is extract and run the script.

 

But like every application you hit the same limitation, if this is a network/web app (which I assume it is), then they will each have to be configured on different ports,

because multiple applications can’t listen on same port (there are some exceptions to this), you may want to jail the application into certain folders so they don’t file:read_file(‘/etc/passwd’) on you. There will also be the backup/restore systems you will have to develop, the gateway to handle routing a user to the right instance of the  app or distribute a host:port url for each user, a lot of other things you will need just to make this operational. As I said earlier this approach is almost pointless.

 

My take will be

  1. You are better off implementing this segregation in the application layer, even if there are different apps you can route the user to the said app quite easily. Segregation can be handled in the app, you can have one dets per user or one table per user in mnesia in a single vm, you will also have all the programmatic tools to handle user idleness and shutdown unused resources.

From your description, I think the traditional solution is also a good solution. If anything you will likely underutilize your resources.

 

  1. If necessary, just use a container then workout the infrastructure to route the requests, this will need some service discovery or a manual management of the route map (eg: nginx config). But the benefit of using this instead of a raw deployment is that a lot of the issues listed have solutions that have been tested.

 

From: [hidden email]
Sent: Tuesday, 14 May 2019 11:59 PM
To: [hidden email]
Cc: [hidden email]
Subject: Re: [erlang-questions] Multiple BEAM nodes on one Linux node — a thought experiment

 

Hi Chinedu,

 

So operationally, would one install a release for each user in separate directories?

 

My musings, uninformed by experience, have progressed more or less along this line:

 

— Data loss is totally unacceptable to the users I have in mind.

 

— There could be many users, each with his/her own corpus of data; no intersection.

 

— Traffic per user would be quite low. 

 

— Data corpus per user would be fairly small. 

 

— Users come and go. When they go, they want to take their data. Some may want to return and pickup where they left off.

 

— The traditional solution would be a capacious, maybe even distributed, database behind a single, well maybe load balanced, front end. 

 

— But this means that there could potentially be an increasing volume of “dead,” e.g. unused data, or one would need a way to archive data when a user leaves. In this second case, archiving and restoring data over many users could tie up a great deal of db’s bandwidth.

 

— Fanciful solution I: Give each user his/her own independent website. 

 

— Linux containers might make this possible.

 

— Transactions are not an issue.

 

— A small Mnesia db or even dets db per user would suffice.

 

— Linux containers could be deployed over multiple hardware or VPS nodes, lowering hardware failure risk and making it relatively easy to scale from just a few users to many many.

 

— LXD provides tools that would make it relatively easy to induct new users and replicate user sites.

 

— But would it be economically feasible?

 

— Fanciful solution II:  Multiple BEAM nodes per Linux node.

 

— Hmmm... might have to functionally replicate tools that LXD already provides.

 

— But it might be more resource efficient.

 

— Can it even work?

 

And so, my head spins for lack of experience.

 

And so, I’d much appreciate insight and guidance from experienced hands.

 

All the best,

 

Lloyd

 

Sent from my iPad


On May 14, 2019, at 5:36 PM, Chinedu Okonkwo <[hidden email]> wrote:

That is pretty much what erlang release is.

 

I think its one of the reason docker/containers didn't catch on in the erlang community. The major problem you may face running it especially in distributed erlang mode may be name collusion in empd

 

On Tue, May 14, 2019 at 9:13 PM Lloyd R. Prentice <[hidden email]> wrote:

Hello,

My Mickey-the-Dunce ignorance and curiosity has led me to explore Linux containers. Very cool. Then, a recent conversation with Oliver Korpilla tweaked my wonder nerve:

Is it possible/practical to run multiple BEAM nodes on a single Linux node?

Assume each BEAM node supports an identical web application with a Mnesia backend serving a set of CRUD services— each for a different client.

Has anyone ever done anything along this line?

Best wishes,

LRP

Sent from my iPad
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

 


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Multiple BEAM nodes on one Linux node — a thought experiment

Roger Lipscombe-2
In reply to this post by Lloyd R. Prentice-2
On Tue, 14 May 2019 at 22:12, Lloyd R. Prentice <[hidden email]> wrote:
> Is it possible/practical to run multiple BEAM nodes on a single Linux node?

Yes. We do this. IoT devices connect to one node; their backend
agents/actors run in a different node on the same machine, connected
by Erlang distribution. We do this for isolation (partly because
there's a large and complex NIF in the second node), plus for
historical reasons -- which we periodically revisit and decide "nah,
not worth the effort".

> Assume each BEAM node supports an identical web application with a Mnesia backend serving a set of CRUD services— each for a different client.

However: I wouldn't bother doing this. It won't scale beyond, say, a
few dozen nodes -- depending on the size of the host -- and you'd be
wasting a bunch of capacity on the host (remember that Unix process
overhead is several orders of magnitude more than Erlang process
overhead, and both are dwarfed by container overhead).

You'd be better served by looking at making a single node multitenant.
You could do this by using a separate mnesia table per client, or use
prefixed keys within whatever tables you're planning to use. Or use a
relational model, where each entity has a related owner.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions