Best approach to creating a process/connection pool
Locked
Best approach to creating a process/connection pool
 
|
I'm fairly new to Erlang and I am trying to write a process pool which
essentially is a connection pool to a database. I have designed the application like this: The main pool gen_server has a queue of PIDs of gen_fsm's. Each gen_fsm calls spawn_links a 3rd party gen_server and carries the state of the connection. So the 3rd party gen_server and the gen_fsm should always live and die together. I currently have set it up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd party gen_server, but I'm wondering how this factors into the supervision tree. I currently have two supervisors: Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2 Sup2 -> simple_one_for_one -> 1 child: gen_fsm So my question is: is it ok to use spawn_link instead of somehow trying to put the 3rd party gen_server into the supervision tree? If it should go into the supervision tree, how would I organize the tree then such that the gen_fsm and 3rd party gen_server are reliant on each other? Thanks for any help! _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Best approach to creating a process/connection pool
|
Take a look at the sources for http://zotonic.com/ as they use a
decent connection pool. On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote: > I'm fairly new to Erlang and I am trying to write a process pool which > essentially is a connection pool to a database. I have designed the > application like this: The main pool gen_server has a queue of PIDs > of gen_fsm's. Each gen_fsm calls spawn_links a 3rd party gen_server and carries > the state of the connection. So the 3rd party gen_server and the > gen_fsm should always live and die together. I currently have set it > up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd > party gen_server, but I'm wondering how this factors into the > supervision tree. I currently have two supervisors: > > Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2 > Sup2 -> simple_one_for_one -> 1 child: gen_fsm > > So my question is: is it ok to use spawn_link instead of somehow > trying to put the 3rd party gen_server into the supervision tree? If > it should go into the supervision tree, how would I organize the tree > then such that the gen_fsm and 3rd party gen_server are reliant on > each other? > > Thanks for any help! > _______________________________________________ > erlang-questions mailing list > [hidden email] > http://erlang.org/mailman/listinfo/erlang-questions > erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Best approach to creating a process/connection pool
|
In reply to this post by Andrew Berman
You may want to take a look at Poolboy, it looks pretty good: https://github.com/devinus/poolboy
It includes an example of pgsql connection pool. Anton On Thu, Aug 4, 2011 at 2:01 AM, Andrew Berman <[hidden email]> wrote: I'm fairly new to Erlang and I am trying to write a process pool which _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Best approach to creating a process/connection pool
|
|
In reply to this post by Tim Watson-5
Thanks Tim. So it looks like they are using erlang:monitor and
demonitor, so is that the best approach in an OTP application? How does it figure into the supervision tree? On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote: > Take a look at the sources for http://zotonic.com/ as they use a > decent connection pool. > > On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote: >> I'm fairly new to Erlang and I am trying to write a process pool which >> essentially is a connection pool to a database. I have designed the >> application like this: The main pool gen_server has a queue of PIDs >> of gen_fsm's. Each gen_fsm calls spawn_links a 3rd party gen_server and carries >> the state of the connection. So the 3rd party gen_server and the >> gen_fsm should always live and die together. I currently have set it >> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd >> party gen_server, but I'm wondering how this factors into the >> supervision tree. I currently have two supervisors: >> >> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2 >> Sup2 -> simple_one_for_one -> 1 child: gen_fsm >> >> So my question is: is it ok to use spawn_link instead of somehow >> trying to put the 3rd party gen_server into the supervision tree? If >> it should go into the supervision tree, how would I organize the tree >> then such that the gen_fsm and 3rd party gen_server are reliant on >> each other? >> >> Thanks for any help! >> _______________________________________________ >> erlang-questions mailing list >> [hidden email] >> http://erlang.org/mailman/listinfo/erlang-questions >> > erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Best approach to creating a process/connection pool
|
Using OTP supervisor tree is intended only for control of restarting
application parts. If you want anything else you have to use links or monitors unless you write your own supervisor which is not recommended way. On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> wrote: > Thanks Tim. So it looks like they are using erlang:monitor and > demonitor, so is that the best approach in an OTP application? How > does it figure into the supervision tree? > > > > On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote: >> Take a look at the sources for http://zotonic.com/ as they use a >> decent connection pool. >> >> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote: >>> I'm fairly new to Erlang and I am trying to write a process pool which >>> essentially is a connection pool to a database. I have designed the >>> application like this: The main pool gen_server has a queue of PIDs >>> of gen_fsm's. Each gen_fsm calls spawn_links a 3rd party gen_server and carries >>> the state of the connection. So the 3rd party gen_server and the >>> gen_fsm should always live and die together. I currently have set it >>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd >>> party gen_server, but I'm wondering how this factors into the >>> supervision tree. I currently have two supervisors: >>> >>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2 >>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm >>> >>> So my question is: is it ok to use spawn_link instead of somehow >>> trying to put the 3rd party gen_server into the supervision tree? If >>> it should go into the supervision tree, how would I organize the tree >>> then such that the gen_fsm and 3rd party gen_server are reliant on >>> each other? >>> >>> Thanks for any help! >>> _______________________________________________ >>> erlang-questions mailing list >>> [hidden email] >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> > _______________________________________________ > erlang-questions mailing list > [hidden email] > http://erlang.org/mailman/listinfo/erlang-questions > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss. Be a data hero! Try GoodData now for free: www.gooddata.com _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Best approach to creating a process/connection pool
|
Technically, the OTP supervision tree is also used for application upgrades (appups & relups). When sending messages to do the broad reloading of code, the code in charge of that just traverses the VM's supervision trees asking the supervisors what kind of modules they have and whether they're workers and supervisors (that's why the child specs contain details about workers, supervisors, [Modules] (and dynamic sending special messages to figure out the same info).
If you want to use appups/relups, you have to use the OTP supervision trees otherwise you will have processes left behind to be purged rather than upgraded. -- Fred Hébert http://www.erlang-solutions.com On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote: > Using OTP supervisor tree is intended only for control of restarting > application parts. If you want anything else you have to use links or > monitors unless you write your own supervisor which is not recommended > way. > > On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> wrote: >> Thanks Tim. So it looks like they are using erlang:monitor and >> demonitor, so is that the best approach in an OTP application? How >> does it figure into the supervision tree? >> >> >> >> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote: >>> Take a look at the sources for http://zotonic.com/ as they use a >>> decent connection pool. >>> >>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote: >>>> I'm fairly new to Erlang and I am trying to write a process pool which >>>> essentially is a connection pool to a database. I have designed the >>>> application like this: The main pool gen_server has a queue of PIDs >>>> of gen_fsm's. Each gen_fsm calls spawn_links a 3rd party gen_server and carries >>>> the state of the connection. So the 3rd party gen_server and the >>>> gen_fsm should always live and die together. I currently have set it >>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd >>>> party gen_server, but I'm wondering how this factors into the >>>> supervision tree. I currently have two supervisors: >>>> >>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2 >>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm >>>> >>>> So my question is: is it ok to use spawn_link instead of somehow >>>> trying to put the 3rd party gen_server into the supervision tree? If >>>> it should go into the supervision tree, how would I organize the tree >>>> then such that the gen_fsm and 3rd party gen_server are reliant on >>>> each other? >>>> >>>> Thanks for any help! >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> [hidden email] >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>> >> _______________________________________________ >> erlang-questions mailing list >> [hidden email] >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > --Hynek (Pichi) Vychodil > > Analyze your data in minutes. Share your insights instantly. Thrill > your boss. Be a data hero! > Try GoodData now for free: www.gooddata.com > _______________________________________________ > erlang-questions mailing list > [hidden email] > http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Best approach to creating a process/connection pool
|
|
Awesome! This makes sense. Thank you for the clarification on this.
2011/8/4 Frédéric Trottier-Hébert <[hidden email]>: > Technically, the OTP supervision tree is also used for application upgrades (appups & relups). When sending messages to do the broad reloading of code, the code in charge of that just traverses the VM's supervision trees asking the supervisors what kind of modules they have and whether they're workers and supervisors (that's why the child specs contain details about workers, supervisors, [Modules] (and dynamic sending special messages to figure out the same info). > > If you want to use appups/relups, you have to use the OTP supervision trees otherwise you will have processes left behind to be purged rather than upgraded. > > -- > Fred Hébert > http://www.erlang-solutions.com > > > > On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote: > >> Using OTP supervisor tree is intended only for control of restarting >> application parts. If you want anything else you have to use links or >> monitors unless you write your own supervisor which is not recommended >> way. >> >> On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> wrote: >>> Thanks Tim. So it looks like they are using erlang:monitor and >>> demonitor, so is that the best approach in an OTP application? How >>> does it figure into the supervision tree? >>> >>> >>> >>> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote: >>>> Take a look at the sources for http://zotonic.com/ as they use a >>>> decent connection pool. >>>> >>>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote: >>>>> I'm fairly new to Erlang and I am trying to write a process pool which >>>>> essentially is a connection pool to a database. I have designed the >>>>> application like this: The main pool gen_server has a queue of PIDs >>>>> of gen_fsm's. Each gen_fsm calls spawn_links a 3rd party gen_server and carries >>>>> the state of the connection. So the 3rd party gen_server and the >>>>> gen_fsm should always live and die together. I currently have set it >>>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd >>>>> party gen_server, but I'm wondering how this factors into the >>>>> supervision tree. I currently have two supervisors: >>>>> >>>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2 >>>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm >>>>> >>>>> So my question is: is it ok to use spawn_link instead of somehow >>>>> trying to put the 3rd party gen_server into the supervision tree? If >>>>> it should go into the supervision tree, how would I organize the tree >>>>> then such that the gen_fsm and 3rd party gen_server are reliant on >>>>> each other? >>>>> >>>>> Thanks for any help! >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> [hidden email] >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> [hidden email] >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> -- >> --Hynek (Pichi) Vychodil >> >> Analyze your data in minutes. Share your insights instantly. Thrill >> your boss. Be a data hero! >> Try GoodData now for free: www.gooddata.com >> _______________________________________________ >> erlang-questions mailing list >> [hidden email] >> http://erlang.org/mailman/listinfo/erlang-questions > > erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Best approach to creating a process/connection pool
|
If the things you are pooling are gen_servers, gen_server_pool may work for you (https://github.com/joelpm/gen_server_pool). That's a slight modification on a version I've used in production to pool thrift and protobuff conns to back-ends. To start a pool you'd simply do gen_server_pool:startlink( ..., GenServerPoolArgs). instead of doing gen_server:start_link(...). You would use the returned PID in the same way. I haven't had time to test that version, though, so YMMV.
Joel
2011/8/4 Andrew Berman <[hidden email]> Awesome! This makes sense. Thank you for the clarification on this. _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
|
The Jobs load regulation system supports a pool concept. Actually, it allows you to combine concepts in a number of ways to tune characteristics. Here is one example, with a 'producer pool' and a passive queue. The producer pool is defined with a fun that is called to top up the pool. The producers are free to do pretty much what they want, and jobs will monitor them to ensure they are replaced if they die. In this case, the producers fetch jobs from a passive queue. In Jobs, that's a queue where you have to explicitly dequeue them. Otherwise, you typically tell jobs at which rate it should draw from the queue. Rates and pool sizes can be made to adapt automatically to load, using load sampler plugins. Configuration/re-configuration can also be done at runtime through the Jobs API. BR, Ulf W On 8 Aug 2011, at 21:44, Joel Meyer wrote: If the things you are pooling are gen_servers, gen_server_pool may work for you (https://github.com/joelpm/gen_server_pool). That's a slight modification on a version I've used in production to pool thrift and protobuff conns to back-ends. To start a pool you'd simply do gen_server_pool:startlink( ..., GenServerPoolArgs). instead of doing gen_server:start_link(...). You would use the returned PID in the same way. I haven't had time to test that version, though, so YMMV. _______________________________________________ erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
Re: Best approach to creating a process/connection pool
|
|
Thanks Ulf. That seems to be way more complicated than what i
need....I think. I actually created a pool application here: https://github.com/aberman/pooly Every pooling app I found or was suggested didn't really take into account certain factors that most connection pools use, such as idle timeout, maximum age of a connection, acquire increment, initial number of connections to start, etc. I wrote my pool with these factors in mind and made them configurable in an external file. It was written for Riak, but it can easily be adapted to support any gen_server. Whether I did things properly, I don't know, but it seems to work how I want it. Thanks for your help, Andrew On Mon, Aug 8, 2011 at 11:32 PM, Ulf Wiger <[hidden email]> wrote: > > The Jobs load regulation system supports a pool concept. Actually, it allows > you to combine concepts in a number of ways to tune characteristics. > Here is one example, with a 'producer pool' and a passive queue. The > producer pool is defined with a fun that is called to top up the pool. The > producers are free to do pretty much what they want, and jobs will monitor > them to ensure they are replaced if they die. > https://gist.github.com/778319 > In this case, the producers fetch jobs from a passive queue. In Jobs, that's > a queue where you have to explicitly dequeue them. Otherwise, you typically > tell jobs at which rate it should draw from the queue. > Rates and pool sizes can be made to adapt automatically to load, using load > sampler plugins. Configuration/re-configuration can also be done at runtime > through the Jobs API. > http://github.com/esl/jobs > BR, > Ulf W > > On 8 Aug 2011, at 21:44, Joel Meyer wrote: > > If the things you are pooling are gen_servers, gen_server_pool may work for > you (https://github.com/joelpm/gen_server_pool). That's a slight > modification on a version I've used in production to pool thrift and > protobuff conns to back-ends. To start a pool you'd simply do > gen_server_pool:startlink( ..., GenServerPoolArgs). instead of doing > gen_server:start_link(...). You would use the returned PID in the same way. > I haven't had time to test that version, though, so YMMV. > Joel > > 2011/8/4 Andrew Berman <[hidden email]> >> >> Awesome! This makes sense. Thank you for the clarification on this. >> >> 2011/8/4 Frédéric Trottier-Hébert <[hidden email]>: >> > Technically, the OTP supervision tree is also used for application >> > upgrades (appups & relups). When sending messages to do the broad reloading >> > of code, the code in charge of that just traverses the VM's supervision >> > trees asking the supervisors what kind of modules they have and whether >> > they're workers and supervisors (that's why the child specs contain details >> > about workers, supervisors, [Modules] (and dynamic sending special messages >> > to figure out the same info). >> > >> > If you want to use appups/relups, you have to use the OTP supervision >> > trees otherwise you will have processes left behind to be purged rather than >> > upgraded. >> > >> > -- >> > Fred Hébert >> > http://www.erlang-solutions.com >> > >> > >> > >> > On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote: >> > >> >> Using OTP supervisor tree is intended only for control of restarting >> >> application parts. If you want anything else you have to use links or >> >> monitors unless you write your own supervisor which is not recommended >> >> way. >> >> >> >> On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> >> >> wrote: >> >>> Thanks Tim. So it looks like they are using erlang:monitor and >> >>> demonitor, so is that the best approach in an OTP application? How >> >>> does it figure into the supervision tree? >> >>> >> >>> >> >>> >> >>> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> >> >>> wrote: >> >>>> Take a look at the sources for http://zotonic.com/ as they use a >> >>>> decent connection pool. >> >>>> >> >>>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote: >> >>>>> I'm fairly new to Erlang and I am trying to write a process pool >> >>>>> which >> >>>>> essentially is a connection pool to a database. I have designed the >> >>>>> application like this: The main pool gen_server has a queue of PIDs >> >>>>> of gen_fsm's. Each gen_fsm calls spawn_links a 3rd party gen_server >> >>>>> and carries >> >>>>> the state of the connection. So the 3rd party gen_server and the >> >>>>> gen_fsm should always live and die together. I currently have set >> >>>>> it >> >>>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd >> >>>>> party gen_server, but I'm wondering how this factors into the >> >>>>> supervision tree. I currently have two supervisors: >> >>>>> >> >>>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2 >> >>>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm >> >>>>> >> >>>>> So my question is: is it ok to use spawn_link instead of somehow >> >>>>> trying to put the 3rd party gen_server into the supervision tree? >> >>>>> If >> >>>>> it should go into the supervision tree, how would I organize the >> >>>>> tree >> >>>>> then such that the gen_fsm and 3rd party gen_server are reliant on >> >>>>> each other? >> >>>>> >> >>>>> Thanks for any help! >> >>>>> _______________________________________________ >> >>>>> erlang-questions mailing list >> >>>>> [hidden email] >> >>>>> http://erlang.org/mailman/listinfo/erlang-questions >> >>>>> >> >>>> >> >>> _______________________________________________ >> >>> erlang-questions mailing list >> >>> [hidden email] >> >>> http://erlang.org/mailman/listinfo/erlang-questions >> >>> >> >> >> >> >> >> >> >> -- >> >> --Hynek (Pichi) Vychodil >> >> >> >> Analyze your data in minutes. Share your insights instantly. Thrill >> >> your boss. Be a data hero! >> >> Try GoodData now for free: www.gooddata.com >> >> _______________________________________________ >> >> erlang-questions mailing list >> >> [hidden email] >> >> http://erlang.org/mailman/listinfo/erlang-questions >> > >> > >> _______________________________________________ >> erlang-questions mailing list >> [hidden email] >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > [hidden email] > http://erlang.org/mailman/listinfo/erlang-questions > > Ulf Wiger, CTO, Erlang Solutions, Ltd. > http://erlang-solutions.com > > > erlang-questions mailing list [hidden email] http://erlang.org/mailman/listinfo/erlang-questions |
«
Return to Erlang Questions
|
1 view|%1 views
Loading...
| Powered by Nabble | Edit this page |