Best approach to creating a process/connection pool

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Best approach to creating a process/connection pool

Andrew Berman
I'm fairly new to Erlang and I am trying to write a process pool which
essentially is a connection pool to a database.  I have designed the
application like this:  The main pool gen_server has a queue of PIDs
of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
the state of the connection.  So the 3rd party gen_server and the
gen_fsm should always live and die together.  I currently have set it
up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
party gen_server, but I'm wondering how this factors into the
supervision tree.  I currently have two supervisors:

Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
Sup2 -> simple_one_for_one -> 1 child: gen_fsm

So my question is: is it ok to use spawn_link instead of somehow
trying to put the 3rd party gen_server into the supervision tree?  If
it should go into the supervision tree, how would I organize the tree
then such that the gen_fsm and 3rd party gen_server are reliant on
each other?

Thanks for any help!
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Tim Watson-5
Take a look at the sources for http://zotonic.com/ as they use a
decent connection pool.

On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote:

> I'm fairly new to Erlang and I am trying to write a process pool which
> essentially is a connection pool to a database.  I have designed the
> application like this:  The main pool gen_server has a queue of PIDs
> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
> the state of the connection.  So the 3rd party gen_server and the
> gen_fsm should always live and die together.  I currently have set it
> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
> party gen_server, but I'm wondering how this factors into the
> supervision tree.  I currently have two supervisors:
>
> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>
> So my question is: is it ok to use spawn_link instead of somehow
> trying to put the 3rd party gen_server into the supervision tree?  If
> it should go into the supervision tree, how would I organize the tree
> then such that the gen_fsm and 3rd party gen_server are reliant on
> each other?
>
> Thanks for any help!
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Anton Lavrik
In reply to this post by Andrew Berman
You may want to take a look at Poolboy, it looks pretty good: https://github.com/devinus/poolboy

It includes an example of pgsql connection pool.

Anton

On Thu, Aug 4, 2011 at 2:01 AM, Andrew Berman <[hidden email]> wrote:
I'm fairly new to Erlang and I am trying to write a process pool which
essentially is a connection pool to a database.  I have designed the
application like this:  The main pool gen_server has a queue of PIDs
of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
the state of the connection.  So the 3rd party gen_server and the
gen_fsm should always live and die together.  I currently have set it
up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
party gen_server, but I'm wondering how this factors into the
supervision tree.  I currently have two supervisors:

Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
Sup2 -> simple_one_for_one -> 1 child: gen_fsm

So my question is: is it ok to use spawn_link instead of somehow
trying to put the 3rd party gen_server into the supervision tree?  If
it should go into the supervision tree, how would I organize the tree
then such that the gen_fsm and 3rd party gen_server are reliant on
each other?

Thanks for any help!
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Andrew Berman
In reply to this post by Tim Watson-5
Thanks Tim.  So it looks like they are using erlang:monitor and
demonitor, so is that the best approach in an OTP application?  How
does it figure into the supervision tree?



On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote:

> Take a look at the sources for http://zotonic.com/ as they use a
> decent connection pool.
>
> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote:
>> I'm fairly new to Erlang and I am trying to write a process pool which
>> essentially is a connection pool to a database.  I have designed the
>> application like this:  The main pool gen_server has a queue of PIDs
>> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
>> the state of the connection.  So the 3rd party gen_server and the
>> gen_fsm should always live and die together.  I currently have set it
>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
>> party gen_server, but I'm wondering how this factors into the
>> supervision tree.  I currently have two supervisors:
>>
>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>>
>> So my question is: is it ok to use spawn_link instead of somehow
>> trying to put the 3rd party gen_server into the supervision tree?  If
>> it should go into the supervision tree, how would I organize the tree
>> then such that the gen_fsm and 3rd party gen_server are reliant on
>> each other?
>>
>> Thanks for any help!
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Hynek Vychodil-2
Using OTP supervisor tree is intended only for control of restarting
application parts. If you want anything else you have to use links or
monitors unless you write your own supervisor which is not recommended
way.

On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> wrote:

> Thanks Tim.  So it looks like they are using erlang:monitor and
> demonitor, so is that the best approach in an OTP application?  How
> does it figure into the supervision tree?
>
>
>
> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote:
>> Take a look at the sources for http://zotonic.com/ as they use a
>> decent connection pool.
>>
>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote:
>>> I'm fairly new to Erlang and I am trying to write a process pool which
>>> essentially is a connection pool to a database.  I have designed the
>>> application like this:  The main pool gen_server has a queue of PIDs
>>> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
>>> the state of the connection.  So the 3rd party gen_server and the
>>> gen_fsm should always live and die together.  I currently have set it
>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
>>> party gen_server, but I'm wondering how this factors into the
>>> supervision tree.  I currently have two supervisors:
>>>
>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>>>
>>> So my question is: is it ok to use spawn_link instead of somehow
>>> trying to put the 3rd party gen_server into the supervision tree?  If
>>> it should go into the supervision tree, how would I organize the tree
>>> then such that the gen_fsm and 3rd party gen_server are reliant on
>>> each other?
>>>
>>> Thanks for any help!
>>> _______________________________________________
>>> erlang-questions mailing list
>>> [hidden email]
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>



--
--Hynek (Pichi) Vychodil

Analyze your data in minutes. Share your insights instantly. Thrill
your boss.  Be a data hero!
Try GoodData now for free: www.gooddata.com
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Frédéric Trottier-Hébert
Technically, the OTP supervision tree is also used for application upgrades (appups & relups). When sending messages to do the broad reloading of code, the code in charge of that just traverses the VM's supervision trees asking the supervisors what kind of modules they have and whether they're workers and supervisors (that's why the child specs contain details about workers, supervisors, [Modules] (and dynamic sending special messages to figure out the same info).

If you want to use appups/relups, you have to use the OTP supervision trees otherwise you will have processes left behind to be purged rather than upgraded.

--
Fred Hébert
http://www.erlang-solutions.com



On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote:

> Using OTP supervisor tree is intended only for control of restarting
> application parts. If you want anything else you have to use links or
> monitors unless you write your own supervisor which is not recommended
> way.
>
> On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> wrote:
>> Thanks Tim.  So it looks like they are using erlang:monitor and
>> demonitor, so is that the best approach in an OTP application?  How
>> does it figure into the supervision tree?
>>
>>
>>
>> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote:
>>> Take a look at the sources for http://zotonic.com/ as they use a
>>> decent connection pool.
>>>
>>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote:
>>>> I'm fairly new to Erlang and I am trying to write a process pool which
>>>> essentially is a connection pool to a database.  I have designed the
>>>> application like this:  The main pool gen_server has a queue of PIDs
>>>> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
>>>> the state of the connection.  So the 3rd party gen_server and the
>>>> gen_fsm should always live and die together.  I currently have set it
>>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
>>>> party gen_server, but I'm wondering how this factors into the
>>>> supervision tree.  I currently have two supervisors:
>>>>
>>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
>>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>>>>
>>>> So my question is: is it ok to use spawn_link instead of somehow
>>>> trying to put the 3rd party gen_server into the supervision tree?  If
>>>> it should go into the supervision tree, how would I organize the tree
>>>> then such that the gen_fsm and 3rd party gen_server are reliant on
>>>> each other?
>>>>
>>>> Thanks for any help!
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> [hidden email]
>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
>
>
> --
> --Hynek (Pichi) Vychodil
>
> Analyze your data in minutes. Share your insights instantly. Thrill
> your boss.  Be a data hero!
> Try GoodData now for free: www.gooddata.com
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Andrew Berman
Awesome!  This makes sense.  Thank you for the clarification on this.

2011/8/4 Frédéric Trottier-Hébert <[hidden email]>:

> Technically, the OTP supervision tree is also used for application upgrades (appups & relups). When sending messages to do the broad reloading of code, the code in charge of that just traverses the VM's supervision trees asking the supervisors what kind of modules they have and whether they're workers and supervisors (that's why the child specs contain details about workers, supervisors, [Modules] (and dynamic sending special messages to figure out the same info).
>
> If you want to use appups/relups, you have to use the OTP supervision trees otherwise you will have processes left behind to be purged rather than upgraded.
>
> --
> Fred Hébert
> http://www.erlang-solutions.com
>
>
>
> On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote:
>
>> Using OTP supervisor tree is intended only for control of restarting
>> application parts. If you want anything else you have to use links or
>> monitors unless you write your own supervisor which is not recommended
>> way.
>>
>> On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> wrote:
>>> Thanks Tim.  So it looks like they are using erlang:monitor and
>>> demonitor, so is that the best approach in an OTP application?  How
>>> does it figure into the supervision tree?
>>>
>>>
>>>
>>> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote:
>>>> Take a look at the sources for http://zotonic.com/ as they use a
>>>> decent connection pool.
>>>>
>>>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote:
>>>>> I'm fairly new to Erlang and I am trying to write a process pool which
>>>>> essentially is a connection pool to a database.  I have designed the
>>>>> application like this:  The main pool gen_server has a queue of PIDs
>>>>> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
>>>>> the state of the connection.  So the 3rd party gen_server and the
>>>>> gen_fsm should always live and die together.  I currently have set it
>>>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
>>>>> party gen_server, but I'm wondering how this factors into the
>>>>> supervision tree.  I currently have two supervisors:
>>>>>
>>>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
>>>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>>>>>
>>>>> So my question is: is it ok to use spawn_link instead of somehow
>>>>> trying to put the 3rd party gen_server into the supervision tree?  If
>>>>> it should go into the supervision tree, how would I organize the tree
>>>>> then such that the gen_fsm and 3rd party gen_server are reliant on
>>>>> each other?
>>>>>
>>>>> Thanks for any help!
>>>>> _______________________________________________
>>>>> erlang-questions mailing list
>>>>> [hidden email]
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>
>>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> [hidden email]
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
>>
>>
>> --
>> --Hynek (Pichi) Vychodil
>>
>> Analyze your data in minutes. Share your insights instantly. Thrill
>> your boss.  Be a data hero!
>> Try GoodData now for free: www.gooddata.com
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Joel Meyer
If the things you are pooling are gen_servers, gen_server_pool may work for you (https://github.com/joelpm/gen_server_pool). That's a slight modification on a version I've used in production to pool thrift and protobuff conns to back-ends. To start a pool you'd simply do gen_server_pool:startlink( ..., GenServerPoolArgs). instead of doing gen_server:start_link(...). You would use the returned PID in the same way. I haven't had time to test that version, though, so YMMV.

Joel

2011/8/4 Andrew Berman <[hidden email]>
Awesome!  This makes sense.  Thank you for the clarification on this.

2011/8/4 Frédéric Trottier-Hébert <[hidden email]>:
> Technically, the OTP supervision tree is also used for application upgrades (appups & relups). When sending messages to do the broad reloading of code, the code in charge of that just traverses the VM's supervision trees asking the supervisors what kind of modules they have and whether they're workers and supervisors (that's why the child specs contain details about workers, supervisors, [Modules] (and dynamic sending special messages to figure out the same info).
>
> If you want to use appups/relups, you have to use the OTP supervision trees otherwise you will have processes left behind to be purged rather than upgraded.
>
> --
> Fred Hébert
> http://www.erlang-solutions.com
>
>
>
> On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote:
>
>> Using OTP supervisor tree is intended only for control of restarting
>> application parts. If you want anything else you have to use links or
>> monitors unless you write your own supervisor which is not recommended
>> way.
>>
>> On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> wrote:
>>> Thanks Tim.  So it looks like they are using erlang:monitor and
>>> demonitor, so is that the best approach in an OTP application?  How
>>> does it figure into the supervision tree?
>>>
>>>
>>>
>>> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote:
>>>> Take a look at the sources for http://zotonic.com/ as they use a
>>>> decent connection pool.
>>>>
>>>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote:
>>>>> I'm fairly new to Erlang and I am trying to write a process pool which
>>>>> essentially is a connection pool to a database.  I have designed the
>>>>> application like this:  The main pool gen_server has a queue of PIDs
>>>>> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
>>>>> the state of the connection.  So the 3rd party gen_server and the
>>>>> gen_fsm should always live and die together.  I currently have set it
>>>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
>>>>> party gen_server, but I'm wondering how this factors into the
>>>>> supervision tree.  I currently have two supervisors:
>>>>>
>>>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
>>>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>>>>>
>>>>> So my question is: is it ok to use spawn_link instead of somehow
>>>>> trying to put the 3rd party gen_server into the supervision tree?  If
>>>>> it should go into the supervision tree, how would I organize the tree
>>>>> then such that the gen_fsm and 3rd party gen_server are reliant on
>>>>> each other?
>>>>>
>>>>> Thanks for any help!
>>>>> _______________________________________________
>>>>> erlang-questions mailing list
>>>>> [hidden email]
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>
>>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> [hidden email]
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
>>
>>
>> --
>> --Hynek (Pichi) Vychodil
>>
>> Analyze your data in minutes. Share your insights instantly. Thrill
>> your boss.  Be a data hero!
>> Try GoodData now for free: www.gooddata.com
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Ulf Wiger

The Jobs load regulation system supports a pool concept. Actually, it allows you to combine concepts in a number of ways to tune characteristics.

Here is one example, with a 'producer pool' and a passive queue. The producer pool is defined with a fun that is called to top up the pool. The producers are free to do pretty much what they want, and jobs will monitor them to ensure they are replaced if they die.


In this case, the producers fetch jobs from a passive queue. In Jobs, that's a queue where you have to explicitly dequeue them. Otherwise, you typically tell jobs at which rate it should draw from the queue.

Rates and pool sizes can be made to adapt automatically to load, using load sampler plugins. Configuration/re-configuration can also be done at runtime through the Jobs API.


BR,
Ulf W


On 8 Aug 2011, at 21:44, Joel Meyer wrote:

If the things you are pooling are gen_servers, gen_server_pool may work for you (https://github.com/joelpm/gen_server_pool). That's a slight modification on a version I've used in production to pool thrift and protobuff conns to back-ends. To start a pool you'd simply do gen_server_pool:startlink( ..., GenServerPoolArgs). instead of doing gen_server:start_link(...). You would use the returned PID in the same way. I haven't had time to test that version, though, so YMMV.

Joel

2011/8/4 Andrew Berman <[hidden email]>
Awesome!  This makes sense.  Thank you for the clarification on this.

2011/8/4 Frédéric Trottier-Hébert <[hidden email]>:
> Technically, the OTP supervision tree is also used for application upgrades (appups & relups). When sending messages to do the broad reloading of code, the code in charge of that just traverses the VM's supervision trees asking the supervisors what kind of modules they have and whether they're workers and supervisors (that's why the child specs contain details about workers, supervisors, [Modules] (and dynamic sending special messages to figure out the same info).
>
> If you want to use appups/relups, you have to use the OTP supervision trees otherwise you will have processes left behind to be purged rather than upgraded.
>
> --
> Fred Hébert
> http://www.erlang-solutions.com
>
>
>
> On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote:
>
>> Using OTP supervisor tree is intended only for control of restarting
>> application parts. If you want anything else you have to use links or
>> monitors unless you write your own supervisor which is not recommended
>> way.
>>
>> On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]> wrote:
>>> Thanks Tim.  So it looks like they are using erlang:monitor and
>>> demonitor, so is that the best approach in an OTP application?  How
>>> does it figure into the supervision tree?
>>>
>>>
>>>
>>> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]> wrote:
>>>> Take a look at the sources for http://zotonic.com/ as they use a
>>>> decent connection pool.
>>>>
>>>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote:
>>>>> I'm fairly new to Erlang and I am trying to write a process pool which
>>>>> essentially is a connection pool to a database.  I have designed the
>>>>> application like this:  The main pool gen_server has a queue of PIDs
>>>>> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server and carries
>>>>> the state of the connection.  So the 3rd party gen_server and the
>>>>> gen_fsm should always live and die together.  I currently have set it
>>>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
>>>>> party gen_server, but I'm wondering how this factors into the
>>>>> supervision tree.  I currently have two supervisors:
>>>>>
>>>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
>>>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>>>>>
>>>>> So my question is: is it ok to use spawn_link instead of somehow
>>>>> trying to put the 3rd party gen_server into the supervision tree?  If
>>>>> it should go into the supervision tree, how would I organize the tree
>>>>> then such that the gen_fsm and 3rd party gen_server are reliant on
>>>>> each other?
>>>>>
>>>>> Thanks for any help!
>>>>> _______________________________________________
>>>>> erlang-questions mailing list
>>>>> [hidden email]
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>
>>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> [hidden email]
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
>>
>>
>> --
>> --Hynek (Pichi) Vychodil
>>
>> Analyze your data in minutes. Share your insights instantly. Thrill
>> your boss.  Be a data hero!
>> Try GoodData now for free: www.gooddata.com
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

Ulf Wiger, CTO, Erlang Solutions, Ltd.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Best approach to creating a process/connection pool

Andrew Berman
Thanks Ulf.  That seems to be way more complicated than what i
need....I think.  I actually created a pool application here:
https://github.com/aberman/pooly

Every pooling app I found or was suggested didn't really take into
account certain factors that most connection pools use, such as idle
timeout, maximum age of a connection, acquire increment, initial
number of connections to start, etc.  I wrote my pool with these
factors in mind and made them configurable in an external file.  It
was written for Riak, but it can easily be adapted to support any
gen_server.  Whether I did things properly, I don't know, but it seems
to work how I want it.

Thanks for your help,

Andrew

On Mon, Aug 8, 2011 at 11:32 PM, Ulf Wiger
<[hidden email]> wrote:

>
> The Jobs load regulation system supports a pool concept. Actually, it allows
> you to combine concepts in a number of ways to tune characteristics.
> Here is one example, with a 'producer pool' and a passive queue. The
> producer pool is defined with a fun that is called to top up the pool. The
> producers are free to do pretty much what they want, and jobs will monitor
> them to ensure they are replaced if they die.
> https://gist.github.com/778319
> In this case, the producers fetch jobs from a passive queue. In Jobs, that's
> a queue where you have to explicitly dequeue them. Otherwise, you typically
> tell jobs at which rate it should draw from the queue.
> Rates and pool sizes can be made to adapt automatically to load, using load
> sampler plugins. Configuration/re-configuration can also be done at runtime
> through the Jobs API.
> http://github.com/esl/jobs
> BR,
> Ulf W
>
> On 8 Aug 2011, at 21:44, Joel Meyer wrote:
>
> If the things you are pooling are gen_servers, gen_server_pool may work for
> you (https://github.com/joelpm/gen_server_pool). That's a slight
> modification on a version I've used in production to pool thrift and
> protobuff conns to back-ends. To start a pool you'd simply do
> gen_server_pool:startlink( ..., GenServerPoolArgs). instead of doing
> gen_server:start_link(...). You would use the returned PID in the same way.
> I haven't had time to test that version, though, so YMMV.
> Joel
>
> 2011/8/4 Andrew Berman <[hidden email]>
>>
>> Awesome!  This makes sense.  Thank you for the clarification on this.
>>
>> 2011/8/4 Frédéric Trottier-Hébert <[hidden email]>:
>> > Technically, the OTP supervision tree is also used for application
>> > upgrades (appups & relups). When sending messages to do the broad reloading
>> > of code, the code in charge of that just traverses the VM's supervision
>> > trees asking the supervisors what kind of modules they have and whether
>> > they're workers and supervisors (that's why the child specs contain details
>> > about workers, supervisors, [Modules] (and dynamic sending special messages
>> > to figure out the same info).
>> >
>> > If you want to use appups/relups, you have to use the OTP supervision
>> > trees otherwise you will have processes left behind to be purged rather than
>> > upgraded.
>> >
>> > --
>> > Fred Hébert
>> > http://www.erlang-solutions.com
>> >
>> >
>> >
>> > On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote:
>> >
>> >> Using OTP supervisor tree is intended only for control of restarting
>> >> application parts. If you want anything else you have to use links or
>> >> monitors unless you write your own supervisor which is not recommended
>> >> way.
>> >>
>> >> On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <[hidden email]>
>> >> wrote:
>> >>> Thanks Tim.  So it looks like they are using erlang:monitor and
>> >>> demonitor, so is that the best approach in an OTP application?  How
>> >>> does it figure into the supervision tree?
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <[hidden email]>
>> >>> wrote:
>> >>>> Take a look at the sources for http://zotonic.com/ as they use a
>> >>>> decent connection pool.
>> >>>>
>> >>>> On 4 August 2011 08:01, Andrew Berman <[hidden email]> wrote:
>> >>>>> I'm fairly new to Erlang and I am trying to write a process pool
>> >>>>> which
>> >>>>> essentially is a connection pool to a database.  I have designed the
>> >>>>> application like this:  The main pool gen_server has a queue of PIDs
>> >>>>> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server
>> >>>>> and carries
>> >>>>> the state of the connection.  So the 3rd party gen_server and the
>> >>>>> gen_fsm should always live and die together.  I currently have set
>> >>>>> it
>> >>>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
>> >>>>> party gen_server, but I'm wondering how this factors into the
>> >>>>> supervision tree.  I currently have two supervisors:
>> >>>>>
>> >>>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
>> >>>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>> >>>>>
>> >>>>> So my question is: is it ok to use spawn_link instead of somehow
>> >>>>> trying to put the 3rd party gen_server into the supervision tree?
>> >>>>>  If
>> >>>>> it should go into the supervision tree, how would I organize the
>> >>>>> tree
>> >>>>> then such that the gen_fsm and 3rd party gen_server are reliant on
>> >>>>> each other?
>> >>>>>
>> >>>>> Thanks for any help!
>> >>>>> _______________________________________________
>> >>>>> erlang-questions mailing list
>> >>>>> [hidden email]
>> >>>>> http://erlang.org/mailman/listinfo/erlang-questions
>> >>>>>
>> >>>>
>> >>> _______________________________________________
>> >>> erlang-questions mailing list
>> >>> [hidden email]
>> >>> http://erlang.org/mailman/listinfo/erlang-questions
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> --Hynek (Pichi) Vychodil
>> >>
>> >> Analyze your data in minutes. Share your insights instantly. Thrill
>> >> your boss.  Be a data hero!
>> >> Try GoodData now for free: www.gooddata.com
>> >> _______________________________________________
>> >> erlang-questions mailing list
>> >> [hidden email]
>> >> http://erlang.org/mailman/listinfo/erlang-questions
>> >
>> >
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
>
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions