patterns or best practices to shut applications down?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

patterns or best practices to shut applications down?

Xavier Noria
Let me see if I can explain this.

When getting into Erlang, I start to think more within the system. Say, a long file upload could be done in a process, periodic tasks could be running in the VM instead of cron, you could have a key-value store in memory, etc. In the context of web development, for example, I think of natural ways to leverage processes that in other systems would normally be delegated to external services.

Kinda what the famous table in Elixir in Action has, I guess.

But there's something that bugs me and prevents me from imagining complete and round solutions, which is shutting down. When shutting down (think autoescaling, deploys, etc.). I have a live system that needs to stop orderly.

Are there patterns or best practises about how to design this aspect of the system?


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: patterns or best practices to shut applications down?

Xavier Noria
Chatted a bit with José Valim about this.

I had read the APIs related to process shutdown, but I had not seen the big picture. A conventionally written OTP application has a way to boot, but had not grasped that it has also a way to shutdown and you program to play well wit it. Aha!

So the auto-answer to my question (please don't hesitate to correct me or add anything) is that every service needs to take a conscious decision about this. Logic, timeouts, and so on. In some cases exiting is going to be inmediate. A periodical task may wait for completion if exiting happens in the middle of it, but schedule no more runs (off the top of my head). In the case of a somewhat long-running process like file uploading, guess a normal strategy would be similar: to allow their exit handlers to wait for the upload to complete and program this logic. Their supervisor would set appropriate generous timeouts or infinity accordingly. In such a system it is a consequence of this that you should expect shutdown to take whatever those processes need.

Beautifull!


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: patterns or best practices to shut applications down?

Fred Hebert-2
In reply to this post by Xavier Noria
On 12/23, Xavier Noria wrote:
>
>But there's something that bugs me and prevents me from imagining complete
>and round solutions, which is shutting down. When shutting down (think
>autoescaling, deploys, etc.). I have a live system that needs to stop
>orderly.
>
>Are there patterns or best practises about how to design this aspect of the
>system?

Good news! You already have this covered with OTP.

When shutting down a supervision tree, the supervisor will send down a
`shutdown` exit signal to each of its children in order. A child
supervisor will trap exit on this, and forward it to its own children
first.

If your worker has nothing special to do, just let it die. If it needs
some special cleanup, trap exit in the worker and `terminate` will be
called.

So the supervision structure you have that tells how the program boots
also tells how it shuts down. You do have to be careful about the
handling of timeouts though (or brutal_kill will catch you up).
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: patterns or best practices to shut applications down?

PAILLEAU Eric
Hi,
in addition to other answers, it is worth noting that you can set
a maximum time to VM to stop.

http://erlang.org/doc/man/init.html

"To limit the shutdown time, the time init is allowed to spend taking
down applications, command-line flag -shutdown_time is to be used."

this can be very useful when some processes are stuck in abnormal
terminate work. But using this you can loose data obviously.
Restart time is sometime more important than loosing some cleanup.

Regards




Le 23/12/2017 à 15:41, Fred Hebert a écrit :

> On 12/23, Xavier Noria wrote:
>>
>> But there's something that bugs me and prevents me from imagining
>> complete
>> and round solutions, which is shutting down. When shutting down (think
>> autoescaling, deploys, etc.). I have a live system that needs to stop
>> orderly.
>>
>> Are there patterns or best practises about how to design this aspect
>> of the
>> system?
>
> Good news! You already have this covered with OTP.
>
> When shutting down a supervision tree, the supervisor will send down a
> `shutdown` exit signal to each of its children in order. A child
> supervisor will trap exit on this, and forward it to its own children
> first.
>
> If your worker has nothing special to do, just let it die. If it needs
> some special cleanup, trap exit in the worker and `terminate` will be
> called.
>
> So the supervision structure you have that tells how the program boots
> also tells how it shuts down. You do have to be careful about the
> handling of timeouts though (or brutal_kill will catch you up).
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: patterns or best practices to shut applications down?

Ryan Stewart
Specifically for web applications, and more specifically for a webapp that can have long-running requests (measured in minutes), I've found it extremely convenient to have a supervised gen_server that simply tracks the start and end of each http request. It traps exits so that when it's told to shut down, it can enter a "terminate loop", and it doesn't terminate until all requests have completed or a timeout elapses. This is complemented by a shutdown timeout in its supervisor, which will hard kill it eventually, in the case that something goes terribly wrong. After that, it's just a matter of getting your supervisor/process start/stop in the correct order.

On Sat, Dec 23, 2017 at 9:07 AM PAILLEAU Eric <[hidden email]> wrote:
Hi,
in addition to other answers, it is worth noting that you can set
a maximum time to VM to stop.

http://erlang.org/doc/man/init.html

"To limit the shutdown time, the time init is allowed to spend taking
down applications, command-line flag -shutdown_time is to be used."

this can be very useful when some processes are stuck in abnormal
terminate work. But using this you can loose data obviously.
Restart time is sometime more important than loosing some cleanup.

Regards




Le 23/12/2017 à 15:41, Fred Hebert a écrit :
> On 12/23, Xavier Noria wrote:
>>
>> But there's something that bugs me and prevents me from imagining
>> complete
>> and round solutions, which is shutting down. When shutting down (think
>> autoescaling, deploys, etc.). I have a live system that needs to stop
>> orderly.
>>
>> Are there patterns or best practises about how to design this aspect
>> of the
>> system?
>
> Good news! You already have this covered with OTP.
>
> When shutting down a supervision tree, the supervisor will send down a
> `shutdown` exit signal to each of its children in order. A child
> supervisor will trap exit on this, and forward it to its own children
> first.
>
> If your worker has nothing special to do, just let it die. If it needs
> some special cleanup, trap exit in the worker and `terminate` will be
> called.
>
> So the supervision structure you have that tells how the program boots
> also tells how it shuts down. You do have to be careful about the
> handling of timeouts though (or brutal_kill will catch you up).
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions