Lager dropping messages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Lager dropping messages

Frank Muller
Hi guys,

I’m seeing this annoying warning in my logs:
[warning] lager_error_logger_h dropped 107 messages in the last second that exceeded the limit of 50 messages/sec.

How can i set Lager (I’m using lager v2 from trunk) and increase this limit of 50msg/sec (ex. set it 300)?


Thank you.
/Frank
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lager dropping messages

Luke Bakken
Hi Frank,

I think this is what you're looking for:
https://github.com/erlang-lager/lager#overload-protection

On Mon, Sep 11, 2017 at 8:38 AM, Frank Muller
<[hidden email]> wrote:

> Hi guys,
>
> I’m seeing this annoying warning in my logs:
> [warning] lager_error_logger_h dropped 107 messages in the last second that
> exceeded the limit of 50 messages/sec.
>
> How can i set Lager (I’m using lager v2 from trunk) and increase this limit
> of 50msg/sec (ex. set it 300)?
>
>
> Thank you.
> /Frank
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>



--
Staff Software Engineer
Pivotal / RabbitMQ
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lager dropping messages

Jesper Louis Andersen-2
One thing to be aware of here: When your Erlang instance boots, it logs quite much. Often so much that this overload protection will trigger. If your node() is dying early on in the boot phase, consider bumping this count to 3000 or so.

On Mon, Sep 11, 2017 at 7:05 PM Luke Bakken <[hidden email]> wrote:
Hi Frank,

I think this is what you're looking for:
https://github.com/erlang-lager/lager#overload-protection

On Mon, Sep 11, 2017 at 8:38 AM, Frank Muller
<[hidden email]> wrote:
> Hi guys,
>
> I’m seeing this annoying warning in my logs:
> [warning] lager_error_logger_h dropped 107 messages in the last second that
> exceeded the limit of 50 messages/sec.
>
> How can i set Lager (I’m using lager v2 from trunk) and increase this limit
> of 50msg/sec (ex. set it 300)?
>
>
> Thank you.
> /Frank
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>



--
Staff Software Engineer
Pivotal / RabbitMQ
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lager dropping messages

Frank Muller
Thanks for the guidance guys. Kind of lost with my new lager's settings:

lager,
    [{error_logger_hwm,30},
     {async_threshold,500},
     {async_threshold_window,250},

I'm still seeing dropped messages :-/

What the correct settings for let's say 3000 as suggested by Jesper?

/Frank


<[hidden email]> wrote :
One thing to be aware of here: When your Erlang instance boots, it logs quite much. Often so much that this overload protection will trigger. If your node() is dying early on in the boot phase, consider bumping this count to 3000 or so.


On Mon, Sep 11, 2017 at 7:05 PM Luke Bakken <[hidden email]> wrote:
Hi Frank,

I think this is what you're looking for:
https://github.com/erlang-lager/lager#overload-protection

On Mon, Sep 11, 2017 at 8:38 AM, Frank Muller
<[hidden email]> wrote:
> Hi guys,
>
> I’m seeing this annoying warning in my logs:
> [warning] lager_error_logger_h dropped 107 messages in the last second that
> exceeded the limit of 50 messages/sec.
>
> How can i set Lager (I’m using lager v2 from trunk) and increase this limit
> of 50msg/sec (ex. set it 300)?
>
>
> Thank you.
> /Frank
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>



--
Staff Software Engineer
Pivotal / RabbitMQ
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lager dropping messages

Andrew Thompson-4
On Mon, Sep 11, 2017 at 06:51:54PM +0000, Frank Muller wrote:

> Thanks for the guidance guys. Kind of lost with my new lager's settings:
>
> lager,
>     [{error_logger_hwm,30},
>      {async_threshold,500},
>      {async_threshold_window,250},
>
> I'm still seeing dropped messages :-/
>
> What the correct settings for let's say 3000 as suggested by Jesper?
>
> /Frank

You will want to significantly raise the error_logger HWM, the HWM is
how many messages lager will pop from the error_logger mailbox before
starting to discard them.

Also, you might want to try lager 3.x, I landed some fixes to the
error_logger discard code a few months ago:

https://github.com/erlang-lager/lager/pull/411

Andrew
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lager dropping messages

Andrew Thompson-4
In reply to this post by Frank Muller
On Mon, Sep 11, 2017 at 06:51:54PM +0000, Frank Muller wrote:

> Thanks for the guidance guys. Kind of lost with my new lager's settings:
>
> lager,
>     [{error_logger_hwm,30},
>      {async_threshold,500},
>      {async_threshold_window,250},
>
> I'm still seeing dropped messages :-/
>
> What the correct settings for let's say 3000 as suggested by Jesper?
>
> /Frank

You will want to significantly raise the error_logger HWM, the HWM is
how many messages lager will pop from the error_logger mailbox before
starting to discard them.

Also, you might want to try lager 3.x, I landed some fixes to the
error_logger discard code a few months ago:

https://github.com/erlang-lager/lager/pull/411

Andrew
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lager dropping messages

Andrew Thompson-4
In reply to this post by Frank Muller
On Mon, Sep 11, 2017 at 06:51:54PM +0000, Frank Muller wrote:

> Thanks for the guidance guys. Kind of lost with my new lager's settings:
>
> lager,
>     [{error_logger_hwm,30},
>      {async_threshold,500},
>      {async_threshold_window,250},
>
> I'm still seeing dropped messages :-/
>
> What the correct settings for let's say 3000 as suggested by Jesper?
>
> /Frank

You will want to significantly raise the error_logger HWM, the HWM is
how many messages lager will pop from the error_logger mailbox before
starting to discard them.

Also, you might want to try lager 3.x, I landed some fixes to the
error_logger discard code a few months ago:

https://github.com/erlang-lager/lager/pull/411

Andrew
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Lager dropping messages

Frank Muller
In reply to this post by Andrew Thompson-4
Working now ({error_logger_hwm,3000}), thanks.

/Frank


On Mon, Sep 11, 2017 at 06:51:54PM +0000, Frank Muller wrote:
> Thanks for the guidance guys. Kind of lost with my new lager's settings:
>
> lager,
>     [{error_logger_hwm,30},
>      {async_threshold,500},
>      {async_threshold_window,250},
>
> I'm still seeing dropped messages :-/
>
> What the correct settings for let's say 3000 as suggested by Jesper?
>
> /Frank

You will want to significantly raise the error_logger HWM, the HWM is
how many messages lager will pop from the error_logger mailbox before
starting to discard them.

Also, you might want to try lager 3.x, I landed some fixes to the
error_logger discard code a few months ago:

https://github.com/erlang-lager/lager/pull/411

Andrew
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions