Thanks for new erlang logger: it is almost perfect

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Thanks for new erlang logger: it is almost perfect

Max Lapshin-2
Hi.

We were using lager for about 7 years and have rewritten it to our own subsystem called "events" (not opensourced and possible will not be).

Why?

Because logging has changed. 

In the ancient times of old sysadmins with beards that are boasting big uptime, logging was a synonym of appending lines to text file and extracting data from these lines with a perl script.

Things like webalyzer or radius were an ancient ancestors of modern ELK/Clickhouse stack.

Nowadays logging is changing and it seems that it has completely splitted into 2 different data streams.

One data stream is consumed by business analytics. It is collecting events from happy path. Request started, request finished, upstream request made, etc.  Here we need to log structured key-value objects that are collected from all servers of the project and linked together according to some metadata like "request-id" or "user-id"



Another data stream is about exceptions. They need to be collected and sent directly to developer tool like Sentry (on-premises and free. It is excellent) or Airbrake (good old service).



So we have changed from   lager:info("HTTP request handled") to

events:http_request([{ip,IP},{user_id,UserId},....]).

"events" is our own module with small preprocessor that takes all calls like  events:http_request on compile time and checks if they are defined in config file.

Thus we can simulate some kind of static typing and give our customers documentation on all methods without grep.

In config file it is possible to declare sinks like:

notify broken_video_stream {
  only media=mystream;
}






Then appeared new logger and I see that it has:

1) metadata.  It is very, very important.  Metadata is more important than text of message.
In our events we have following approach:
events:http_request([{ip,IP},{user_id,UserId},{method,Method}]).

and

{http_request,  ["HTTP request from ip ",ip, {user_id, [" user_id ",user_id],""}, " to ",method]}.

Yes, it is lager conditional formatting and it is really good. Metadata for robots, text line for sysadmins debugging this software.

2) remote proxy. Very cool.  It happened to be designed exactly 1-1 as we have designed in Flussonic, so it seems that we have done it right =)

3) lack of central process.

error_logger and lager were a big problem for us because they had single bottleneck that was working even for debug messages. Very often lager_event became a reason of crash for flussonic, so we have removed it and now it is ok.

However, I see that we have a bit more complicated mechanism of log buffering and handler overload.



Before sending message to log handler, we check if it is overloaded, then check its configuration and only after this we send message to log sink.

Overloading is checked by each sink itself: it sends message to itself:


All this is very important, because logging must not kill system due to slow disk or network.




I want to thank again for new system, I will think how to combine our private logging subsystem with standard infrastructure.

If our system is of some interest, I can open it code and explain our design ideas.






_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions