Small performance hint: avoid abusing module_info(attributes)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Small performance hint: avoid abusing module_info(attributes)

Max Lapshin-2
Hi.

We have moved from plain text lager logging to our own event system that has main difference: we have limited amount of possible events that are compiled in code.

In known_events.erl I have:


-event({stream_started, info, ["stream started"]}).

and then I write in code:


events:stream_started([{media,Name}]).

Later all this is printed (using lager formatting, it is cool).



So we are close to the problem.  My code that was handling such:  events:stream_started was searching for predefined event via:  known_events:module_info(attributes).

Suddenly eprof told that I need to look carefully at this place and I wrote parse_transform that takes all attributes from module AST and compiles them into code. Also I've generated function  known_events:find(Name) with all possible clauses.

On single core of my macbook I got 4000 times speedup.  100 000 of fetches known_events:all()  took about 8 seconds with module_info(attributes) and about 2 milliseconds with generated and compiled code.


So lesson from here is: try to avoid calling module_info(attributes) in innner loops. It may be suprisingly expensive.





_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Small performance hint: avoid abusing module_info(attributes)

Сергей Прохоров-2
It looks like attribute list is stored in `term_to_binary` format inside of a .beam file.

So, each time you call `M:module_info(attributes)`, it decodes whole attribute list using `binary_to_term`:


That may be ok for small list, but you, probably, have a lot of them (how many?).

Another question is how did you lookup this attribute list? lists:keyfind? This one, obviously, may be slow on a large lists.

But, at least, it doesn't look like `module_info(attributes)` uses any locks.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Small performance hint: avoid abusing module_info(attributes)

Ulf Wiger-2
In reply to this post by Max Lapshin-2
At the very least, one should exercise caution on some particular modules, and perhaps generally modules that have lots of type specs.

Interestingly, when fetching attributes from beam_lib, dialyzer-generated attributes are moved to a subgroup, so at least in OTP 20.0 code, attribute lists are no longer than 4.

Fetching via M:module_info/1, however, the list is flat, and can be quite long, mostly depending on the number of types. Looking again at OTP, but loading each module and fetching the attributes via M:module_info/1 (the example below lists all modules with at least 20 attributes):

19> lists:foldr(fun(D,Acc) -> filelib:fold_files(D, ".*", false, fun(F,A) -> M = list_to_atom(filename:basename(F,".beam")), try M:module_info(attributes) of As -> case length(As) of L when L >= 20 -> [{M,L}|A]; _ -> A end catch error:_ -> A end end, []) ++ Acc end, [], code:get_path()).
[{gen_fsm,20},
 {erl_parse,208},
 {xmerl_xpath_parse,45},
 {xref_parser,34},
 {snmpc_mib_gram,232},
 {megaco_text_parser_v3,413},
 {megaco_text_parser_v2,378},
 {megaco_text_parser_v1,325},
 {megaco_text_parser_prev3c,410},
 {megaco_text_parser_prev3b,396},
 {megaco_text_parser_prev3a,389},
 {megaco_text_mini_parser,25},
 {icparse,185},
 {edoc_parser,94},
 {diameter_dict_parser,76},
 {cosNotification_Grammar,36},
 {core_parse,224}]


BR,
Ulf W

2017-09-28 8:56 GMT+02:00 Max Lapshin <[hidden email]>:
Hi.

We have moved from plain text lager logging to our own event system that has main difference: we have limited amount of possible events that are compiled in code.

In known_events.erl I have:


-event({stream_started, info, ["stream started"]}).

and then I write in code:


events:stream_started([{media,Name}]).

Later all this is printed (using lager formatting, it is cool).



So we are close to the problem.  My code that was handling such:  events:stream_started was searching for predefined event via:  known_events:module_info(attributes).

Suddenly eprof told that I need to look carefully at this place and I wrote parse_transform that takes all attributes from module AST and compiles them into code. Also I've generated function  known_events:find(Name) with all possible clauses.

On single core of my macbook I got 4000 times speedup.  100 000 of fetches known_events:all()  took about 8 seconds with module_info(attributes) and about 2 milliseconds with generated and compiled code.


So lesson from here is: try to avoid calling module_info(attributes) in innner loops. It may be suprisingly expensive.





_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions