BEAM performance?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

BEAM performance?

Oliver Korpilla
Hello.
 
We wrote a moderately complex telco application at work using elixir, but I think the performance we observe would be similar in Erlang since basically it all depends on BEAM and OTP anyway, so please don't mind me asking here.
 
We run message scenarios that involve ASN.1 encoding/decoding, several IP protocols, a proprietary transport we access through an Erlang port, etc. Message sequences will be handled through gen_fsm processes.
 
The application is split over several nodes.
 
What we've observed is:
* First time scenario runs is super slow.
* Second time, time is halved.
* Third time, performance is very good, especially given the complex scenario.
 
When analyzing bottlenecks in the software, we observed and tried the following:
* I preloaded all modules into the VM and times improved significantly. 
* ASN.1 encoding on 1st try can take as much as 20ms on first run but go <1ms after for same message with minimally different content.
* Using HiPE on the ASN.1 generated codec did worsen performance.
* Executing the message codec at least once during startup improved ASN.1 performance the most.
* Two DB writes to same table row in short order can significantly decrease performance by causing multi-ms delays.
 
(I know this is probably no surprise to anyone here but we're learning how the system behaves during runtime.)
 
The curve described above still persists - the totals are just less.
 
Our working assumption is that caching is impacting performance here, but we actually know too little about the BEAM runtime.
 
* Are there mechanisms internal to BEAM impacting this? Or is it purely a property of the CPU architecture?
* Could I stimulate BEAM to cache certain parts of the code ahead of time? 
 
The measurements are rough ones, done by log timestamps. ms granularity is enough for us to gauge overall performance of our signalling.
 
Message performance for user 3 and onward are okay but users 1 and/or 2 could be dropped because of timeouts, so optimizing these still matters.
 
Thank you for any advice you can give,
Oliver

(Resent on Raimo's request)
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: BEAM performance?

Jesper Louis Andersen-2
On Wed, May 24, 2017 at 3:02 PM Oliver Korpilla <[hidden email]> wrote:
 
* Are there mechanisms internal to BEAM impacting this? Or is it purely a property of the CPU architecture?

If you experience latencies in the millisecond range, I don't think you are looking at a CPU caching problem. A DRAM hit is 300ns or 0.3us. So you'd need about 3000 of those to hit 1ms run time. The napkin math doesn't really pan out too well.

Do you run the ERTS in embedded mode[0]? I'd look at code loading. Or perhaps ASN.1 loading initially if it compiles stuff the first time around.

As for the two DB writes to the same row in short order: Mnesia uses an optimistic locking strategy. If two transactions write to the same row at the same time, both will aborted, wait a bit and try again. If you're unlucky, those two writes will keep interfering with each other and you will get lag. A good way around this is a design which serializes such writes. Or perhaps you can make the write dirty (this can actually be safe in some situations, depending on your setup).

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: BEAM performance?

Oliver Korpilla
Hello, Jesper.

My "worst" Mnesia write collision I could fix by making a single transaction out of two. That was just bad coding. We also occasionally encounter slow writes but I think that's due to what you described.

Unfortunately due to a restriction that is completely our fault we're not running in embedded mode. But I call ensure_loaded on all BEAM modules I can find in our build directory... (There seems to be no easy way to extract all beam modules in the system or to enter embedded mode during runtime?)

I will still investigate whether I can get it to run in embedded mode.

Thank you,
Oliver

 
 

Gesendet: Mittwoch, 24. Mai 2017 um 15:19 Uhr
Von: "Jesper Louis Andersen" <[hidden email]>
An: "Oliver Korpilla" <[hidden email]>, [hidden email]
Betreff: Re: [erlang-questions] BEAM performance?

On Wed, May 24, 2017 at 3:02 PM Oliver Korpilla <[hidden email][mailto:[hidden email]]> wrote:
 
* Are there mechanisms internal to BEAM impacting this? Or is it purely a property of the CPU architecture?
 
If you experience latencies in the millisecond range, I don't think you are looking at a CPU caching problem. A DRAM hit is 300ns or 0.3us. So you'd need about 3000 of those to hit 1ms run time. The napkin math doesn't really pan out too well.
 
Do you run the ERTS in embedded mode[0]? I'd look at code loading. Or perhaps ASN.1 loading initially if it compiles stuff the first time around.
 
As for the two DB writes to the same row in short order: Mnesia uses an optimistic locking strategy. If two transactions write to the same row at the same time, both will aborted, wait a bit and try again. If you're unlucky, those two writes will keep interfering with each other and you will get lag. A good way around this is a design which serializes such writes. Or perhaps you can make the write dirty (this can actually be safe in some situations, depending on your setup).
[0] http://erlang.org/doc/system_principles/system_principles.html#code_loading[http://erlang.org/doc/system_principles/system_principles.html#code_loading]
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: BEAM performance?

Håkan Mattsson
In reply to this post by Jesper Louis Andersen-2

On Wed, May 24, 2017 at 3:19 PM, Jesper Louis Andersen <[hidden email]> wrote:

As for the two DB writes to the same row in short order: Mnesia uses an optimistic locking strategy. If two transactions write to the same row at the same time, both will aborted, wait a bit and try again. If you're unlucky, those two writes will keep interfering with each other and you will get lag.

​No, only one transaction will be restarted.​
 
​The "oldest" one will be completed.

/Håkan

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...