R14B, ETS, read_concurrency

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

R14B, ETS, read_concurrency

Rickard Green-2
Ryan Zezeski wrote:

> After glancing at the R14B release notes I noticed the SMP related
> improvements that were made.  I specifically find the new read_concurrency
> ETS opt interesting.  I currently have an app that performs roughly 480
> million lookups (in worst case) on multiple ETS tables, with some tables
> having up to 10 million entries.  I perform these lookups in parallel using
> 8 processes (on a 16 core machine).  This is the longest running part of my
> program with a runtime of > 3min.
>
> My point--I am doing a _huge_ amount of concurrent reads and every
> microsecond counts.  I am curious if anyone could throw a minimum % increase
> of lookups per second I might expect?
>
> -Ryan

What the effect will be on your application is impossible to guess. This
depends on how writes are done in the tables using the read_concurrency
option (if you have lots of write locked operations too you may even get
a performance degradation), what else your processes do (also other
processes not accessing the tables), what hardware you use, etc...

In order to give some figures, though. When doing lots of ets lookups
(only lookups) of small records on a public table from lots (hundreds)
of processes I get a speedup of about 5 using read_concurrency compared
to not using it on a dual quad-core machine we got, and about 2.5 on a
quad-core machine we got. The only thing these processes do are repeated
ets lookups. You should, however, not expect figures like these for
normal apps, since they typically do other stuff than ets lookup too.

The only way to know what effect it will have on your app is to try it out.

Regards,
Rickard Green
--
Rickard Green, Erlang/OTP, Ericsson AB.

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: R14B, ETS, read_concurrency

rzezeski
On Thu, Sep 16, 2010 at 8:13 AM, Rickard Green <[hidden email]> wrote:

>
> The only way to know what effect it will have on your app is to try it
> out.


Rickard, you're exactly right.  I should have worded my question
differently.  I'm interested in any results anyone has using this new
feature, and that's exactly what you gave me, thanks.  In my case, I have
highly concurrent reads with _zero_ interrupting writes.  The writes are
always done in a sequential manner before the reads.  Also, I'm not only
doing reads in my tight loop, but it is the dominator in my profile.

It seems my app may benefit from this new feature.  I'll report back when I
get some time to try it.