high performance cache

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

high performance cache

Senthilkumar Peelikkampatti
Hi,

I am looking for LRU caching framework in Erlang with thresholds to control
the amount of data written to it. I explored memcached Erlang derivatives
and ETS based solutions but not sure about which way to go. Please share
your experience using caching framework. I am not looking for distributed
cache.

--
Regards,
Senthilkumar Peelikkampatti,
Reply | Threaded
Open this post in threaded view
|

Re: high performance cache

Jesper Louis Andersen-2
On Sat, Sep 4, 2010 at 12:07 AM, Senthilkumar Peelikkampatti
<[hidden email]> wrote:

> I am looking for LRU caching framework in Erlang with thresholds to control
> the amount of data written to it. I explored memcached Erlang derivatives
> and ETS based solutions but not sure about which way to go. Please share
> your experience using caching framework. I am not looking for distributed
> cache.

Etorrent has a very very simple LRU janitorialization on file
descriptors. The principle is this: Every FD is governed by a process.
These processes enters themself into a janitor ETS table. Upon
reaching a high watermark, we do a full scan of the ETS table, find
the oldest beasts not accessed recently and then kill them. It is
viable because the number of open descriptors tend to be fairly small,
so we are not paying that much for a full table scan. Code:

http://github.com/jlouis/etorrent/blob/master/src/etorrent_fs_janitor.erl

If the table ends up being very big, I guess one should clean it up
based on a timer and a heuristic on how much stale data is in the
table. Also, beware of storing too much data in ETS as it is copied
into the memory space of the process in question. Same caveats apply
as when you try to send a 32mb binary tree from one process to the
other (in the default GC/allocator).

My guess is you can take the code above and use it for inspiration to
build an LRU ETS cache. It shouldn't take more than a couple of days
max to get it into shape.


--
J.

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: high performance cache

Gleb Peregud
In reply to this post by Senthilkumar Peelikkampatti
Cherly[1] might be interesting for you

1: http://github.com/cliffmoon/cherly

On Sat, Sep 4, 2010 at 00:07, Senthilkumar Peelikkampatti
<[hidden email]> wrote:

> Hi,
>
> I am looking for LRU caching framework in Erlang with thresholds to control
> the amount of data written to it. I explored memcached Erlang derivatives
> and ETS based solutions but not sure about which way to go. Please share
> your experience using caching framework. I am not looking for distributed
> cache.
>
> --
> Regards,
> Senthilkumar Peelikkampatti,
>

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: high performance cache

Nicholas Frechette-3
I believe the most efficient caching algorithm is the landlord algo. Google
it, i'm sure you'll find it. Great algo for caching slow fetching resources
(files, sounds, webpages, etc.). Not sure if the is an implementation in
erlang already or not but it should be fairly trivial to implement yourself.

On Sat, Sep 4, 2010 at 10:13 AM, Gleb Peregud <[hidden email]> wrote:

> Cherly[1] might be interesting for you
>
> 1: http://github.com/cliffmoon/cherly
>
> On Sat, Sep 4, 2010 at 00:07, Senthilkumar Peelikkampatti
> <[hidden email]> wrote:
> > Hi,
> >
> > I am looking for LRU caching framework in Erlang with thresholds to
> control
> > the amount of data written to it. I explored memcached Erlang derivatives
> > and ETS based solutions but not sure about which way to go. Please share
> > your experience using caching framework. I am not looking for distributed
> > cache.
> >
> > --
> > Regards,
> > Senthilkumar Peelikkampatti,
> >
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:[hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: high performance cache

Nicholas Frechette-3
For those interested, the landlord algorithm was published by Neal E. Young
at (a quick google search later):
http://www.cs.ucr.edu/~neal/publications

The algorithm paper itself is under On-line file caching (click on it to see
a brief description and to download/view the full pdf).
Some of his other papers are also of interest.

Per the webpage above:
*© Copyrights are reserved by the publishers.
Download for personal and limited academic use only. *

On Sat, Sep 4, 2010 at 5:00 PM, Nicholas Frechette <[hidden email]>wrote:

> I believe the most efficient caching algorithm is the landlord algo. Google
> it, i'm sure you'll find it. Great algo for caching slow fetching resources
> (files, sounds, webpages, etc.). Not sure if the is an implementation in
> erlang already or not but it should be fairly trivial to implement yourself.
>
>
> On Sat, Sep 4, 2010 at 10:13 AM, Gleb Peregud <[hidden email]> wrote:
>
>> Cherly[1] might be interesting for you
>>
>> 1: http://github.com/cliffmoon/cherly
>>
>> On Sat, Sep 4, 2010 at 00:07, Senthilkumar Peelikkampatti
>> <[hidden email]> wrote:
>> > Hi,
>> >
>> > I am looking for LRU caching framework in Erlang with thresholds to
>> control
>> > the amount of data written to it. I explored memcached Erlang
>> derivatives
>> > and ETS based solutions but not sure about which way to go. Please share
>> > your experience using caching framework. I am not looking for
>> distributed
>> > cache.
>> >
>> > --
>> > Regards,
>> > Senthilkumar Peelikkampatti,
>> >
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:[hidden email]
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: high performance cache

Zabrane Mickael
Thanks Nicolas

2010/9/5 Nicholas Frechette <[hidden email]>:

> For those interested, the landlord algorithm was published by Neal E. Young
> at (a quick google search later):
> http://www.cs.ucr.edu/~neal/publications
>
> The algorithm paper itself is under On-line file caching (click on it to see
> a brief description and to download/view the full pdf).
> Some of his other papers are also of interest.
>
> Per the webpage above:
> *© Copyrights are reserved by the publishers.
> Download for personal and limited academic use only. *
>
> On Sat, Sep 4, 2010 at 5:00 PM, Nicholas Frechette <[hidden email]>wrote:
>
>> I believe the most efficient caching algorithm is the landlord algo. Google
>> it, i'm sure you'll find it. Great algo for caching slow fetching resources
>> (files, sounds, webpages, etc.). Not sure if the is an implementation in
>> erlang already or not but it should be fairly trivial to implement yourself.
>>
>>
>> On Sat, Sep 4, 2010 at 10:13 AM, Gleb Peregud <[hidden email]> wrote:
>>
>>> Cherly[1] might be interesting for you
>>>
>>> 1: http://github.com/cliffmoon/cherly
>>>
>>> On Sat, Sep 4, 2010 at 00:07, Senthilkumar Peelikkampatti
>>> <[hidden email]> wrote:
>>> > Hi,
>>> >
>>> > I am looking for LRU caching framework in Erlang with thresholds to
>>> control
>>> > the amount of data written to it. I explored memcached Erlang
>>> derivatives
>>> > and ETS based solutions but not sure about which way to go. Please share
>>> > your experience using caching framework. I am not looking for
>>> distributed
>>> > cache.
>>> >
>>> > --
>>> > Regards,
>>> > Senthilkumar Peelikkampatti,
>>> >
>>>
>>> ________________________________________________________________
>>> erlang-questions (at) erlang.org mailing list.
>>> See http://www.erlang.org/faq.html
>>> To unsubscribe; mailto:[hidden email]
>>>
>>>
>>
>



--
Regards
Zabrane

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: high performance cache

ricardo_bcl
This post has NOT been accepted by the mailing list yet.
In reply to this post by Senthilkumar Peelikkampatti
After searching for this, I decided to implement a very simple cache mechanism using ETS:

https://github.com/ricardobcl/ETScache

For now, it only controls the number of elements in cache, because it's faster and it did what I wanted, but can be easily modified to prune it terms of bytes.


Senthilkumar Peelikkampatti wrote
Hi,

I am looking for LRU caching framework in Erlang with thresholds to control
the amount of data written to it. I explored memcached Erlang derivatives
and ETS based solutions but not sure about which way to go. Please share
your experience using caching framework. I am not looking for distributed
cache.

--
Regards,
Senthilkumar Peelikkampatti,