Looking for an erlang function benchmarking library/framework

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Looking for an erlang function benchmarking library/framework

Martin Karlsson-2
Hello,

I want to benchmark individual functions to see how they perform. I've looked around for some library or framework which provides a  repeatable and structured way of doing benchmarking (much like eunit and common tests do for unit and integration tests) but haven't found anything on a micro-level.

Currently I have an ad-hoc collection of test functions wrapping the function under test in timer:tc but as mentioned looking for something more structured with proper reporting. Before writing it myself I wonder if there is anything out there already doing this?

Ideally doing:

$ make benchmark 
or
$ rebar benchmark

would run the benchmark tests and output a nice report.

I've played with golangs benchmark utilities in the testing package (http://golang.org/pkg/testing/) and found it quite useful and wonder if there is an erlang equivalent?

Perhaps this is trivial to write in a normal eunit/ct test together with timer:tc or fprof or something and if so I'd be happy to get any pointers on where to start.


Cheers,
Martin

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for an erlang function benchmarking library/framework

Martin Karlsson-2
Obviously a second after this email Loïc, on IRC,  told me about
"horse"(https://github.com/extend/horse)  which seems to be a good fit
and I'll start looking into that. Any other suggestions welcome.

Cheers,
Martin


On 7 April 2015 at 20:48, Martin Karlsson <[hidden email]> wrote:

> Hello,
>
> I want to benchmark individual functions to see how they perform. I've
> looked around for some library or framework which provides a  repeatable and
> structured way of doing benchmarking (much like eunit and common tests do
> for unit and integration tests) but haven't found anything on a micro-level.
>
> Currently I have an ad-hoc collection of test functions wrapping the
> function under test in timer:tc but as mentioned looking for something more
> structured with proper reporting. Before writing it myself I wonder if there
> is anything out there already doing this?
>
> Ideally doing:
>
> $ make benchmark
> or
> $ rebar benchmark
>
> would run the benchmark tests and output a nice report.
>
> I've played with golangs benchmark utilities in the testing package
> (http://golang.org/pkg/testing/) and found it quite useful and wonder if
> there is an erlang equivalent?
>
> Perhaps this is trivial to write in a normal eunit/ct test together with
> timer:tc or fprof or something and if so I'd be happy to get any pointers on
> where to start.
>
>
> Cheers,
> Martin
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for an erlang function benchmarking library/framework

Darach Ennis
In reply to this post by Martin Karlsson-2
Hi Martin,

You won't find a general purpose benchmarking framework suitable for all tasks but there are plenty of examples:

https://github.com/gar1t/erlang-bench - Garrett uses erlang scripts for his
* https://github.com/extend/horse - Loïc's horse [ glad i waited a second before responding... ]

For small or microbenchmarks there's also HDR histogram which will give fairly extensive
statistics and allows compensating for coordinated omission for latency sensitive tests if
that is relevant:


Cheers,

Darach.


On Tue, Apr 7, 2015 at 9:48 AM, Martin Karlsson <[hidden email]> wrote:
Hello,

I want to benchmark individual functions to see how they perform. I've looked around for some library or framework which provides a  repeatable and structured way of doing benchmarking (much like eunit and common tests do for unit and integration tests) but haven't found anything on a micro-level.

Currently I have an ad-hoc collection of test functions wrapping the function under test in timer:tc but as mentioned looking for something more structured with proper reporting. Before writing it myself I wonder if there is anything out there already doing this?

Ideally doing:

$ make benchmark 
or
$ rebar benchmark

would run the benchmark tests and output a nice report.

I've played with golangs benchmark utilities in the testing package (http://golang.org/pkg/testing/) and found it quite useful and wonder if there is an erlang equivalent?

Perhaps this is trivial to write in a normal eunit/ct test together with timer:tc or fprof or something and if so I'd be happy to get any pointers on where to start.


Cheers,
Martin

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for an erlang function benchmarking library/framework

Ignas Vyšniauskas-4
In reply to this post by Martin Karlsson-2
During Spawnfest 2012  we developed a unit-testing style library for
this purpose, though it was never completed:

    https://github.com/yfyf/perforator

It is probably broken under newer versions of rebar / R17 and not
really documented much, but I think the main idea is exactly what
you're looking for and if you are interesting I could walk you through
the thing / help you fix it.

--
Ignas
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for an erlang function benchmarking library/framework

Martin Karlsson-2
In reply to this post by Darach Ennis
> * http://release.softlab.ntua.gr/bencherl/ - targets scalability
> * https://github.com/gar1t/erlang-bench - Garrett uses erlang scripts for
> his
> * https://github.com/extend/horse - Loïc's horse [ glad i waited a second
> before responding... ]
>  *https://github.com/hdrhistogram/hdr_histogram_erl
>
> Cheers,
>
> Darach.
>
Thanks! Lots of useful information
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for an erlang function benchmarking library/framework

Martin Karlsson-2
In reply to this post by Ignas Vyšniauskas-4
>     https://github.com/yfyf/perforator
>
> but I think the main idea is exactly what
> you're looking for

Thanks Ignas,

Indeed, this looks pretty good. I will look into this as well as the
other suggestions and will let you know if I get stuck.

Cheers,
Martin
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for an erlang function benchmarking library/framework

Loïc Hoguin-3
In reply to this post by Darach Ennis
On 04/07/2015 12:03 PM, Ignas Vyšniauskas wrote:> During Spawnfest 2012
  we developed a unit-testing style library for
 > this purpose, though it was never completed:
 >
 >      https://github.com/yfyf/perforator

Actually wrote horse inspired by your project. Just I wanted the tests
in the same module alongside eunit tests (putting function then its
eunit/triq tests then its micro benchmarks in that order in the file,
repeat for each function).

On 04/07/2015 12:00 PM, Darach Ennis wrote:
> For small or microbenchmarks there's also HDR histogram which will give
> fairly extensive
> statistics and allows compensating for coordinated omission for latency
> sensitive tests if
> that is relevant:
>
> https://github.com/hdrhistogram/hdr_histogram_erl

Looks interesting. Might be good to use inside horse. I will have to try it.

Cheers,

--
Loïc Hoguin
http://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions