Performance measurement for er_zauker

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance measurement for er_zauker

Giovanni Giorgi

Hi all,

 I am writing a code indexer called er_zaueker: it is based on erlang and redis: https://github.com/daitangio/er_zauker

 

It is also my "erlang learning project".

I am trying to optimize it.

What are the OTP API I can use to find out where is spent much of the time?

I need an aggregate timing value to find the slowest functions.

er_zauker breaks the files in "trigram" and push them to redis. This part is done sequentially on a single file, whereas file processing is spread via a lot of processes.

The entire work is higly I/O bound, but I hope to break the 10 files per second barrier soon.

I'd like tu understang if it make sense to have more processes per single file.

To collect trigram I am using the sets module, but I think R17 maps could be even more fast.

 

---
Gio's Blog http://gioorgi.com

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Performance measurement for er_zauker

dmkolesnikov
Hi,


- Dmitry


On 31 Jul 2014, at 15:02, Giovanni Giorgi <[hidden email]> wrote:

Hi all,

 I am writing a code indexer called er_zaueker: it is based on erlang and redis: https://github.com/daitangio/er_zauker

 

It is also my "erlang learning project".

I am trying to optimize it.

What are the OTP API I can use to find out where is spent much of the time?

I need an aggregate timing value to find the slowest functions.

er_zauker breaks the files in "trigram" and push them to redis. This part is done sequentially on a single file, whereas file processing is spread via a lot of processes.

The entire work is higly I/O bound, but I hope to break the 10 files per second barrier soon.

I'd like tu understang if it make sense to have more processes per single file.

To collect trigram I am using the sets module, but I think R17 maps could be even more fast.

 
---
Gio's Blog http://gioorgi.com
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Performance measurement for er_zauker

Pablo Polvorin-3
I wrote a blog post time ago explaining a simple usage of fprof to discover possible bottlenecks, maybe it can help you getting started.
http://blog.process-one.net/optimizing-erlang-applications-emysql/

 fprof is only one of many available profiling tools,  each has different trade-offs.


On 31 July 2014 12:15, Dmitry Kolesnikov <[hidden email]> wrote:
Hi,


- Dmitry


On 31 Jul 2014, at 15:02, Giovanni Giorgi <[hidden email]> wrote:

Hi all,

 I am writing a code indexer called er_zaueker: it is based on erlang and redis: https://github.com/daitangio/er_zauker

 

It is also my "erlang learning project".

I am trying to optimize it.

What are the OTP API I can use to find out where is spent much of the time?

I need an aggregate timing value to find the slowest functions.

er_zauker breaks the files in "trigram" and push them to redis. This part is done sequentially on a single file, whereas file processing is spread via a lot of processes.

The entire work is higly I/O bound, but I hope to break the 10 files per second barrier soon.

I'd like tu understang if it make sense to have more processes per single file.

To collect trigram I am using the sets module, but I think R17 maps could be even more fast.

 
---
Gio's Blog http://gioorgi.com
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Pablo Polvorin
ProcessOne

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions