Using prim_file and the binary module produces:
[mevans@scream ~]$ time perl reader.pl
Found 6032291 lines.
real 0m1.368s
user 0m1.295s
sys 0m0.068s
[mevans@scream ~]$ time erl -noshell -s reader main
Found 6048490 lines.
real 0m2.090s
user 0m1.965s
sys 0m0.111s
The Erlang code is reporting more lines because I am double counting cases where a line is split between reads. Of course, this would be easy to handle - I just wanted a proof of concept :
-module(reader).
-export([main/0]).
-define(NL,10).
main() ->
{ok, File} = prim_file:open("data.log", [read,binary]),
Lines = count_lines(File, 0),
io:format("Found ~w lines.~n", [Lines]),
halt().
count_lines(File, Count) ->
case prim_file:read(File,8192) of
{ok, Line} ->
TC = length(binary:split(Line,[<<10>>],[global])),
count_lines(File, Count+TC);
_ ->
Count
end.
________________________________________
From:
[hidden email] [
[hidden email]] On Behalf Of Bob Ippolito [
[hidden email]]
Sent: Saturday, July 09, 2011 7:33 PM
To: Kenny Stone
Cc: erlang-questions Questions
Subject: Re: [erlang-questions] Erlang read file benchmark
I think what most people want (especially for benchmarks) is something
that doesn't care about encodings and doesn't have a lot of
indirection. The current solution is VERY flexible, which comes at a
severe cost to performance in this case.
If you read the source code you'll see that file:read_file/1 calls
into io:request/2 which eventually (in another process) ends up in
file_io_server:io_request/2 and ends up reading either 128 bytes or
8kb at a time, doing some unicode junk, and ends up calling
io_lib:collect_line/4 to collect each line chunk at a time.
If I was trying to win a benchmark I'd probably go directly to
prim_file, do my own buffering, and use erlang:decode_packet/3 or the
binary module to split on the newlines. If I wanted to make a nicer
API I'd put that in a process to manage the buffering.
On Sat, Jul 9, 2011 at 4:08 PM, Kenny Stone <
[hidden email]> wrote:
> Why is it awful?
>
> On Sat, Jul 9, 2011 at 6:07 PM, Bob Ippolito <
[hidden email]> wrote:
>>
>> file:read_line does some pretty awful things, I'd expect it to be very
>> slow. That said, there should be a much faster yet still easy way to
>> do this quickly but there isn't one baked into OTP that I know of.
>>
>> On Saturday, July 9, 2011, Michael Truog <
[hidden email]> wrote:
>> > He only showed the results on the command-line. It would be nice to see
>> > results that show runtime without the startup/teardown overhead that the
>> > Erlang VM has, since it has a lot more going on than the perl interpreter.
>> > I know he briefly mentioned that the difference seemed minimal, but he
>> > posted no results to show that.
>> >
>> > On 07/09/2011 12:15 PM, Evans, Matthew wrote:
>> >> Sorry if this is a duplicate email.
>> >>
>> >> I can understand Erlang being a bit slower than Perl for this. Can't
>> >> see an excuse for such a difference though.
>> >>
>> >>
http://agentzh.org/#ErlangFileReadLineBenchmark>> >>
>> >> Matt
>> >>
>> >> Sent from my iPhone
>> >> _______________________________________________
>> >> erlang-questions mailing list
>> >>
[hidden email]
>> >>
http://erlang.org/mailman/listinfo/erlang-questions>> >>
>> >
>> > _______________________________________________
>> > erlang-questions mailing list
>> >
[hidden email]
>> >
http://erlang.org/mailman/listinfo/erlang-questions>> >
>> _______________________________________________
>> erlang-questions mailing list
>>
[hidden email]
>>
http://erlang.org/mailman/listinfo/erlang-questions>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Locked
