Erlang Web Servers challenge

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Erlang Web Servers challenge

Zabrane Mickael
Hi guys,

I'd like your help for this small Web servers challenge.

The idea is ver simple: measure "how fast" you can serve a "100 bytes" file (attached as "small.html") 
in "less than 01 second" by using one of the following web servers:

- Bare metal web server using "gen_tcp"

The rules are very simple :

1. The first time you served the file, you MUST read it from disk.
All subsequent requests to that file can use any caching mechanism!

2. Any (hidden) trick is allowed (eg. prim_inet, SMP enabled ...)

3. Erlang versions R14B to R14B03

4. The HTTP client to use for this benchmark needs to be "Apache AB" (http://httpd.apache.org/docs/2.0/programs/ab.html) with:
$ ab -n 200000 -c 100 -t1 -k "http://127.0.0.1:8080/small.html"

I hope that the authors if these web servers will also contribute to this bench as they
know their software better than anybody else.

Challenge's open ;-)

Regards,
Zabrane



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

small.html (100 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Loïc Hoguin
On 07/04/2011 05:11 PM, Zabrane Mickael wrote:

> Hi guys,
>
> I'd like your help for this small Web servers challenge.
>
> The idea is ver simple: measure "how fast" you can serve a "100 bytes"
> file (attached as "small.html")
> in "less than 01 second" by using one of the following web servers:
>
> - Yaws (http://yaws.hyber.org/download/yaws-1.90.tar.gz)
> - Mochiweb (https://github.com/mochi/mochiweb)
> - Misultin (https://github.com/ostinelli/misultin)
> - Cowboy (https://github.com/extend/cowboy)
> - Bare metal web server using "gen_tcp"

I don't really understand where the challenge is.

Also yaws should be faster considering it supports sendfile whereas the
others don't without much additional work beforehand. So regardless of
the server being used you'd probably just be benchmarking sendfile and
not the servers.

--
Loïc Hoguin
Dev:Extend
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Zabrane Mickael
>> I don't really understand where the challenge is.

The challenge as stated is very really simple.
How fast you can serve a 100 bytes files with your web server "Cowboy" for example in less than a second.

> Also yaws should be faster considering it supports sendfile whereas the
> others don't without much additional work beforehand. So regardless of
> the server being used you'd probably just be benchmarking sendfile and
> not the servers.

Don't suppose anything (sendfile) before benchmarking it Loïc.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Loïc Hoguin
On 07/04/2011 07:41 PM, Zabrane Mickael wrote:
>>> I don't really understand where the challenge is.
>
> The challenge as stated is very really simple.
> How fast you can serve a 100 bytes files with your web server "Cowboy" for example in less than a second.

Only yaws is suited for this "challenge", the others aren't meant to
serve files and would be better coupled with something like cherokee or
nginx deployed on multiple servers each using separate domains.

Even yaws with sendfile would probably be beaten by the C servers for
that purpose. Though I guess measuring that one would be interesting.

>> Also yaws should be faster considering it supports sendfile whereas the
>> others don't without much additional work beforehand. So regardless of
>> the server being used you'd probably just be benchmarking sendfile and
>> not the servers.
>
> Don't suppose anything (sendfile) before benchmarking it Loïc.

I don't suppose anything. Kernel cache is faster than userland cache.

--
Loïc Hoguin
Dev:Extend
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Zabrane Mickael
Also yaws should be faster considering it supports sendfile whereas the
others don't without much additional work beforehand. So regardless of
the server being used you'd probably just be benchmarking sendfile and
not the servers.

Don't suppose anything (sendfile) before benchmarking it Loïc.

I don't suppose anything. Kernel cache is faster than userland cache.

I don't think so. I'm using "sendfile" (extracted from yaws) https://github.com/tuncer/sendfile/commits/master/
(thanks to Steve and Tuncer) in our production app server for while now.
Except a noticed decrease in CPU usage, it didn't make huge difference compared to "gen_tcp:send" in term of speed.
But again, that's not fair because our app server is very special.

I really hope you can fine tune Cowboy (if possible) and let us know how fast can it serve a 100 bytes file in less than a second.
Thanks Loïc!

N.B:  for the other web servers authors, please take 10mn for the challenge!

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Joe Armstrong-2
In reply to this post by Zabrane Mickael
On Mon, Jul 4, 2011 at 5:11 PM, Zabrane Mickael <[hidden email]> wrote:
> Hi guys,
> I'd like your help for this small Web servers challenge.
> The idea is ver simple: measure "how fast" you can serve a "100 bytes" file
> (attached as "small.html")

Why measure this? Serving static content is one of the least
interesting things I can think of
to study. Any network worth the name won't bother to access the "root"
server but will cache the
content upstream.

How fast you can serve up dynamic content is far more interesting,
since it can't be cached.
Something that evokes a computation on the server is far more
interesting, and something that stresses
the concurrency.

How about processing send a GET /fac/N command (ie return factorial N)
The server should

1) send factorial N
2) wait 30 seconds
3) send factorial N (again)
4) close the socket

The 30 seconds gap will cause lots of nastiness in the server. Such a
benchmark tests many useful things
(for example the inter-session protection mechanism, as soon as we
perform computation on behalf of
the client we have to protect server sessions against each other,
secondly handling concurrency. If we wait
30 sections and do 10K requests/sec, we going to build up 300K
processes, which tests a lot of internal stuff)

 /Joe


> in "less than 01 second" by using one of the following web servers:
> - Yaws (http://yaws.hyber.org/download/yaws-1.90.tar.gz)
> - Mochiweb (https://github.com/mochi/mochiweb)
> - Misultin (https://github.com/ostinelli/misultin)
> - Cowboy (https://github.com/extend/cowboy)
> - Bare metal web server using "gen_tcp"
> The rules are very simple :
> 1. The first time you served the file, you MUST read it from disk.
> All subsequent requests to that file can use any caching mechanism!
> 2. Any (hidden) trick is allowed (eg. prim_inet, SMP enabled ...)
> 3. Erlang versions R14B to R14B03
> 4. The HTTP client to use for this benchmark needs to be "Apache AB"
> (http://httpd.apache.org/docs/2.0/programs/ab.html) with:
> $ ab -n 200000 -c 100 -t1 -k "http://127.0.0.1:8080/small.html"
> I hope that the authors if these web servers will also contribute to this
> bench as they
> know their software better than anybody else.
> Challenge's open ;-)
> Regards,
> Zabrane
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Zabrane Mickael
> Why measure this? Serving static content is one of the least
> interesting things I can think of
> to study. Any network worth the name won't bother to access the "root"
> server but will cache the content upstream.

I disagree. That's not what the challenge is about.
Why don't just try it and avoid (useless) questions!

> How fast you can serve up dynamic content is far more interesting,
> since it can't be cached.

This will be part of my second challenge of course.
For now, let's stick with the first challenge rules please.

Regards
/Zab
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Amy Lear
On Wed, Jul 6, 2011 at 2:45 AM, Zabrane Mickael <[hidden email]> wrote:
>
> > Why measure this? Serving static content is one of the least
> > interesting things I can think of
> > to study. Any network worth the name won't bother to access the "root"
> > server but will cache the content upstream.
>
> I disagree. That's not what the challenge is about.
> Why don't just try it and avoid (useless) questions!

These aren't useless questions. They're genuine concerns about the
validity and applicability of your 'challenge'.

As Joe notes, anything that doesn't have a short TTL can be presumed
to be cached for at least some of the clients at least some of the
time. In some cases, it could be all of the time for some of the
clients; most significant ISPs make some attempt at this to limit
outside bandwidth. Caching within the network is often cheaper, after
all.

For some time now, the useful places to be applying effort are
non-trivial results, where the actual content being received is either
in flux, or a direct result of the client's actions.

You're wanting to measure something that's inaccurate, unnecessary,
and uninformative.

Could you give us a problem space defininition covering what exactly
you're trying to achieve?
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Jesper Louis Andersen-2
In reply to this post by Zabrane Mickael
On Mon, Jul 4, 2011 at 17:11, Zabrane Mickael <[hidden email]> wrote:

> The idea is ver simple: measure "how fast" you can serve a "100 bytes" file
> (attached as "small.html")

What are you trying to show with this benchmark? That is, what is the
goal of it? Serving a static file is something nginx and a couple of
Haskell HTTP servers are very good at (see the Warp-project). The main
problem is that what you are trying to show might not be reflected
very well from your test.

> The rules are very simple :

> 4. The HTTP client to use for this benchmark needs to be "Apache AB"
> (http://httpd.apache.org/docs/2.0/programs/ab.html) with:
> $ ab -n 200000 -c 100 -t1 -k "http://127.0.0.1:8080/small.html"

This violates Rule 1 in Mark Nottingham's http benchmark advice:

http://www.mnot.net/blog/2011/05/18/http_benchmark_rules

Quote: "The most common mistake I see people making is benchmarking a
server on the same box where the load is generated. This doesn’t just
put your results out a little bit, it makes them completely
unreliable, because the load generator’s “steal” of resources varies
depending on how the server handles the load, which depends on
resource availability."

Is there any specific reason why you want to break that advice? Also,
Steve Vinoski did a talk at the recent Erlang Factory on Yaws and the
last part are benchmark and performance advice:

http://www.erlang-factory.com/upload/presentations/409/vinoski-ef-london-2011.pdf

I am with Joe here. Dynamically generated content is much more
interesting for real problems. Static files can be outsourced to a CDN
or Varnish easily. The reason you still serve them from the same
webserver is convenience.

--
J.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Zabrane Mickael
What are you trying to show with this benchmark? That is, what is the
goal of it?

At the end of the day, I'll try to compare Erlang Web servers with others
competitors (not written in Erlang).

Serving a static file is something nginx and a couple of
Haskell HTTP servers are very good at (see the Warp-project).

Please, just follow and stick to the rules.
Nginx or Varnish (https://www.varnish-cache.org/) or any HTTP accelerator are out of this challenge scope.

This violates Rule 1 in Mark Nottingham's http benchmark advice:
http://www.mnot.net/blog/2011/05/18/http_benchmark_rules

I'm aware of that rule (Steve Vinoski post: http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking/). 
Thanks anyway.

Why it's so hard to just play the game guys?

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Ivan Uemlianin
On 06/07/2011 12:42, Zabrane Mickael wrote:

>> What are you trying to show with this benchmark? That is, what is the
>> goal of it?
>
> At the end of the day, I'll try to compare Erlang Web servers with others
> competitors (not written in Erlang).
>
>> Serving a static file is something nginx and a couple of
>> Haskell HTTP servers are very good at (see the Warp-project).
>
> Please, just follow and stick to the rules.
> Nginx or Varnish (https://www.varnish-cache.org/) or any HTTP
> accelerator are out of this challenge scope.

Why is nginx outside of the challenge scope?  If you want "to compare
Erlang Web servers with other competitors (not written in Erlang)",
surely nginx is one of the main competitors?

Best wishes

Ivan


--
============================================================
Ivan A. Uemlianin
Speech Technology Research and Development

                     [hidden email]
                      www.llaisdy.com
                          llaisdy.wordpress.com
                      www.linkedin.com/in/ivanuemlianin

     "Froh, froh! Wie seine Sonnen, seine Sonnen fliegen"
                      (Schiller, Beethoven)
============================================================
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Zabrane Mickael

Why is nginx outside of the challenge scope?

Nginx or anything else are out of scope[1] at this time.
This will not prevent you to write a simple cache in (pure) Erlang for the challenge.

 If you want "to compare Erlang Web servers with other competitors (not written in Erlang)", surely nginx is one of the main competitors?

Yes, but Nginx is written in C.

[1] I'm planning to ask the authors of the other tools (like Cherokee, Apache2, Lighttpd + Nginx, Varnish, ...)
and let them know about the challenge (with the same rules of course). 

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Joe Armstrong-2
In reply to this post by Zabrane Mickael
On Wed, Jul 6, 2011 at 1:42 PM, Zabrane Mickael <[hidden email]> wrote:

> What are you trying to show with this benchmark? That is, what is the
> goal of it?
>
> At the end of the day, I'll try to compare Erlang Web servers with others
> competitors (not written in Erlang).
>
> Serving a static file is something nginx and a couple of
> Haskell HTTP servers are very good at (see the Warp-project).
>
> Please, just follow and stick to the rules.
> Nginx or Varnish (https://www.varnish-cache.org/) or any HTTP accelerator
> are out of this challenge scope.
>
> This violates Rule 1 in Mark Nottingham's http benchmark advice:
> http://www.mnot.net/blog/2011/05/18/http_benchmark_rules
>
> I'm aware of that rule (Steve Vinoski
> post: http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking/).
> Thanks anyway.
> Why it's so hard to just play the game guys?

Because if you propose a new game then people will question and try to
change the rules.

That's how baseball got invented, and I know cricket is far better,
but that is a bit off-topic here.

In the particular case just about the last thing I want to know about
is how fast I can serve static content - there is an entire industry devoted
to exactly this question.

Erlang was designed for building fault-tolerant systems, it was *not* designed
for building fast systems. I would expect therefore that it would do badly
at things that non-fault tolerant systems are good at (like serving
static content).

(( I'm sure we could build a fast content delivery network in Erlang -
but this is not easy - the low hanging
fruit was taken year ago - and it's got nothing to do with Erlang, its
all about algorithms, caching, locality
peering agreements  etc.))

Like it or not, the results of such a measurement will be
misinterpreted and abused.
The fastest ever Erlang web server should be way slower than the
fastest non-fault
tolerant web server.

A result of the form "the fastest Erlang web server is way slower than
the fastest non fault-tolerant
server" will be widely quoted and blogged, and the results will be
taken out-of-context.

For me I like to "make things work" before I "make them fast" - so the
question of exactly what happens
under stress, just before things fail is extremely interesting.

I am very interested to know exactly how and when things fail when you
stress them.

Sending 100 bytes from a file seems to measure nothing interesting.
Since it's the same file
I expect it to be read *once* then cached in memory. It's so small
that I expect no IP fragmentation at all.

You won't really be measuring the performance of the web server, but
rather the perfomance of
a tiny sub-set of gen_tcp and file. If the authors of the various web
serveres havn't made some catestripohuic
mistales in their implamentaion I expect their code to be just a thin
layer of the standard librarie rountines and
I expect the results to be broadly similar.


I always like to ask the question:

     What would you do with the results of such a measurement?

Because unless you can give a good answer to this, there does not seem
much point in doing the experiment.

/joe


> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Ivan Uemlianin
In reply to this post by Zabrane Mickael
Most of the web server benchmarks I've come across on the web suffer
from the same flaw: a partisan of server X tests a highly tuned and
optimised install of their favourite server against default installs of
competitor servers.  These kinds of benchmarks are really not worth
anything.

I think it's a good idea to set a challenge, and invite server
developers to show how their own project can meet the challenge.

The first step is to devise a challenge, and this is not a trivial step.
  This step is actually listing the essential functional requirements of
a web server.  The ability to serve a static file promptly from disk is
one requirement, but these days I shouldn't think it's the only or even
the main one.

I think it might be interesting to agree on a small set of requirements
and then devise a challenge which could test those requirements.

Best wishes

Ivan


On 06/07/2011 12:59, Zabrane Mickael wrote:

>>>
>> Why is nginx outside of the challenge scope?
>
> Nginx or anything else are out of scope[1] at this time.
> This will not prevent you to write a simple cache in (pure) Erlang for
> the challenge.
>
>> If you want "to compare Erlang Web servers with other competitors (not
>> written in Erlang)", surely nginx is one of the main competitors?
>
> Yes, but Nginx is written in C.
>
> [1] I'm planning to ask the authors of the other tools (like Cherokee,
> Apache2, Lighttpd + Nginx, Varnish, ...)
> and let them know about the challenge (with the same rules of course).


--
============================================================
Ivan A. Uemlianin
Speech Technology Research and Development

                     [hidden email]
                      www.llaisdy.com
                          llaisdy.wordpress.com
                      www.linkedin.com/in/ivanuemlianin

     "Froh, froh! Wie seine Sonnen, seine Sonnen fliegen"
                      (Schiller, Beethoven)
============================================================
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Joe Armstrong-2
On Wed, Jul 6, 2011 at 2:18 PM, Ivan Uemlianin <[hidden email]> wrote:

> Most of the web server benchmarks I've come across on the web suffer from
> the same flaw: a partisan of server X tests a highly tuned and optimised
> install of their favourite server against default installs of competitor
> servers.  These kinds of benchmarks are really not worth anything.
>
> I think it's a good idea to set a challenge, and invite server developers to
> show how their own project can meet the challenge.
>
> The first step is to devise a challenge, and this is not a trivial step.
>  This step is actually listing the essential functional requirements of a
> web server.  The ability to serve a static file promptly from disk is one
> requirement, but these days I shouldn't think it's the only or even the main
> one.

I know that sounds fine, but it's the non-functional requirements that
always seem
to make or break a project.

   How good is the documentation?
   How mature is the product?
   How easy is it to maintain?
   Can I understand how the thing works?
   Is it covered by the blue-river-snowy public license?
   ...

The answers to these questions are just not quantifiable.

Not being covered by the blue-river-snowy license and having no
documentation would be a very good reason not to use a web server
even if it could serve pages faster than a shark can bite off your leg.


/Joe

>
> I think it might be interesting to agree on a small set of requirements and
> then devise a challenge which could test those requirements.
>
> Best wishes
>
> Ivan
>
>
> On 06/07/2011 12:59, Zabrane Mickael wrote:
>>>>
>>> Why is nginx outside of the challenge scope?
>>
>> Nginx or anything else are out of scope[1] at this time.
>> This will not prevent you to write a simple cache in (pure) Erlang for
>> the challenge.
>>
>>> If you want "to compare Erlang Web servers with other competitors (not
>>> written in Erlang)", surely nginx is one of the main competitors?
>>
>> Yes, but Nginx is written in C.
>>
>> [1] I'm planning to ask the authors of the other tools (like Cherokee,
>> Apache2, Lighttpd + Nginx, Varnish, ...)
>> and let them know about the challenge (with the same rules of course).
>
>
> --
> ============================================================
> Ivan A. Uemlianin
> Speech Technology Research and Development
>
>                    [hidden email]
>                     www.llaisdy.com
>                         llaisdy.wordpress.com
>                     www.linkedin.com/in/ivanuemlianin
>
>    "Froh, froh! Wie seine Sonnen, seine Sonnen fliegen"
>                     (Schiller, Beethoven)
> ============================================================
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

David Goehrig
In reply to this post by Zabrane Mickael

On Jul 6, 2011, at 5:45 AM, Zabrane Mickael <[hidden email]> wrote:


I disagree. That's not what the challenge is about.
Why don't just try it and avoid (useless) questions!

Because your "benchmark" is meaningless without proper controls and variables. This supposed challenge characterizes nothing, because you've not defined any meaningful constraints on the system. 

A meaningful benchmark measures changes in response to one or more environmental pressures. For example, you can measure OS process starvation's effect on tcp backlog across each server with memory, io, CPU load, and payload all remaining constant. 

Similarly you can look at performance with respect to slow connections, high packet loss, CPU starvation, memory starvation, file descriptor starvation, and io blocking. And just about any other variable you can control for. 

Throughput will vary as a function of all these factors and will demonstrate a lot of important issues for each system design. 

So to answer your question, the best reason not to try the challenge is because it is a terrible waste of time that could be better spent properly characterizing a production system. 

Dave

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Jesper Louis Andersen-2
In reply to this post by Zabrane Mickael
On Wed, Jul 6, 2011 at 13:42, Zabrane Mickael <[hidden email]> wrote:
> What are you trying to show with this benchmark? That is, what is the
> goal of it?
>
> At the end of the day, I'll try to compare Erlang Web servers with others
> competitors (not written in Erlang).

That was not the answer I had in mind when phrasing the question. I'll
try to be more precise: What are you going to measure? If you are
going to count requests, then the problem is that a 100 bytes transfer
is way less than the size of the average HTTP header. In other words,
you will be measuring who cheats and does not provide full headers as
per the HTTP spec. If you are going to measure throughput, on a single
machine, then 'ab' and whatever web server you have will fight for the
resources and thus show nothing at all.

What I am after is why you think your benchmark will give you any
indication of the goal you had in mind. What is performance to you? I
am interested in your answer because it is important to me if I want
to use the results for anything. It may be flawed or not, that is not
the concern for me. I am interested in the reason for why you laid out
the rules of the game as your did.

I disagree that you can see web servers as competitors to each other.
There is always a greater purpose behind writing a web server. From
the point where it is a personal exercise (can I do it?), to providing
a web server in your favorite language for ease of integration, to
writing a server because you want to earn money. It is not as much a
competition as it is a convenience of use. Or said otherwise, many of
the competitors are not even aware they are competing :)

> I'm aware of that rule (Steve Vinoski
> post: http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking/).
> Thanks anyway.

What are you going to do about it? Those who will have multiple CPUs
in their machine and pins 'ab' to one and gives Erlang free reign over
the others will definitely be at an advantage here. In fact, those who
disable SMP from the Erlang runtime might also be faster. If you ran
many tests and then used a bootstrapping technique, then you might be
able to tell if the localhost test generator tool interferes too much
with the results for instance. That is one way to circumvent that
problem.

> Why it's so hard to just play the game guys?

We are playing the game on another level. Our game is "The scientific
method and experimental testing". To be precise, I think your test has
way to many confounding factors to underpin your hypothesis that your
experiment can be used to show anything about web servers in
comparison to each other. When you design a simple experiment, it is
mandatory in our game to point out exactly what the confounding
factors are and what trouble there will be when drawing conclusions
from the test itself. I have pointed out some factors and Joe has
pointed out some different factors. That is all.

--
J.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

dmercer
In reply to this post by Joe Armstrong-2
On Wednesday, July 06, 2011, Joe Armstrong wrote:

> Erlang was designed for building fault-tolerant systems, it was *not*
> designed
> for building fast systems

+1.  Salient point worth repeating.  This often gets forgotten.

> The fastest ever Erlang web server should be way slower than the
> fastest non-fault
> tolerant web server.

Cheers,

DBM

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang Web Servers challenge

Garrett Smith-5
Interesting thread. Reminds me of a conversation I overhead, in
different time and place:

http://www.youtube.com/watch?v=xEJ1n13soWU

Very slightly NSFW language.

Garrett

P.S. I hope this isn't viewed as piling on. I just couldn't resist.
This is absolutely not personally directed. It's just a little fun
poking at a certain ideology that we've all been guilty of adopting :)

On Wed, Jul 6, 2011 at 10:08 AM, David Mercer <[hidden email]> wrote:

> On Wednesday, July 06, 2011, Joe Armstrong wrote:
>
>> Erlang was designed for building fault-tolerant systems, it was *not*
>> designed
>> for building fast systems
>
> +1.  Salient point worth repeating.  This often gets forgotten.
>
>> The fastest ever Erlang web server should be way slower than the
>> fastest non-fault
>> tolerant web server.
>
> Cheers,
>
> DBM
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Erlang Web Servers challenge

Steve Davis-2
In reply to this post by Zabrane Mickael
I'm thinking that the real comparison that everyone would like to see
(obviously) would be to build a full web application infrastructure
using various current technologies and with all the regression bases
covered and major stress tests.... for an app like facebook.

Of course, you'd need to use the best expertise from each technology
domain to make sure the comparison was even slightly "fair".

And of course, this would require such a monumental investment it
could never be done (what's more most all platforms never manage to
test themselves until put into the wild).

So we are left making value judgements over and over again.

So I'm thinking: What exactly is the value of this test beyond serving
favicons (i'd choose nginx for that probably)?

My negative 2c. :(

/s

On Jul 4, 10:11?am, Zabrane Mickael <zabra...> wrote:

> Hi guys,
>
> I'd like your help for this small Web servers challenge.
>
> The idea is ver simple: measure "how fast" you can serve a "100 bytes" file (attached as "small.html")
> in "less than 01 second" by using one of the following web servers:
>
> - Yaws (http://yaws.hyber.org/download/yaws-1.90.tar.gz)
> - Mochiweb (https://github.com/mochi/mochiweb)
> - Misultin (https://github.com/ostinelli/misultin)
> - Cowboy (https://github.com/extend/cowboy)
> - Bare metal web server using "gen_tcp"
>
> The rules are very simple :
>
> 1. The first time you served the file, you MUST read it from disk.
> All subsequent requests to that file can use any caching mechanism!
>
> 2. Any (hidden) trick is allowed (eg. prim_inet, SMP enabled ...)
>
> 3. Erlang versions R14B to R14B03
>
> 4. The HTTP client to use for this benchmark needs to be "Apache AB" (http://httpd.apache.org/docs/2.0/programs/ab.html) with:
> $ ab -n 200000 -c 100 -t1 -k "http://127.0.0.1:8080/small.html"
>
> I hope that the authors if these web servers will also contribute to this bench as they
> know their software better than anybody else.
>
> Challenge's open ;-)
>
> Regards,
> Zabrane
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questi...://erlang.org/mailman/listinfo/erlang-questions


12