Amazon AWS benchmarking of erlang-diameter not upto the mark

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Amazon AWS benchmarking of erlang-diameter not upto the mark

Arshad Ansari
Hello there,

Today, as a part of evalution of erlang diameter client, relay & server, I was benchmarking for requests per second. This same test was performed by me on my local network and I was able to get 30K requests per second between client and server using a single connection. However, on Aws I got to only about 11-12K requests per second. I had used c3.x4large instance with 16 cores and 30 GB ram for both client and server, which is 4 times the core I have and twice the ram I had when I was testing locally. These are compute enhanced instances to make sure I get to use core to the maximum, but I wasn't even using 60% of cores on server and 80% of cores on client.  I even used the same placement group to get 10GBps network bandwidth. I can't think of any reason why there would be just a downgrade. Has anyone experienced something of this sort on Aws while benchmarking erlang?

Thanks in advance for the help.

Regards,
Arshad

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Amazon AWS benchmarking of erlang-diameter not upto the mark

dmkolesnikov
Hello,

You’ve experienced difference due to network configuration. CPU factor is important but I’ll spent more time to study network config.

You have not defined to us your local network config, its latency, used operating system, networking kernel options and Erlang flags. Similarly, your AWS config is not know to us either.

Best Regards,
Dmitry


> On Aug 29, 2016, at 6:55 PM, Arshad Ansari <[hidden email]> wrote:
>
> Hello there,
>
> Today, as a part of evalution of erlang diameter client, relay & server, I was benchmarking for requests per second. This same test was performed by me on my local network and I was able to get 30K requests per second between client and server using a single connection. However, on Aws I got to only about 11-12K requests per second. I had used c3.x4large instance with 16 cores and 30 GB ram for both client and server, which is 4 times the core I have and twice the ram I had when I was testing locally. These are compute enhanced instances to make sure I get to use core to the maximum, but I wasn't even using 60% of cores on server and 80% of cores on client.  I even used the same placement group to get 10GBps network bandwidth. I can't think of any reason why there would be just a downgrade. Has anyone experienced something of this sort on Aws while benchmarking erlang?
>
> Thanks in advance for the help.
>
> Regards,
> Arshad
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Amazon AWS benchmarking of erlang-diameter not upto the mark

Arshad Ansari
Hello Dmitry,

On my localbox I had a simple lan configuration with a single gateway/switch to internet, so I'm not sure what else there is to mention about the lan config. However; the machine configurations are as below:
1. Server-1 [4 core 4 GB]
2. Server-2 [4 core 4 GB]
3. Relay [Works as a load Balancer] [4 core 4 GB]
4. Client 1 [4 core 4 GB]
5. Client 2 [4 core 4 GB]

Each machine runs a single node as mentioned. There is a single tcp socket connection between peers where alll Clients connect to Relay and Relay in turn connects with the 2 servers and acts a proxy while also balancing the request between the two servers using round robin algorithm. This connection is used to send all the request using the gen_server:call (blocking) method. Following is the erl vm params I'm using for all of the nodes:

erl +A 1024 +P 134217727 +Q 134217727 -env ERL_MAX_PORTS 134217727 +K true 

I'm able to get around 26K req/sec on relay which divides the requests between two servers around 13K req/sec. Each and every node was utilizing near about 99% of all the cores available on the system. This I was able to get on my local network.

As for AWS, I had the following setup:

1. Server - 1 [4 core 16GB]
2. Server - 2 [4 core 16GB]
3. Relay [16 core 30GB]
4. Client [16 core 30GB]

I had launched four of them as part of single placement group to remove bandwidth restriction. When I did run the nodes, the processors were not being utilized more than 60% and yet the relay was getting around 12K req/sec which it was dividing between servers with each one getting around 6K req/sec. Erlang VM args were the same as mentioned above. I was using internal private IPs on AWS to connect the peers with one another to avoid bandwidth limitation that comes with public IP usage. Each of the aws instance has EBS volume attached to it. The only disk access I'm doing is to write log at the per second rate, besides accessing the ETS table for getting counter instances from each instance. 

I hope I've covered all there is to the configurations and also that I was clear enough. 

Thanks again,
Regards,
Arshad

On Tue, Aug 30, 2016 at 12:34 AM Dmitry Kolesnikov <[hidden email]> wrote:
Hello,

You’ve experienced difference due to network configuration. CPU factor is important but I’ll spent more time to study network config.

You have not defined to us your local network config, its latency, used operating system, networking kernel options and Erlang flags. Similarly, your AWS config is not know to us either.

Best Regards,
Dmitry


> On Aug 29, 2016, at 6:55 PM, Arshad Ansari <[hidden email]> wrote:
>
> Hello there,
>
> Today, as a part of evalution of erlang diameter client, relay & server, I was benchmarking for requests per second. This same test was performed by me on my local network and I was able to get 30K requests per second between client and server using a single connection. However, on Aws I got to only about 11-12K requests per second. I had used c3.x4large instance with 16 cores and 30 GB ram for both client and server, which is 4 times the core I have and twice the ram I had when I was testing locally. These are compute enhanced instances to make sure I get to use core to the maximum, but I wasn't even using 60% of cores on server and 80% of cores on client.  I even used the same placement group to get 10GBps network bandwidth. I can't think of any reason why there would be just a downgrade. Has anyone experienced something of this sort on Aws while benchmarking erlang?
>
> Thanks in advance for the help.
>
> Regards,
> Arshad
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions