Quantcast

Is Erlang a good tool for this particular project?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Is Erlang a good tool for this particular project?

Matthew Hillsborough
Greetings Erlang community,

Let me further elaborate on my question that's in the subject of this message. I tried to reach out with this question on StackOverflow, however I did not have much luck there. Perhaps the community here can provide some feedback here for me to let me know if I'm on the right track or if Erlang is not the right tool for what I'm trying to accomplish.

I'm building native mobile applications in both iOS and Android. These apps require "realtime" updates from and to the server, same as many (but not all) network-based application does (Facebook, social games like Words with Friends, Finance applications, etc). The communication here is bi-directional, in the sense that the server might have updates for the mobile clients and the clients will be pushing data down to the server whenever necessary.

I think using HTTP long polling for this is over kill in the sense that long polling can be detrimental to battery life, especially with a lot of TCP setup/teardown for every HTTP connection the device needs to send out through the wire. It might make sense to have the mobile applications use persistent TCP sockets to establish a connection to the server, and send RPC style commands to the server for all web service communication. This ofcourse, would require a server to handle the long-lived TCP connection and be able to speak to a web service once it makes sense of the data passed down the TCP pipe. I'm thinking of passing data in plain text using JSON or XML and then using some kind of Erlang interface to HTTP to call a web service to handle all the REST type communication. The responses would then go back to the "RPC" Erlang instance, which would send the updates to the appropriate client(s).

Perhaps an Erlang based RPC server would do well for a network based application like this. It would allow for the mobile apps to send and receive data from the server all over one connection without multiple setup/tear down that individual HTTP requests would do. Since no web browser is involved, we do not need to deal with the nuances of HTTP long-polling at the mobile client level. I also haven't seen great long polling/keep-alive support on the client-side in iOS, but that's irrelevant for the community here.

A lot of these "COMET" and long-polling/streaming servers are built with HTTP in mind. I'm thinking just using a plain-text protocol over TCP is better catered for the type of app I'm building, will make the client more responsive, allow for receiving of updates from the server without constantly polling the server, etc.

I also looked into HTTP pipelining, but it doesn't look to be worth the trouble when it comes to implementing it on the clients. Also, I'm not sure if it would allow for bi-directional communication in the client<->server communication channel. 

Am I completely out of line in thinking that building a custom solution in Erlang is a good idea here? To my understanding, Erlang excels at servers like this, and if I run the server on tcp/80, I should be able to avoid most firewall/port issues. The client would need work to deal with timeouts, re connections, acknowledging receipt of asynchronous requests, but that's not Erlang's problem. 

Has anyone built something similar before? Should I just stick to a web server and deal with "COMET" type technologies? (WebSockets, long-polling, client-side polling).

Was hoping someone could solidify that I'm not entirely insane for wanting a better solution than HTTP would serve in this case, at least at the client level. I'll still be using HTTP/REST extensively, the Erlang server would just handle the persistent connections and messaging to the Web Service (which would probably be something like Django or Rails).

Sorry for the long post; I am just excited to get into the heads of people who are smarter than I.

Happy hacking!

Matthew



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Ale
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Ale
Hi,

Just a thought here... Erlang is great for scalability and handling
multiple connections, and well I'm sure a lot of people can give you a
better description fo this, but it occurred to me if you considered
other protocols instead of HTTP? The problem seems to be there rather
than the language you write your sever. It occurred to me that you
might benefit from using Jabber/XMPP. Googling I found some iOS jabber
clients http://code.google.com/p/xmppframework/ and the cannonical
jabber server is written in Erlang.

Regards,

2011/7/12 Matthew Hillsborough <[hidden email]>:

> Greetings Erlang community,
> Let me further elaborate on my question that's in the subject of this
> message. I tried to reach out with this question on StackOverflow, however I
> did not have much luck there. Perhaps the community here can provide some
> feedback here for me to let me know if I'm on the right track or if Erlang
> is not the right tool for what I'm trying to accomplish.
>
> I'm building native mobile applications in both iOS and Android. These apps
> require "realtime" updates from and to the server, same as many (but not
> all) network-based application does (Facebook, social games like Words with
> Friends, Finance applications, etc). The communication here is
> bi-directional, in the sense that the server might have updates for the
> mobile clients and the clients will be pushing data down to the server
> whenever necessary.
>
> I think using HTTP long polling for this is over kill in the sense that long
> polling can be detrimental to battery life, especially with a lot of TCP
> setup/teardown for every HTTP connection the device needs to send out
> through the wire. It might make sense to have the mobile applications use
> persistent TCP sockets to establish a connection to the server, and send RPC
> style commands to the server for all web service communication. This
> ofcourse, would require a server to handle the long-lived TCP connection and
> be able to speak to a web service once it makes sense of the data passed
> down the TCP pipe. I'm thinking of passing data in plain text using JSON or
> XML and then using some kind of Erlang interface to HTTP to call a web
> service to handle all the REST type communication. The responses would then
> go back to the "RPC" Erlang instance, which would send the updates to the
> appropriate client(s).
>
> Perhaps an Erlang based RPC server would do well for a network based
> application like this. It would allow for the mobile apps to send and
> receive data from the server all over one connection without multiple
> setup/tear down that individual HTTP requests would do. Since no web browser
> is involved, we do not need to deal with the nuances of HTTP long-polling at
> the mobile client level. I also haven't seen great long polling/keep-alive
> support on the client-side in iOS, but that's irrelevant for the community
> here.
>
> A lot of these "COMET" and long-polling/streaming servers are built with
> HTTP in mind. I'm thinking just using a plain-text protocol over TCP is
> better catered for the type of app I'm building, will make the client more
> responsive, allow for receiving of updates from the server without
> constantly polling the server, etc.
>
> I also looked into HTTP pipelining, but it doesn't look to be worth the
> trouble when it comes to implementing it on the clients. Also, I'm not sure
> if it would allow for bi-directional communication in the client<->server
> communication channel.
>
> Am I completely out of line in thinking that building a custom solution in
> Erlang is a good idea here? To my understanding, Erlang excels at servers
> like this, and if I run the server on tcp/80, I should be able to avoid most
> firewall/port issues. The client would need work to deal with timeouts, re
> connections, acknowledging receipt of asynchronous requests, but that's not
> Erlang's problem.
>
> Has anyone built something similar before? Should I just stick to a web
> server and deal with "COMET" type technologies? (WebSockets, long-polling,
> client-side polling).
>
> Was hoping someone could solidify that I'm not entirely insane for wanting a
> better solution than HTTP would serve in this case, at least at the client
> level. I'll still be using HTTP/REST extensively, the Erlang server would
> just handle the persistent connections and messaging to the Web Service
> (which would probably be something like Django or Rails).
>
> Sorry for the long post; I am just excited to get into the heads of people
> who are smarter than I.
>
> Happy hacking!
>
> Matthew
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>



--
Ale.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Miguel Morales
In reply to this post by Matthew Hillsborough
I agree with the above poster, sounds like something like XMPP would
be good for.  There is a smack library for Android that I've heard
good things about.  When I was starting, because there was nothing
specific to mobile devices I ended up using RabbitMQ and its Java
bindings to build a realtime service that users could push/subscribe
to.

The way I do it, is on the server side I have some queue setup and a
bunch of worker processes running.
When the server gets a new message from the clients, one of the
workers processes it and updates the client information.  If the
workers are overloaded, the message is left in the queue until it can
be processed.

Polling on a device will shorten the battery life.  From some tests
I've ran (and I suspect is how google implements its C2DM service) it
appears that having a socket open doesn't activate the radio, it's
only activated until data is ready to be sent or read.

You can also implement RPC with a MQ server.  Although, it's not
strictly bi-directional.


On Tue, Jul 12, 2011 at 1:25 PM, Matthew Hillsborough
<[hidden email]> wrote:

> Greetings Erlang community,
> Let me further elaborate on my question that's in the subject of this
> message. I tried to reach out with this question on StackOverflow, however I
> did not have much luck there. Perhaps the community here can provide some
> feedback here for me to let me know if I'm on the right track or if Erlang
> is not the right tool for what I'm trying to accomplish.
>
> I'm building native mobile applications in both iOS and Android. These apps
> require "realtime" updates from and to the server, same as many (but not
> all) network-based application does (Facebook, social games like Words with
> Friends, Finance applications, etc). The communication here is
> bi-directional, in the sense that the server might have updates for the
> mobile clients and the clients will be pushing data down to the server
> whenever necessary.
>
> I think using HTTP long polling for this is over kill in the sense that long
> polling can be detrimental to battery life, especially with a lot of TCP
> setup/teardown for every HTTP connection the device needs to send out
> through the wire. It might make sense to have the mobile applications use
> persistent TCP sockets to establish a connection to the server, and send RPC
> style commands to the server for all web service communication. This
> ofcourse, would require a server to handle the long-lived TCP connection and
> be able to speak to a web service once it makes sense of the data passed
> down the TCP pipe. I'm thinking of passing data in plain text using JSON or
> XML and then using some kind of Erlang interface to HTTP to call a web
> service to handle all the REST type communication. The responses would then
> go back to the "RPC" Erlang instance, which would send the updates to the
> appropriate client(s).
>
> Perhaps an Erlang based RPC server would do well for a network based
> application like this. It would allow for the mobile apps to send and
> receive data from the server all over one connection without multiple
> setup/tear down that individual HTTP requests would do. Since no web browser
> is involved, we do not need to deal with the nuances of HTTP long-polling at
> the mobile client level. I also haven't seen great long polling/keep-alive
> support on the client-side in iOS, but that's irrelevant for the community
> here.
>
> A lot of these "COMET" and long-polling/streaming servers are built with
> HTTP in mind. I'm thinking just using a plain-text protocol over TCP is
> better catered for the type of app I'm building, will make the client more
> responsive, allow for receiving of updates from the server without
> constantly polling the server, etc.
>
> I also looked into HTTP pipelining, but it doesn't look to be worth the
> trouble when it comes to implementing it on the clients. Also, I'm not sure
> if it would allow for bi-directional communication in the client<->server
> communication channel.
>
> Am I completely out of line in thinking that building a custom solution in
> Erlang is a good idea here? To my understanding, Erlang excels at servers
> like this, and if I run the server on tcp/80, I should be able to avoid most
> firewall/port issues. The client would need work to deal with timeouts, re
> connections, acknowledging receipt of asynchronous requests, but that's not
> Erlang's problem.
>
> Has anyone built something similar before? Should I just stick to a web
> server and deal with "COMET" type technologies? (WebSockets, long-polling,
> client-side polling).
>
> Was hoping someone could solidify that I'm not entirely insane for wanting a
> better solution than HTTP would serve in this case, at least at the client
> level. I'll still be using HTTP/REST extensively, the Erlang server would
> just handle the persistent connections and messaging to the Web Service
> (which would probably be something like Django or Rails).
>
> Sorry for the long post; I am just excited to get into the heads of people
> who are smarter than I.
>
> Happy hacking!
>
> Matthew
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>



--
~ Jeremiah:9:23-24
Android 2D MMORPG: http://solrpg.com/http://www.youtube.com/user/revoltingx
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Tim Watson-5
On 12 July 2011 22:32, Miguel Morales <[hidden email]> wrote:
> I agree with the above poster, sounds like something like XMPP would
> be good for.  There is a smack library for Android that I've heard
> good things about.  When I was starting, because there was nothing
> specific to mobile devices I ended up using RabbitMQ and its Java
> bindings to build a realtime service that users could push/subscribe
> to.

Also, if your target environment (e.g., IOS) makes HTTP necessary, due
to protocol restrictions or whatever, most of these messaging and/or
integration technologies tend to support some kind of transport over
HTTP anyway. So for example, you can do XMPP over an HTTP connection
using XMPP-over-BOSH (which is a long polling technique), rabbitmq has
a Stomp plugin and there are probably other HTTP interfaces I don't
know of, and so on.

But in general, if you're not being forced to use HTTP, then as the
other posters have mentioned it might be worth considering the
alternatives.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Joe Armstrong-2
In reply to this post by Ale
On Tue, Jul 12, 2011 at 11:03 PM, Ale <[hidden email]> wrote:
> Hi,
>
> Just a thought here... Erlang is great for scalability and handling
> multiple connections, and well I'm sure a lot of people can give you a
> better description fo this, but it occurred to me if you considered
> other protocols instead of HTTP?

YES - YES - YES

Listen carefully - radio bandwidth is *extremely* limited - if you are
making an andriod
application you should weigh each byte on golden scales.

At the moment the demand curve for radio bandwidth is far steeper than
the supply curve.
We are running out of bandwidth fast - which is why I can't read
aftonbladet on the underground
any more - the first page takes longer than the journey time. If I was
the only person
with a smart phone that would be fine - but I'm not. Even the
low-bandwidth text applications
crawl along when in on the underground. All of this (looking at
aftonbladet etc) when I'm out of
the congested zones.

We (Ericsson) are churning out radio base stations as fast as we can
manufacture them, the
demand is insatiable.

As people move to smart phones they use c. 4 x the bandwidth they used
prior to smartphones.

Within one 3G cell there is a total of c. 2.4 MBits/second that must
be shared by *all* users in the cell.
bandwidth is not unlimited it is a finite and limited resource. 4G
makes life better with better protocols and
encoding methods.

Bandwidth (ie the 2.4 MBits) is a limited *shared* resource - this not
fibre - this is a limitation
reflecting the laws of physics and has nothing to do with engineering.

Since bandwidth is a limited resource people will get really pissed
off when they get bad response times.
As the 3G network get congested they drop back to GRPS/edge etc and do
packet switching which places even
more load on the networks.

So the rule for mobile apps is

Cache as much as possible - send and receive as little as possible and
weigh each sent and received
bytes on golden scales.

Bandwidth and bytes sent/received costs money. Most accounts cap data sent/month
so even though you have unlimited access they may well cut you back to
64KB/s after you have sent the
first Gbits of data.


If I were building an adriod app I'd do as follows:

1) Cache as much as possible when bandwidth is high and cheap
    Most phones have dual access - when they come into WiFi range
pro-actively cache as much as possible

    If you assume that the cost of sending data is a few hundred times
the cost of caching the data
    then you won't go far wrong


2) When you get congested weigh every byte on golden scales.

Do not send HTML with all those unnecessary headers:

Be proactive - It will be years before the standards change and HTML
is dumped for radio access - the good
reactive applications will use internal non-standard protocols.


/Joe



> The problem seems to be there rather
> than the language you write your sever. It occurred to me that you
> might benefit from using Jabber/XMPP. Googling I found some iOS jabber
> clients http://code.google.com/p/xmppframework/ and the cannonical
> jabber server is written in Erlang.
>
> Regards,
>
> 2011/7/12 Matthew Hillsborough <[hidden email]>:
>> Greetings Erlang community,
>> Let me further elaborate on my question that's in the subject of this
>> message. I tried to reach out with this question on StackOverflow, however I
>> did not have much luck there. Perhaps the community here can provide some
>> feedback here for me to let me know if I'm on the right track or if Erlang
>> is not the right tool for what I'm trying to accomplish.
>>
>> I'm building native mobile applications in both iOS and Android. These apps
>> require "realtime" updates from and to the server, same as many (but not
>> all) network-based application does (Facebook, social games like Words with
>> Friends, Finance applications, etc). The communication here is
>> bi-directional, in the sense that the server might have updates for the
>> mobile clients and the clients will be pushing data down to the server
>> whenever necessary.
>>
>> I think using HTTP long polling for this is over kill in the sense that long
>> polling can be detrimental to battery life, especially with a lot of TCP
>> setup/teardown for every HTTP connection the device needs to send out
>> through the wire. It might make sense to have the mobile applications use
>> persistent TCP sockets to establish a connection to the server, and send RPC
>> style commands to the server for all web service communication. This
>> ofcourse, would require a server to handle the long-lived TCP connection and
>> be able to speak to a web service once it makes sense of the data passed
>> down the TCP pipe. I'm thinking of passing data in plain text using JSON or
>> XML and then using some kind of Erlang interface to HTTP to call a web
>> service to handle all the REST type communication. The responses would then
>> go back to the "RPC" Erlang instance, which would send the updates to the
>> appropriate client(s).
>>
>> Perhaps an Erlang based RPC server would do well for a network based
>> application like this. It would allow for the mobile apps to send and
>> receive data from the server all over one connection without multiple
>> setup/tear down that individual HTTP requests would do. Since no web browser
>> is involved, we do not need to deal with the nuances of HTTP long-polling at
>> the mobile client level. I also haven't seen great long polling/keep-alive
>> support on the client-side in iOS, but that's irrelevant for the community
>> here.
>>
>> A lot of these "COMET" and long-polling/streaming servers are built with
>> HTTP in mind. I'm thinking just using a plain-text protocol over TCP is
>> better catered for the type of app I'm building, will make the client more
>> responsive, allow for receiving of updates from the server without
>> constantly polling the server, etc.
>>
>> I also looked into HTTP pipelining, but it doesn't look to be worth the
>> trouble when it comes to implementing it on the clients. Also, I'm not sure
>> if it would allow for bi-directional communication in the client<->server
>> communication channel.
>>
>> Am I completely out of line in thinking that building a custom solution in
>> Erlang is a good idea here? To my understanding, Erlang excels at servers
>> like this, and if I run the server on tcp/80, I should be able to avoid most
>> firewall/port issues. The client would need work to deal with timeouts, re
>> connections, acknowledging receipt of asynchronous requests, but that's not
>> Erlang's problem.
>>
>> Has anyone built something similar before? Should I just stick to a web
>> server and deal with "COMET" type technologies? (WebSockets, long-polling,
>> client-side polling).
>>
>> Was hoping someone could solidify that I'm not entirely insane for wanting a
>> better solution than HTTP would serve in this case, at least at the client
>> level. I'll still be using HTTP/REST extensively, the Erlang server would
>> just handle the persistent connections and messaging to the Web Service
>> (which would probably be something like Django or Rails).
>>
>> Sorry for the long post; I am just excited to get into the heads of people
>> who are smarter than I.
>>
>> Happy hacking!
>>
>> Matthew
>>
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
>
>
> --
> Ale.
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Gustav Simonsson-2
In reply to this post by Matthew Hillsborough

If you feel bold and adventurous, let the client and server communicate with Erlang messages!

http://www.burbas.se/2011/04/12/download-erlang-for-ios/
http://www.burbas.se/2010/11/07/erlang-for-android-now-available-for-download/

Cheers,
Gustav Simonsson

----- Original Message -----
From: "Matthew Hillsborough" <[hidden email]>
To: [hidden email]
Sent: Tuesday, July 12, 2011 10:25:48 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
Subject: [erlang-questions] Is Erlang a good tool for this particular project?


Greetings Erlang community,


Let me further elaborate on my question that's in the subject of this message. I tried to reach out with this question on StackOverflow, however I did not have much luck there. Perhaps the community here can provide some feedback here for me to let me know if I'm on the right track or if Erlang is not the right tool for what I'm trying to accomplish.




I'm building native mobile applications in both iOS and Android. These apps require "realtime" updates from and to the server, same as many (but not all) network-based application does (Facebook, social games like Words with Friends, Finance applications, etc). The communication here is bi-directional, in the sense that the server might have updates for the mobile clients and the clients will be pushing data down to the server whenever necessary.

I think using HTTP long polling for this is over kill in the sense that long polling can be detrimental to battery life, especially with a lot of TCP setup/teardown for every HTTP connection the device needs to send out through the wire. It might make sense to have the mobile applications use persistent TCP sockets to establish a connection to the server, and send RPC style commands to the server for all web service communication. This ofcourse, would require a server to handle the long-lived TCP connection and be able to speak to a web service once it makes sense of the data passed down the TCP pipe. I'm thinking of passing data in plain text using JSON or XML and then using some kind of Erlang interface to HTTP to call a web service to handle all the REST type communication. The responses would then go back to the "RPC" Erlang instance, which would send the updates to the appropriate client(s).

Perhaps an Erlang based RPC server would do well for a network based application like this. It would allow for the mobile apps to send and receive data from the server all over one connection without multiple setup/tear down that individual HTTP requests would do. Since no web browser is involved, we do not need to deal with the nuances of HTTP long-polling at the mobile client level. I also haven't seen great long polling/keep-alive support on the client-side in iOS, but that's irrelevant for the community here.

A lot of these "COMET" and long-polling/streaming servers are built with HTTP in mind. I'm thinking just using a plain-text protocol over TCP is better catered for the type of app I'm building, will make the client more responsive, allow for receiving of updates from the server without constantly polling the server, etc.

I also looked into HTTP pipelining, but it doesn't look to be worth the trouble when it comes to implementing it on the clients. Also, I'm not sure if it would allow for bi-directional communication in the client<->server communication channel.

Am I completely out of line in thinking that building a custom solution in Erlang is a good idea here? To my understanding, Erlang excels at servers like this, and if I run the server on tcp/80, I should be able to avoid most firewall/port issues. The client would need work to deal with timeouts, re connections, acknowledging receipt of asynchronous requests, but that's not Erlang's problem.

Has anyone built something similar before? Should I just stick to a web server and deal with "COMET" type technologies? (WebSockets, long-polling, client-side polling).

Was hoping someone could solidify that I'm not entirely insane for wanting a better solution than HTTP would serve in this case, at least at the client level. I'll still be using HTTP/REST extensively, the Erlang server would just handle the persistent connections and messaging to the Web Service (which would probably be something like Django or Rails).

Sorry for the long post; I am just excited to get into the heads of people who are smarter than I.

Happy hacking!

Matthew



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Matthew Hillsborough
In reply to this post by Ale
Hi Ale and everyone else who replied,

That's exactly the point, I don't think HTTP is necessary at all! There is the overhead of sending extraneous HTTP headers over the wire. All of those additional bits take additional CPU time and bandwidth (on a mobile device with limited CPU and even more limited bandwidth!). I see absolutely no need for sending HTTP headers and parsing them from the response for this, particularly because I am not building a web browser based application. I have access to C/C++/Objective-C (on iOS) and Java (on Android) and these are perfectly capable of working with TCP sockets. A friend of mine suggested that I just pass messages to the server as JSON using a prefixed header that specifies the length of the message. That would be it! Simple and compact. 

I just wanted to validate the idea and see if others thing I'll run into too many edge cases uses TCP sockets via an RPC type server to build this server/client architecture out. I want to really make it reusable so all of my mobile products can use it. 

Loving all the advice coming in! Thanks all.

Matthew. 

On Tue, Jul 12, 2011 at 5:03 PM, Ale <[hidden email]> wrote:
Hi,

Just a thought here... Erlang is great for scalability and handling
multiple connections, and well I'm sure a lot of people can give you a
better description fo this, but it occurred to me if you considered
other protocols instead of HTTP? The problem seems to be there rather
than the language you write your sever. It occurred to me that you
might benefit from using Jabber/XMPP. Googling I found some iOS jabber
clients http://code.google.com/p/xmppframework/ and the cannonical
jabber server is written in Erlang.

Regards,

2011/7/12 Matthew Hillsborough <[hidden email]>:
> Greetings Erlang community,
> Let me further elaborate on my question that's in the subject of this
> message. I tried to reach out with this question on StackOverflow, however I
> did not have much luck there. Perhaps the community here can provide some
> feedback here for me to let me know if I'm on the right track or if Erlang
> is not the right tool for what I'm trying to accomplish.
>
> I'm building native mobile applications in both iOS and Android. These apps
> require "realtime" updates from and to the server, same as many (but not
> all) network-based application does (Facebook, social games like Words with
> Friends, Finance applications, etc). The communication here is
> bi-directional, in the sense that the server might have updates for the
> mobile clients and the clients will be pushing data down to the server
> whenever necessary.
>
> I think using HTTP long polling for this is over kill in the sense that long
> polling can be detrimental to battery life, especially with a lot of TCP
> setup/teardown for every HTTP connection the device needs to send out
> through the wire. It might make sense to have the mobile applications use
> persistent TCP sockets to establish a connection to the server, and send RPC
> style commands to the server for all web service communication. This
> ofcourse, would require a server to handle the long-lived TCP connection and
> be able to speak to a web service once it makes sense of the data passed
> down the TCP pipe. I'm thinking of passing data in plain text using JSON or
> XML and then using some kind of Erlang interface to HTTP to call a web
> service to handle all the REST type communication. The responses would then
> go back to the "RPC" Erlang instance, which would send the updates to the
> appropriate client(s).
>
> Perhaps an Erlang based RPC server would do well for a network based
> application like this. It would allow for the mobile apps to send and
> receive data from the server all over one connection without multiple
> setup/tear down that individual HTTP requests would do. Since no web browser
> is involved, we do not need to deal with the nuances of HTTP long-polling at
> the mobile client level. I also haven't seen great long polling/keep-alive
> support on the client-side in iOS, but that's irrelevant for the community
> here.
>
> A lot of these "COMET" and long-polling/streaming servers are built with
> HTTP in mind. I'm thinking just using a plain-text protocol over TCP is
> better catered for the type of app I'm building, will make the client more
> responsive, allow for receiving of updates from the server without
> constantly polling the server, etc.
>
> I also looked into HTTP pipelining, but it doesn't look to be worth the
> trouble when it comes to implementing it on the clients. Also, I'm not sure
> if it would allow for bi-directional communication in the client<->server
> communication channel.
>
> Am I completely out of line in thinking that building a custom solution in
> Erlang is a good idea here? To my understanding, Erlang excels at servers
> like this, and if I run the server on tcp/80, I should be able to avoid most
> firewall/port issues. The client would need work to deal with timeouts, re
> connections, acknowledging receipt of asynchronous requests, but that's not
> Erlang's problem.
>
> Has anyone built something similar before? Should I just stick to a web
> server and deal with "COMET" type technologies? (WebSockets, long-polling,
> client-side polling).
>
> Was hoping someone could solidify that I'm not entirely insane for wanting a
> better solution than HTTP would serve in this case, at least at the client
> level. I'll still be using HTTP/REST extensively, the Erlang server would
> just handle the persistent connections and messaging to the Web Service
> (which would probably be something like Django or Rails).
>
> Sorry for the long post; I am just excited to get into the heads of people
> who are smarter than I.
>
> Happy hacking!
>
> Matthew
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>



--
Ale.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Joseph Wayne Norton-3

Matt -

There is already a client and server implementation similar to your idea that uses JSON as the wire format over TCP/IP.  See the section "JSF" for a brief description:


Download and build instructions are here:


Hope you find it helpful.

- Joe N.


On Jul 13, 2011, at 9:42 PM, Matthew Hillsborough wrote:

Hi Ale and everyone else who replied,

That's exactly the point, I don't think HTTP is necessary at all! There is the overhead of sending extraneous HTTP headers over the wire. All of those additional bits take additional CPU time and bandwidth (on a mobile device with limited CPU and even more limited bandwidth!). I see absolutely no need for sending HTTP headers and parsing them from the response for this, particularly because I am not building a web browser based application. I have access to C/C++/Objective-C (on iOS) and Java (on Android) and these are perfectly capable of working with TCP sockets. A friend of mine suggested that I just pass messages to the server as JSON using a prefixed header that specifies the length of the message. That would be it! Simple and compact. 

I just wanted to validate the idea and see if others thing I'll run into too many edge cases uses TCP sockets via an RPC type server to build this server/client architecture out. I want to really make it reusable so all of my mobile products can use it. 

Loving all the advice coming in! Thanks all.

Matthew. 

On Tue, Jul 12, 2011 at 5:03 PM, Ale <[hidden email]> wrote:
Hi,

Just a thought here... Erlang is great for scalability and handling
multiple connections, and well I'm sure a lot of people can give you a
better description fo this, but it occurred to me if you considered
other protocols instead of HTTP? The problem seems to be there rather
than the language you write your sever. It occurred to me that you
might benefit from using Jabber/XMPP. Googling I found some iOS jabber
clients http://code.google.com/p/xmppframework/ and the cannonical
jabber server is written in Erlang.

Regards,

2011/7/12 Matthew Hillsborough <[hidden email]>:
> Greetings Erlang community,
> Let me further elaborate on my question that's in the subject of this
> message. I tried to reach out with this question on StackOverflow, however I
> did not have much luck there. Perhaps the community here can provide some
> feedback here for me to let me know if I'm on the right track or if Erlang
> is not the right tool for what I'm trying to accomplish.
>
> I'm building native mobile applications in both iOS and Android. These apps
> require "realtime" updates from and to the server, same as many (but not
> all) network-based application does (Facebook, social games like Words with
> Friends, Finance applications, etc). The communication here is
> bi-directional, in the sense that the server might have updates for the
> mobile clients and the clients will be pushing data down to the server
> whenever necessary.
>
> I think using HTTP long polling for this is over kill in the sense that long
> polling can be detrimental to battery life, especially with a lot of TCP
> setup/teardown for every HTTP connection the device needs to send out
> through the wire. It might make sense to have the mobile applications use
> persistent TCP sockets to establish a connection to the server, and send RPC
> style commands to the server for all web service communication. This
> ofcourse, would require a server to handle the long-lived TCP connection and
> be able to speak to a web service once it makes sense of the data passed
> down the TCP pipe. I'm thinking of passing data in plain text using JSON or
> XML and then using some kind of Erlang interface to HTTP to call a web
> service to handle all the REST type communication. The responses would then
> go back to the "RPC" Erlang instance, which would send the updates to the
> appropriate client(s).
>
> Perhaps an Erlang based RPC server would do well for a network based
> application like this. It would allow for the mobile apps to send and
> receive data from the server all over one connection without multiple
> setup/tear down that individual HTTP requests would do. Since no web browser
> is involved, we do not need to deal with the nuances of HTTP long-polling at
> the mobile client level. I also haven't seen great long polling/keep-alive
> support on the client-side in iOS, but that's irrelevant for the community
> here.
>
> A lot of these "COMET" and long-polling/streaming servers are built with
> HTTP in mind. I'm thinking just using a plain-text protocol over TCP is
> better catered for the type of app I'm building, will make the client more
> responsive, allow for receiving of updates from the server without
> constantly polling the server, etc.
>
> I also looked into HTTP pipelining, but it doesn't look to be worth the
> trouble when it comes to implementing it on the clients. Also, I'm not sure
> if it would allow for bi-directional communication in the client<->server
> communication channel.
>
> Am I completely out of line in thinking that building a custom solution in
> Erlang is a good idea here? To my understanding, Erlang excels at servers
> like this, and if I run the server on tcp/80, I should be able to avoid most
> firewall/port issues. The client would need work to deal with timeouts, re
> connections, acknowledging receipt of asynchronous requests, but that's not
> Erlang's problem.
>
> Has anyone built something similar before? Should I just stick to a web
> server and deal with "COMET" type technologies? (WebSockets, long-polling,
> client-side polling).
>
> Was hoping someone could solidify that I'm not entirely insane for wanting a
> better solution than HTTP would serve in this case, at least at the client
> level. I'll still be using HTTP/REST extensively, the Erlang server would
> just handle the persistent connections and messaging to the Web Service
> (which would probably be something like Django or Rails).
>
> Sorry for the long post; I am just excited to get into the heads of people
> who are smarter than I.
>
> Happy hacking!
>
> Matthew
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>



--
Ale.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

Joseph Norton




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Matthew Hillsborough
In reply to this post by Matthew Hillsborough
Hi Joe,

Thanks for posting links to this. Surprised Joe A. did not mention it in his post, it has him listed as an author. :)

Just from a quick glimpse, this appears to be using MochiWeb, which is completely HTTP based. I am not entirely sure if that's the route I am looking to go, unless I am overlooking something here. With that said, there are probably parts of this code I can pick out and re-use in a pure TCP socket implementation (if this is indeed purely HTTP, which I suspect it might be).

Would love to hear I am wrong though so I do not need to reinvent the wheel.

Thank you and happy hacking -

Matthew

On Wed, Jul 13, 2011 at 9:43 AM, Joseph Norton <[hidden email]> wrote:

Matt -

There is already a client and server implementation similar to your idea that uses JSON as the wire format over TCP/IP.  See the section "JSF" for a brief description:


Download and building instructions are here:


Hope you find it helpful.

- Joe N.


On Jul 13, 2011, at 9:42 PM, Matthew Hillsborough wrote:

Hi Ale and everyone else who replied,

That's exactly the point, I don't think HTTP is necessary at all! There is the overhead of sending extraneous HTTP headers over the wire. All of those additional bits take additional CPU time and bandwidth (on a mobile device with limited CPU and even more limited bandwidth!). I see absolutely no need for sending HTTP headers and parsing them from the response for this, particularly because I am not building a web browser based application. I have access to C/C++/Objective-C (on iOS) and Java (on Android) and these are perfectly capable of working with TCP sockets. A friend of mine suggested that I just pass messages to the server as JSON using a prefixed header that specifies the length of the message. That would be it! Simple and compact. 

I just wanted to validate the idea and see if others thing I'll run into too many edge cases uses TCP sockets via an RPC type server to build this server/client architecture out. I want to really make it reusable so all of my mobile products can use it. 

Loving all the advice coming in! Thanks all.

Matthew. 

On Tue, Jul 12, 2011 at 5:03 PM, Ale <[hidden email]> wrote:
Hi,

Just a thought here... Erlang is great for scalability and handling
multiple connections, and well I'm sure a lot of people can give you a
better description fo this, but it occurred to me if you considered
other protocols instead of HTTP? The problem seems to be there rather
than the language you write your sever. It occurred to me that you
might benefit from using Jabber/XMPP. Googling I found some iOS jabber
clients http://code.google.com/p/xmppframework/ and the cannonical
jabber server is written in Erlang.

Regards,

2011/7/12 Matthew Hillsborough <[hidden email]>:
> Greetings Erlang community,
> Let me further elaborate on my question that's in the subject of this
> message. I tried to reach out with this question on StackOverflow, however I
> did not have much luck there. Perhaps the community here can provide some
> feedback here for me to let me know if I'm on the right track or if Erlang
> is not the right tool for what I'm trying to accomplish.
>
> I'm building native mobile applications in both iOS and Android. These apps
> require "realtime" updates from and to the server, same as many (but not
> all) network-based application does (Facebook, social games like Words with
> Friends, Finance applications, etc). The communication here is
> bi-directional, in the sense that the server might have updates for the
> mobile clients and the clients will be pushing data down to the server
> whenever necessary.
>
> I think using HTTP long polling for this is over kill in the sense that long
> polling can be detrimental to battery life, especially with a lot of TCP
> setup/teardown for every HTTP connection the device needs to send out
> through the wire. It might make sense to have the mobile applications use
> persistent TCP sockets to establish a connection to the server, and send RPC
> style commands to the server for all web service communication. This
> ofcourse, would require a server to handle the long-lived TCP connection and
> be able to speak to a web service once it makes sense of the data passed
> down the TCP pipe. I'm thinking of passing data in plain text using JSON or
> XML and then using some kind of Erlang interface to HTTP to call a web
> service to handle all the REST type communication. The responses would then
> go back to the "RPC" Erlang instance, which would send the updates to the
> appropriate client(s).
>
> Perhaps an Erlang based RPC server would do well for a network based
> application like this. It would allow for the mobile apps to send and
> receive data from the server all over one connection without multiple
> setup/tear down that individual HTTP requests would do. Since no web browser
> is involved, we do not need to deal with the nuances of HTTP long-polling at
> the mobile client level. I also haven't seen great long polling/keep-alive
> support on the client-side in iOS, but that's irrelevant for the community
> here.
>
> A lot of these "COMET" and long-polling/streaming servers are built with
> HTTP in mind. I'm thinking just using a plain-text protocol over TCP is
> better catered for the type of app I'm building, will make the client more
> responsive, allow for receiving of updates from the server without
> constantly polling the server, etc.
>
> I also looked into HTTP pipelining, but it doesn't look to be worth the
> trouble when it comes to implementing it on the clients. Also, I'm not sure
> if it would allow for bi-directional communication in the client<->server
> communication channel.
>
> Am I completely out of line in thinking that building a custom solution in
> Erlang is a good idea here? To my understanding, Erlang excels at servers
> like this, and if I run the server on tcp/80, I should be able to avoid most
> firewall/port issues. The client would need work to deal with timeouts, re
> connections, acknowledging receipt of asynchronous requests, but that's not
> Erlang's problem.
>
> Has anyone built something similar before? Should I just stick to a web
> server and deal with "COMET" type technologies? (WebSockets, long-polling,
> client-side polling).
>
> Was hoping someone could solidify that I'm not entirely insane for wanting a
> better solution than HTTP would serve in this case, at least at the client
> level. I'll still be using HTTP/REST extensively, the Erlang server would
> just handle the persistent connections and messaging to the Web Service
> (which would probably be something like Django or Rails).
>
> Sorry for the long post; I am just excited to get into the heads of people
> who are smarter than I.
>
> Happy hacking!
>
> Matthew
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>



--
Ale.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

Joseph Norton





_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Joseph Wayne Norton-3

Matt -

This particular application is only using Mochiweb's json encoder/decoder.

If you have time and interest, please review the UBF user's guide from the beginning.  Hopefully, it will be clearer.

The TCP/IP wire formats - UBF, EBF, and JSF all work using the same basic client and server implementation.  The only difference being the network wire format.

If you have other questions, let me know.

- Joe


On Jul 13, 2011, at 11:16 PM, Matthew Hillsborough wrote:

Hi Joe,

Thanks for posting links to this. Surprised Joe A. did not mention it in his post, it has him listed as an author. :)

Just from a quick glimpse, this appears to be using MochiWeb, which is completely HTTP based. I am not entirely sure if that's the route I am looking to go, unless I am overlooking something here. With that said, there are probably parts of this code I can pick out and re-use in a pure TCP socket implementation (if this is indeed purely HTTP, which I suspect it might be).

Would love to hear I am wrong though so I do not need to reinvent the wheel.

Thank you and happy hacking -

Matthew

On Wed, Jul 13, 2011 at 9:43 AM, Joseph Norton <[hidden email]> wrote:

Matt -

There is already a client and server implementation similar to your idea that uses JSON as the wire format over TCP/IP.  See the section "JSF" for a brief description:


Download and building instructions are here:


Hope you find it helpful.

- Joe N.


On Jul 13, 2011, at 9:42 PM, Matthew Hillsborough wrote:

Hi Ale and everyone else who replied,

That's exactly the point, I don't think HTTP is necessary at all! There is the overhead of sending extraneous HTTP headers over the wire. All of those additional bits take additional CPU time and bandwidth (on a mobile device with limited CPU and even more limited bandwidth!). I see absolutely no need for sending HTTP headers and parsing them from the response for this, particularly because I am not building a web browser based application. I have access to C/C++/Objective-C (on iOS) and Java (on Android) and these are perfectly capable of working with TCP sockets. A friend of mine suggested that I just pass messages to the server as JSON using a prefixed header that specifies the length of the message. That would be it! Simple and compact. 

I just wanted to validate the idea and see if others thing I'll run into too many edge cases uses TCP sockets via an RPC type server to build this server/client architecture out. I want to really make it reusable so all of my mobile products can use it. 

Loving all the advice coming in! Thanks all.

Matthew. 

On Tue, Jul 12, 2011 at 5:03 PM, Ale <[hidden email]> wrote:
Hi,

Just a thought here... Erlang is great for scalability and handling
multiple connections, and well I'm sure a lot of people can give you a
better description fo this, but it occurred to me if you considered
other protocols instead of HTTP? The problem seems to be there rather
than the language you write your sever. It occurred to me that you
might benefit from using Jabber/XMPP. Googling I found some iOS jabber
clients http://code.google.com/p/xmppframework/ and the cannonical
jabber server is written in Erlang.

Regards,

2011/7/12 Matthew Hillsborough <[hidden email]>:
> Greetings Erlang community,
> Let me further elaborate on my question that's in the subject of this
> message. I tried to reach out with this question on StackOverflow, however I
> did not have much luck there. Perhaps the community here can provide some
> feedback here for me to let me know if I'm on the right track or if Erlang
> is not the right tool for what I'm trying to accomplish.
>
> I'm building native mobile applications in both iOS and Android. These apps
> require "realtime" updates from and to the server, same as many (but not
> all) network-based application does (Facebook, social games like Words with
> Friends, Finance applications, etc). The communication here is
> bi-directional, in the sense that the server might have updates for the
> mobile clients and the clients will be pushing data down to the server
> whenever necessary.
>
> I think using HTTP long polling for this is over kill in the sense that long
> polling can be detrimental to battery life, especially with a lot of TCP
> setup/teardown for every HTTP connection the device needs to send out
> through the wire. It might make sense to have the mobile applications use
> persistent TCP sockets to establish a connection to the server, and send RPC
> style commands to the server for all web service communication. This
> ofcourse, would require a server to handle the long-lived TCP connection and
> be able to speak to a web service once it makes sense of the data passed
> down the TCP pipe. I'm thinking of passing data in plain text using JSON or
> XML and then using some kind of Erlang interface to HTTP to call a web
> service to handle all the REST type communication. The responses would then
> go back to the "RPC" Erlang instance, which would send the updates to the
> appropriate client(s).
>
> Perhaps an Erlang based RPC server would do well for a network based
> application like this. It would allow for the mobile apps to send and
> receive data from the server all over one connection without multiple
> setup/tear down that individual HTTP requests would do. Since no web browser
> is involved, we do not need to deal with the nuances of HTTP long-polling at
> the mobile client level. I also haven't seen great long polling/keep-alive
> support on the client-side in iOS, but that's irrelevant for the community
> here.
>
> A lot of these "COMET" and long-polling/streaming servers are built with
> HTTP in mind. I'm thinking just using a plain-text protocol over TCP is
> better catered for the type of app I'm building, will make the client more
> responsive, allow for receiving of updates from the server without
> constantly polling the server, etc.
>
> I also looked into HTTP pipelining, but it doesn't look to be worth the
> trouble when it comes to implementing it on the clients. Also, I'm not sure
> if it would allow for bi-directional communication in the client<->server
> communication channel.
>
> Am I completely out of line in thinking that building a custom solution in
> Erlang is a good idea here? To my understanding, Erlang excels at servers
> like this, and if I run the server on tcp/80, I should be able to avoid most
> firewall/port issues. The client would need work to deal with timeouts, re
> connections, acknowledging receipt of asynchronous requests, but that's not
> Erlang's problem.
>
> Has anyone built something similar before? Should I just stick to a web
> server and deal with "COMET" type technologies? (WebSockets, long-polling,
> client-side polling).
>
> Was hoping someone could solidify that I'm not entirely insane for wanting a
> better solution than HTTP would serve in this case, at least at the client
> level. I'll still be using HTTP/REST extensively, the Erlang server would
> just handle the persistent connections and messaging to the Web Service
> (which would probably be something like Django or Rails).
>
> Sorry for the long post; I am just excited to get into the heads of people
> who are smarter than I.
>
> Happy hacking!
>
> Matthew
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>



--
Ale.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

Joseph Norton




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

Joseph Norton




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Joel Meyer
In reply to this post by Matthew Hillsborough


On Wed, Jul 13, 2011 at 7:16 AM, Matthew Hillsborough <[hidden email]> wrote:
Hi Joe,

Thanks for posting links to this. Surprised Joe A. did not mention it in his post, it has him listed as an author. :)

Just from a quick glimpse, this appears to be using MochiWeb, which is completely HTTP based. I am not entirely sure if that's the route I am looking to go, unless I am overlooking something here. With that said, there are probably parts of this code I can pick out and re-use in a pure TCP socket implementation (if this is indeed purely HTTP, which I suspect it might be).

Would love to hear I am wrong though so I do not need to reinvent the wheel.

Thank you and happy hacking -

Out of curiosity, is there a reason you wouldn't use thrift or protocol buffers? Even if you don't use their RPC mechanisms, you could use one of those as the serialized form for message passing (over plain sockets, if you so desire). I think a binary serialization format will still be more compact than JSON and faster to deserialize as well.

Cheers,
Joel
 

Matthew

On Wed, Jul 13, 2011 at 9:43 AM, Joseph Norton <[hidden email]> wrote:

Matt -

There is already a client and server implementation similar to your idea that uses JSON as the wire format over TCP/IP.  See the section "JSF" for a brief description:


Download and building instructions are here:


Hope you find it helpful.

- Joe N.


On Jul 13, 2011, at 9:42 PM, Matthew Hillsborough wrote:

Hi Ale and everyone else who replied,

That's exactly the point, I don't think HTTP is necessary at all! There is the overhead of sending extraneous HTTP headers over the wire. All of those additional bits take additional CPU time and bandwidth (on a mobile device with limited CPU and even more limited bandwidth!). I see absolutely no need for sending HTTP headers and parsing them from the response for this, particularly because I am not building a web browser based application. I have access to C/C++/Objective-C (on iOS) and Java (on Android) and these are perfectly capable of working with TCP sockets. A friend of mine suggested that I just pass messages to the server as JSON using a prefixed header that specifies the length of the message. That would be it! Simple and compact. 

I just wanted to validate the idea and see if others thing I'll run into too many edge cases uses TCP sockets via an RPC type server to build this server/client architecture out. I want to really make it reusable so all of my mobile products can use it. 

Loving all the advice coming in! Thanks all.

Matthew. 

On Tue, Jul 12, 2011 at 5:03 PM, Ale <[hidden email]> wrote:
Hi,

Just a thought here... Erlang is great for scalability and handling
multiple connections, and well I'm sure a lot of people can give you a
better description fo this, but it occurred to me if you considered
other protocols instead of HTTP? The problem seems to be there rather
than the language you write your sever. It occurred to me that you
might benefit from using Jabber/XMPP. Googling I found some iOS jabber
clients http://code.google.com/p/xmppframework/ and the cannonical
jabber server is written in Erlang.

Regards,

2011/7/12 Matthew Hillsborough <[hidden email]>:
> Greetings Erlang community,
> Let me further elaborate on my question that's in the subject of this
> message. I tried to reach out with this question on StackOverflow, however I
> did not have much luck there. Perhaps the community here can provide some
> feedback here for me to let me know if I'm on the right track or if Erlang
> is not the right tool for what I'm trying to accomplish.
>
> I'm building native mobile applications in both iOS and Android. These apps
> require "realtime" updates from and to the server, same as many (but not
> all) network-based application does (Facebook, social games like Words with
> Friends, Finance applications, etc). The communication here is
> bi-directional, in the sense that the server might have updates for the
> mobile clients and the clients will be pushing data down to the server
> whenever necessary.
>
> I think using HTTP long polling for this is over kill in the sense that long
> polling can be detrimental to battery life, especially with a lot of TCP
> setup/teardown for every HTTP connection the device needs to send out
> through the wire. It might make sense to have the mobile applications use
> persistent TCP sockets to establish a connection to the server, and send RPC
> style commands to the server for all web service communication. This
> ofcourse, would require a server to handle the long-lived TCP connection and
> be able to speak to a web service once it makes sense of the data passed
> down the TCP pipe. I'm thinking of passing data in plain text using JSON or
> XML and then using some kind of Erlang interface to HTTP to call a web
> service to handle all the REST type communication. The responses would then
> go back to the "RPC" Erlang instance, which would send the updates to the
> appropriate client(s).
>
> Perhaps an Erlang based RPC server would do well for a network based
> application like this. It would allow for the mobile apps to send and
> receive data from the server all over one connection without multiple
> setup/tear down that individual HTTP requests would do. Since no web browser
> is involved, we do not need to deal with the nuances of HTTP long-polling at
> the mobile client level. I also haven't seen great long polling/keep-alive
> support on the client-side in iOS, but that's irrelevant for the community
> here.
>
> A lot of these "COMET" and long-polling/streaming servers are built with
> HTTP in mind. I'm thinking just using a plain-text protocol over TCP is
> better catered for the type of app I'm building, will make the client more
> responsive, allow for receiving of updates from the server without
> constantly polling the server, etc.
>
> I also looked into HTTP pipelining, but it doesn't look to be worth the
> trouble when it comes to implementing it on the clients. Also, I'm not sure
> if it would allow for bi-directional communication in the client<->server
> communication channel.
>
> Am I completely out of line in thinking that building a custom solution in
> Erlang is a good idea here? To my understanding, Erlang excels at servers
> like this, and if I run the server on tcp/80, I should be able to avoid most
> firewall/port issues. The client would need work to deal with timeouts, re
> connections, acknowledging receipt of asynchronous requests, but that's not
> Erlang's problem.
>
> Has anyone built something similar before? Should I just stick to a web
> server and deal with "COMET" type technologies? (WebSockets, long-polling,
> client-side polling).
>
> Was hoping someone could solidify that I'm not entirely insane for wanting a
> better solution than HTTP would serve in this case, at least at the client
> level. I'll still be using HTTP/REST extensively, the Erlang server would
> just handle the persistent connections and messaging to the Web Service
> (which would probably be something like Django or Rails).
>
> Sorry for the long post; I am just excited to get into the heads of people
> who are smarter than I.
>
> Happy hacking!
>
> Matthew
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>



--
Ale.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

Joseph Norton





_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Steve Davis
In reply to this post by Matthew Hillsborough
I am tempted to answer the OP with just a simple "Yes".

Rationale: Erlang/OTP is a good tool for any project, it has excellent
expressiveness, it encourages clarity of thought about a problem, and
it is fast to implement appropriate solutions.

However, of course, it may not be pandemically the best or most
appropriate tool... and I suspect that's the comfort that was
requested.

/s
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Jon Watte
In reply to this post by Matthew Hillsborough
Why do you say that HTTP long-poll is bad for battery, but persistent TCP connections are not?
A proper long-poll with a long timeout should be no different from a persistent TCP connection with the same time-out.

Sincerely,

jw


--
Americans might object: there is no way we would sacrifice our living standards for the benefit of people in the rest of the world. Nevertheless, whether we get there willingly or not, we shall soon have lower consumption rates, because our present rates are unsustainable.



On Tue, Jul 12, 2011 at 1:25 PM, Matthew Hillsborough <[hidden email]> wrote:
Greetings Erlang community,

Let me further elaborate on my question that's in the subject of this message. I tried to reach out with this question on StackOverflow, however I did not have much luck there. Perhaps the community here can provide some feedback here for me to let me know if I'm on the right track or if Erlang is not the right tool for what I'm trying to accomplish.

I'm building native mobile applications in both iOS and Android. These apps require "realtime" updates from and to the server, same as many (but not all) network-based application does (Facebook, social games like Words with Friends, Finance applications, etc). The communication here is bi-directional, in the sense that the server might have updates for the mobile clients and the clients will be pushing data down to the server whenever necessary.

I think using HTTP long polling for this is over kill in the sense that long polling can be detrimental to battery life, especially with a lot of TCP setup/teardown for every HTTP connection the device needs to send out through the wire. It might make sense to have the mobile applications use persistent TCP sockets to establish a connection to the server, and send RPC style commands to the server for all web service communication. This ofcourse, would require a server to handle the long-lived TCP connection and be able to speak to a web service once it makes sense of the data passed down the TCP pipe. I'm thinking of passing data in plain text using JSON or XML and then using some kind of Erlang interface to HTTP to call a web service to handle all the REST type communication. The responses would then go back to the "RPC" Erlang instance, which would send the updates to the appropriate client(s).

Perhaps an Erlang based RPC server would do well for a network based application like this. It would allow for the mobile apps to send and receive data from the server all over one connection without multiple setup/tear down that individual HTTP requests would do. Since no web browser is involved, we do not need to deal with the nuances of HTTP long-polling at the mobile client level. I also haven't seen great long polling/keep-alive support on the client-side in iOS, but that's irrelevant for the community here.

A lot of these "COMET" and long-polling/streaming servers are built with HTTP in mind. I'm thinking just using a plain-text protocol over TCP is better catered for the type of app I'm building, will make the client more responsive, allow for receiving of updates from the server without constantly polling the server, etc.

I also looked into HTTP pipelining, but it doesn't look to be worth the trouble when it comes to implementing it on the clients. Also, I'm not sure if it would allow for bi-directional communication in the client<->server communication channel. 

Am I completely out of line in thinking that building a custom solution in Erlang is a good idea here? To my understanding, Erlang excels at servers like this, and if I run the server on tcp/80, I should be able to avoid most firewall/port issues. The client would need work to deal with timeouts, re connections, acknowledging receipt of asynchronous requests, but that's not Erlang's problem. 

Has anyone built something similar before? Should I just stick to a web server and deal with "COMET" type technologies? (WebSockets, long-polling, client-side polling).

Was hoping someone could solidify that I'm not entirely insane for wanting a better solution than HTTP would serve in this case, at least at the client level. I'll still be using HTTP/REST extensively, the Erlang server would just handle the persistent connections and messaging to the Web Service (which would probably be something like Django or Rails).

Sorry for the long post; I am just excited to get into the heads of people who are smarter than I.

Happy hacking!

Matthew



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Chad Depue
In reply to this post by Matthew Hillsborough
Matthew,

Inaka has built native applications exactly like this for iPad and iPhone and started w/HTTP-comet style polling and we started with comet-style long polling but ended up at simple REST for the api commands and a TCP socket connection for the persistant stuff... The socket is far more efficient and quite simply with a mobile app there's no need to use long-polling.

We also use Rails for admin but Erlang for all API commands, even HTTP. Ping me off-list and I can share some of what we learned.

Chad DePue
inakanetworks.com - development consulting | skype cdepue | @chaddepue
+1 206.866.5707 



On Tue, Jul 12, 2011 at 5:25 PM, Matthew Hillsborough <[hidden email]> wrote:
Greetings Erlang community,

Let me further elaborate on my question that's in the subject of this message. I tried to reach out with this question on StackOverflow, however I did not have much luck there. Perhaps the community here can provide some feedback here for me to let me know if I'm on the right track or if Erlang is not the right tool for what I'm trying to accomplish.

I'm building native mobile applications in both iOS and Android. These apps require "realtime" updates from and to the server, same as many (but not all) network-based application does (Facebook, social games like Words with Friends, Finance applications, etc). The communication here is bi-directional, in the sense that the server might have updates for the mobile clients and the clients will be pushing data down to the server whenever necessary.

I think using HTTP long polling for this is over kill in the sense that long polling can be detrimental to battery life, especially with a lot of TCP setup/teardown for every HTTP connection the device needs to send out through the wire. It might make sense to have the mobile applications use persistent TCP sockets to establish a connection to the server, and send RPC style commands to the server for all web service communication. This ofcourse, would require a server to handle the long-lived TCP connection and be able to speak to a web service once it makes sense of the data passed down the TCP pipe. I'm thinking of passing data in plain text using JSON or XML and then using some kind of Erlang interface to HTTP to call a web service to handle all the REST type communication. The responses would then go back to the "RPC" Erlang instance, which would send the updates to the appropriate client(s).

Perhaps an Erlang based RPC server would do well for a network based application like this. It would allow for the mobile apps to send and receive data from the server all over one connection without multiple setup/tear down that individual HTTP requests would do. Since no web browser is involved, we do not need to deal with the nuances of HTTP long-polling at the mobile client level. I also haven't seen great long polling/keep-alive support on the client-side in iOS, but that's irrelevant for the community here.

A lot of these "COMET" and long-polling/streaming servers are built with HTTP in mind. I'm thinking just using a plain-text protocol over TCP is better catered for the type of app I'm building, will make the client more responsive, allow for receiving of updates from the server without constantly polling the server, etc.

I also looked into HTTP pipelining, but it doesn't look to be worth the trouble when it comes to implementing it on the clients. Also, I'm not sure if it would allow for bi-directional communication in the client<->server communication channel. 

Am I completely out of line in thinking that building a custom solution in Erlang is a good idea here? To my understanding, Erlang excels at servers like this, and if I run the server on tcp/80, I should be able to avoid most firewall/port issues. The client would need work to deal with timeouts, re connections, acknowledging receipt of asynchronous requests, but that's not Erlang's problem. 

Has anyone built something similar before? Should I just stick to a web server and deal with "COMET" type technologies? (WebSockets, long-polling, client-side polling).

Was hoping someone could solidify that I'm not entirely insane for wanting a better solution than HTTP would serve in this case, at least at the client level. I'll still be using HTTP/REST extensively, the Erlang server would just handle the persistent connections and messaging to the Web Service (which would probably be something like Django or Rails).

Sorry for the long post; I am just excited to get into the heads of people who are smarter than I.

Happy hacking!

Matthew



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Is Erlang a good tool for this particular project?

Mike Oxford
In reply to this post by Joel Meyer
On Wed, Jul 13, 2011 at 11:03 AM, Joel Meyer <[hidden email]> wrote:

> On Wed, Jul 13, 2011 at 7:16 AM, Matthew Hillsborough
> <[hidden email]> wrote:
>> Thanks for posting links to this. Surprised Joe A. did not mention it in
>> his post, it has him listed as an author. :)
>> Just from a quick glimpse, this appears to be using MochiWeb, which is
>> completely HTTP based. I am not entirely sure if that's the route I am
>
> Out of curiosity, is there a reason you wouldn't use thrift or protocol
> buffers? Even if you don't use their RPC mechanisms, you could use one of
> those as the serialized form for message passing (over plain sockets, if you
> so desire). I think a binary serialization format will still be more compact
> than JSON and faster to deserialize as well.

Both of your points are true, but thrift and protobufs both have hard-IDLs.
There are games you can play with non-IDL-based protocols (without
giving up security) that you simply cannot do with binary protocols unless
you force a client upgrade with the new IDL.

-mox
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...