gen_tcp and large HTTP headers

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

gen_tcp and large HTTP headers

Roberto Ostinelli
dear list,

i'm using gen_tcp in {packet, http} and {active, once} modes to receive HTTP requests.

whenever i use a very large GET header, like such:
http://localhost:8080/foo?var=foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar

the controlling process receives from the socket the message:
{SocketMode, Sock,
    {http_error, "oobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar HTTP/1.1\r\n"}}

this looks to me like a MTU issue: the GET header is larger than the MTU, and thus the erlang:decode_packet/3 returns the error of a badly formed HTTP header since it is not complete.

since i'm in active mode, my loop looks like this:

headers(Sock, H) ->
    gen_tcp:setopts(Sock, [{active, once}]),
    receive
        {SocketMode, Sock, {http_header, _, Header, _, Val} = _Head} ->
            headers(Sock, [{Header, Val}|H]);
       
        ...

    end.

thus, should i use an accumulator? but then should i manually trigger the erlang:decode_packet/3 function?

thank you for any insights on this.

r.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Roberto Ostinelli
additional information: this can be solved simply by setting:
{recbuf, 2000}

any reasons for keeping this value to minimum sizes?


2011/6/22 Roberto Ostinelli <[hidden email]>
dear list,

i'm using gen_tcp in {packet, http} and {active, once} modes to receive HTTP requests.

whenever i use a very large GET header, like such:
http://localhost:8080/foo?var=foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar

the controlling process receives from the socket the message:
{SocketMode, Sock,
    {http_error, "oobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar HTTP/1.1\r\n"}}

this looks to me like a MTU issue: the GET header is larger than the MTU, and thus the erlang:decode_packet/3 returns the error of a badly formed HTTP header since it is not complete.

since i'm in active mode, my loop looks like this:

headers(Sock, H) ->
    gen_tcp:setopts(Sock, [{active, once}]),
    receive
        {SocketMode, Sock, {http_header, _, Header, _, Val} = _Head} ->
            headers(Sock, [{Header, Val}|H]);
       
        ...

    end.

thus, should i use an accumulator? but then should i manually trigger the erlang:decode_packet/3 function?

thank you for any insights on this.

r.



--
-------------------------------------------------------------------
Roberto Ostinelli
CTO, WideTag Inc. - Realtime, Social, Green
widetag.com
skype: rostinelli
twitter: ostinelli
mobile: +39 335 6 100 22 6

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Steve Vinoski-2
IMO this is really a bug in the http packet parser. Setting recbuf
might work for this example but if you get a larger URI then it will
obviously fail, as you can't set it large enough.

For this case the packet parser must realize the http request line is
incomplete and complete the parsing only once the whole request line
arrives. Getting this completely right might well require allowing a
way for the application to control how long a URI it wants to allow,
since if the parser just keeps reading in packets to find the end of
the request line someone could deny service by just sending request
packets that contain a really really long URI.

--steve

On Wed, Jun 22, 2011 at 3:53 PM, Roberto Ostinelli <[hidden email]> wrote:

> additional information: this can be solved simply by setting:
> {recbuf, 2000}
>
> any reasons for keeping this value to minimum sizes?
>
>
> 2011/6/22 Roberto Ostinelli <[hidden email]>
>>
>> dear list,
>>
>> i'm using gen_tcp in {packet, http} and {active, once} modes to receive
>> HTTP requests.
>>
>> whenever i use a very large GET header, like such:
>>
>> http://localhost:8080/foo?var=foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar
>>
>> the controlling process receives from the socket the message:
>> {SocketMode, Sock,
>>     {http_error, "oobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar
>> HTTP/1.1\r\n"}}
>>
>> this looks to me like a MTU issue: the GET header is larger than the MTU,
>> and thus the erlang:decode_packet/3 returns the error of a badly formed HTTP
>> header since it is not complete.
>>
>> since i'm in active mode, my loop looks like this:
>>
>> headers(Sock, H) ->
>>     gen_tcp:setopts(Sock, [{active, once}]),
>>     receive
>>         {SocketMode, Sock, {http_header, _, Header, _, Val} = _Head} ->
>>             headers(Sock, [{Header, Val}|H]);
>>
>>         ...
>>
>>     end.
>>
>> thus, should i use an accumulator? but then should i manually trigger the
>> erlang:decode_packet/3 function?
>>
>> thank you for any insights on this.
>>
>> r.
>
>
>
> --
> -------------------------------------------------------------------
> Roberto Ostinelli
> CTO, WideTag Inc. - Realtime, Social, Green
> widetag.com
> skype: rostinelli
> twitter: ostinelli
> mobile: +39 335 6 100 22 6
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Roberto Ostinelli
hi steve,

it does look so indeed. i actually do receive the http_request message:

{http, Sock, {http_request, Method, Path, Version}}

even though the GET request has obviously been shortened, so that the remaining part of the GET request gets passed over and then generates the http_error message while parsing headers.

i guess i actually should not receive the http_request message, since the GET header is incomplete.


2011/6/22 Steve Vinoski <[hidden email]>
IMO this is really a bug in the http packet parser. Setting recbuf
might work for this example but if you get a larger URI then it will
obviously fail, as you can't set it large enough.

For this case the packet parser must realize the http request line is
incomplete and complete the parsing only once the whole request line
arrives. Getting this completely right might well require allowing a
way for the application to control how long a URI it wants to allow,
since if the parser just keeps reading in packets to find the end of
the request line someone could deny service by just sending request
packets that contain a really really long URI.

--steve

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Steve Vinoski-2
Looking at the packet parser, I'm thinking it always treats this case
as HTTP version 0.9, so that's wrong as well. Can you verify that
major and minor are set to 0 and 9 respectively in the version tuple
within the http_request tuple you get?

--steve

On Wed, Jun 22, 2011 at 4:25 PM, Roberto Ostinelli <[hidden email]> wrote:

> hi steve,
>
> it does look so indeed. i actually do receive the http_request message:
>
> {http, Sock, {http_request, Method, Path, Version}}
>
> even though the GET request has obviously been shortened, so that the
> remaining part of the GET request gets passed over and then generates the
> http_error message while parsing headers.
>
> i guess i actually should not receive the http_request message, since the
> GET header is incomplete.
>
>
> 2011/6/22 Steve Vinoski <[hidden email]>
>>
>> IMO this is really a bug in the http packet parser. Setting recbuf
>> might work for this example but if you get a larger URI then it will
>> obviously fail, as you can't set it large enough.
>>
>> For this case the packet parser must realize the http request line is
>> incomplete and complete the parsing only once the whole request line
>> arrives. Getting this completely right might well require allowing a
>> way for the application to control how long a URI it wants to allow,
>> since if the parser just keeps reading in packets to find the end of
>> the request line someone could deny service by just sending request
>> packets that contain a really really long URI.
>>
>> --steve
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Loïc Hoguin
In reply to this post by Roberto Ostinelli
More info here:
  https://github.com/extend/cowboy/issues/3

We just switched to calling decode_packet directly. It's a bug in
gen_tcp when in HTTP mode, not decode_packet, so it works fine now. I
haven't had the time to find the exact source of this bug but it's on my
todo list.

Cheers.

On 06/22/2011 10:25 PM, Roberto Ostinelli wrote:

> hi steve,
>
> it does look so indeed. i actually do receive the http_request message:
>
> {http, Sock, {http_request, Method, Path, Version}}
>
> even though the GET request has obviously been shortened, so that the
> remaining part of the GET request gets passed over and then generates
> the http_error message while parsing headers.
>
> i guess i actually should not receive the http_request message, since
> the GET header is incomplete.
>
>
> 2011/6/22 Steve Vinoski <[hidden email] <mailto:[hidden email]>>
>
>     IMO this is really a bug in the http packet parser. Setting recbuf
>     might work for this example but if you get a larger URI then it will
>     obviously fail, as you can't set it large enough.
>
>     For this case the packet parser must realize the http request line is
>     incomplete and complete the parsing only once the whole request line
>     arrives. Getting this completely right might well require allowing a
>     way for the application to control how long a URI it wants to allow,
>     since if the parser just keeps reading in packets to find the end of
>     the request line someone could deny service by just sending request
>     packets that contain a really really long URI.
>
>     --steve
>
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


--
Loïc Hoguin
Dev:Extend
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Steve Vinoski-2
Yep, I see the same. Looks like the inet driver isn't checking packet
length properly like decode_packet does, and as a result it passes
incomplete data to the packet parser, which isn't coded to handle
that.

--steve

On Wed, Jun 22, 2011 at 5:17 PM, Loïc Hoguin <[hidden email]> wrote:

> More info here:
>  https://github.com/extend/cowboy/issues/3
>
> We just switched to calling decode_packet directly. It's a bug in
> gen_tcp when in HTTP mode, not decode_packet, so it works fine now. I
> haven't had the time to find the exact source of this bug but it's on my
> todo list.
>
> Cheers.
>
> On 06/22/2011 10:25 PM, Roberto Ostinelli wrote:
>> hi steve,
>>
>> it does look so indeed. i actually do receive the http_request message:
>>
>> {http, Sock, {http_request, Method, Path, Version}}
>>
>> even though the GET request has obviously been shortened, so that the
>> remaining part of the GET request gets passed over and then generates
>> the http_error message while parsing headers.
>>
>> i guess i actually should not receive the http_request message, since
>> the GET header is incomplete.
>>
>>
>> 2011/6/22 Steve Vinoski <[hidden email] <mailto:[hidden email]>>
>>
>>     IMO this is really a bug in the http packet parser. Setting recbuf
>>     might work for this example but if you get a larger URI then it will
>>     obviously fail, as you can't set it large enough.
>>
>>     For this case the packet parser must realize the http request line is
>>     incomplete and complete the parsing only once the whole request line
>>     arrives. Getting this completely right might well require allowing a
>>     way for the application to control how long a URI it wants to allow,
>>     since if the parser just keeps reading in packets to find the end of
>>     the request line someone could deny service by just sending request
>>     packets that contain a really really long URI.
>>
>>     --steve
>>
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
> --
> Loïc Hoguin
> Dev:Extend
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Roberto Ostinelli
In reply to this post by Steve Vinoski-2
steve,

your intuition is correct: version is {0, 9}.

i guess the only solution is to avoid {packet, http} and handle these manually? seems such a waste.

2011/6/22 Steve Vinoski <[hidden email]>
Looking at the packet parser, I'm thinking it always treats this case
as HTTP version 0.9, so that's wrong as well. Can you verify that
major and minor are set to 0 and 9 respectively in the version tuple
within the http_request tuple you get?

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Steve Vinoski-2
Unfortunately, it appears that if you want to be completely safe and
handle long request lines like your example, then yes, calling
decode_packet is the only way to go. :-( I've stared at the inet
driver code some more and the way it's calling the packet parser just
doesn't allow it to be informed that a buffer is incomplete and more
data from the next packet is needed.

--steve

On Wed, Jun 22, 2011 at 6:34 PM, Roberto Ostinelli <[hidden email]> wrote:

> steve,
>
> your intuition is correct: version is {0, 9}.
>
> i guess the only solution is to avoid {packet, http} and handle these
> manually? seems such a waste.
>
> 2011/6/22 Steve Vinoski <[hidden email]>
>>
>> Looking at the packet parser, I'm thinking it always treats this case
>> as HTTP version 0.9, so that's wrong as well. Can you verify that
>> major and minor are set to 0 and 9 respectively in the version tuple
>> within the http_request tuple you get?
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

ERLANG-3
Hi Steve, Roberto,

Maybe the Erlang team should consider and fix that problem
on next releases!

/Zab

Le 23 juin 11 à 05:35, Steve Vinoski a écrit :

> Unfortunately, it appears that if you want to be completely safe and
> handle long request lines like your example, then yes, calling
> decode_packet is the only way to go. :-( I've stared at the inet
> driver code some more and the way it's calling the packet parser just
> doesn't allow it to be informed that a buffer is incomplete and more
> data from the next packet is needed.
>
> --steve
>
> On Wed, Jun 22, 2011 at 6:34 PM, Roberto Ostinelli <[hidden email]
> > wrote:
>> steve,
>>
>> your intuition is correct: version is {0, 9}.
>>
>> i guess the only solution is to avoid {packet, http} and handle these
>> manually? seems such a waste.
>>
>> 2011/6/22 Steve Vinoski <[hidden email]>
>>>
>>> Looking at the packet parser, I'm thinking it always treats this  
>>> case
>>> as HTTP version 0.9, so that's wrong as well. Can you verify that
>>> major and minor are set to 0 and 9 respectively in the version tuple
>>> within the http_request tuple you get?
>>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Paolo Negri
Hi Everyone,

Thanks for discussing this, we encountered the problem on our system
and filed the bug that originated Roberto's email [1].
We're currently working around the problem by setting recbuf on the
socket, this temporary solution is not too bad in real life if the
need is only to cope with requests originated by browsers as currently
fairly popular ones can't handle over 2048 characters [2].
While this is suitable as a temporary solution it really feels like
the current implementation of inet is incorrect as it states "The
packets are returned with the format according to HttpPacket described
in erlang:decode_packet/3" [3] and instead the results are different
from those one would get by calling erlang:decode_packet/3

[1] https://github.com/ostinelli/misultin/issues/40
[2] http://support.microsoft.com/kb/208427
[3] http://www.erlang.org/doc/man/inet.html#setopts-2

Thanks,

Paolo

On Thu, Jun 23, 2011 at 7:59 AM, ERLANG <[hidden email]> wrote:
> Hi Steve, Roberto,
>
> Maybe the Erlang team should consider and fix that problem
> on next releases!
>
> /Zab
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Roberto Ostinelli
2011/6/23 Paolo Negri <[hidden email]>
Hi Everyone,

Thanks for discussing this, we encountered the problem on our system
and filed the bug that originated Roberto's email [1].

indeed, i didn't refer you because i didn't associate the nickname 'hungryblank' to you :)

 
We're currently working around the problem by setting recbuf on the
socket, this temporary solution is not too bad in real life if the
need is only to cope with requests originated by browsers as currently
fairly popular ones can't handle over 2048 characters [2].

i agree that recbuf can provide a solution on GET urls, however as seen on cowboy's issues list the same does apply on other headers, such as long Cookies for instance.

ps: i've provided a patch exactly to do this, is it the one you're using or have you patched misultin yourself?
 
While this is suitable as a temporary solution it really feels like
the current implementation of inet is incorrect as it states "The
packets are returned with the format according to HttpPacket described
in erlang:decode_packet/3" [3] and instead the results are different
from those one would get by calling erlang:decode_packet/3

[1] https://github.com/ostinelli/misultin/issues/40
[2] http://support.microsoft.com/kb/208427
[3] http://www.erlang.org/doc/man/inet.html#setopts-2

Thanks,

Paolo


let's see if the OTP team pops in in this discussion. it probably is a known issue.

r.
 

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Sverker Eriksson
In reply to this post by Paolo Negri
The bug is causing http lines to get truncated when the inet_drv
in-buffer gets full.

It would be nicer to dynamically expand the in-buffer and let
inet:setopts option {packet_size, MaxSize} be used to protect against
DoS attacks as it can for other packet types.**

Steve Vinoski wrote:

> Looking at the packet parser, I'm thinking it always treats this case
> as HTTP version 0.9, so that's wrong as well.
>

What's wrong? I didn't understand that.

/Sverker, Erlang/OTP


Paolo Negri wrote:

> Hi Everyone,
>
> Thanks for discussing this, we encountered the problem on our system
> and filed the bug that originated Roberto's email [1].
> We're currently working around the problem by setting recbuf on the
> socket, this temporary solution is not too bad in real life if the
> need is only to cope with requests originated by browsers as currently
> fairly popular ones can't handle over 2048 characters [2].
> While this is suitable as a temporary solution it really feels like
> the current implementation of inet is incorrect as it states "The
> packets are returned with the format according to HttpPacket described
> in erlang:decode_packet/3" [3] and instead the results are different
> from those one would get by calling erlang:decode_packet/3
>
> [1] https://github.com/ostinelli/misultin/issues/40
> [2] http://support.microsoft.com/kb/208427
> [3] http://www.erlang.org/doc/man/inet.html#setopts-2
>
> Thanks,
>
> Paolo
>
> On Thu, Jun 23, 2011 at 7:59 AM, ERLANG <[hidden email]> wrote:
>  
>> Hi Steve, Roberto,
>>
>> Maybe the Erlang team should consider and fix that problem
>> on next releases!
>>
>> /Zab
>>    
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>  

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Robert Virding-2
In reply to this post by Roberto Ostinelli
Wouldn't it be better to rewrite the packet parser to work in a re-entrant fashion so as to be able to span over many packets? It would save a lot of fiddling with the recbuf size.

Robert

----- "Sverker Eriksson" <[hidden email]> wrote:

> The bug is causing http lines to get truncated when the inet_drv
> in-buffer gets full.
>
> It would be nicer to dynamically expand the in-buffer and let
> inet:setopts option {packet_size, MaxSize} be used to protect against
>
> DoS attacks as it can for other packet types.**
>
> Steve Vinoski wrote:
>
> > Looking at the packet parser, I'm thinking it always treats this
> case
> > as HTTP version 0.9, so that's wrong as well.
> >
>
> What's wrong? I didn't understand that.
>
> /Sverker, Erlang/OTP
>
>
> Paolo Negri wrote:
> > Hi Everyone,
> >
> > Thanks for discussing this, we encountered the problem on our
> system
> > and filed the bug that originated Roberto's email [1].
> > We're currently working around the problem by setting recbuf on the
> > socket, this temporary solution is not too bad in real life if the
> > need is only to cope with requests originated by browsers as
> currently
> > fairly popular ones can't handle over 2048 characters [2].
> > While this is suitable as a temporary solution it really feels like
> > the current implementation of inet is incorrect as it states "The
> > packets are returned with the format according to HttpPacket
> described
> > in erlang:decode_packet/3" [3] and instead the results are
> different
> > from those one would get by calling erlang:decode_packet/3
> >
> > [1] https://github.com/ostinelli/misultin/issues/40
> > [2] http://support.microsoft.com/kb/208427
> > [3] http://www.erlang.org/doc/man/inet.html#setopts-2
> >
> > Thanks,
> >
> > Paolo
> >
> > On Thu, Jun 23, 2011 at 7:59 AM, ERLANG <[hidden email]> wrote:
> >  
> >> Hi Steve, Roberto,
> >>
> >> Maybe the Erlang team should consider and fix that problem
> >> on next releases!
> >>
> >> /Zab
> >>    
> > _______________________________________________
> > erlang-questions mailing list
> > [hidden email]
> > http://erlang.org/mailman/listinfo/erlang-questions
> >
> >  
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Max Lapshin-2
On Fri, Jun 24, 2011 at 12:10 AM, Robert Virding
<[hidden email]> wrote:
> Wouldn't it be better to rewrite the packet parser to work in a re-entrant fashion so as to be able to span over many packets? It would save a lot of fiddling with the recbuf size.
>

You always have to keep buffer: either in receiver, either in parser.

Easier to use when parser is stateless, but it requires more work when
you parse again and again old data on receiving new.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Robert Virding-2
Yes, of course you cannot get around some buffering. It is (should be) easier for the parser to keep track of its buffer as it can parse as it goes and only keep its already parsed data and perhaps that bit in the input buffer which was not enough for a parse, or partial parse.

Robert


----- "Max Lapshin" <[hidden email]> wrote:

> On Fri, Jun 24, 2011 at 12:10 AM, Robert Virding
> <[hidden email]> wrote:
> > Wouldn't it be better to rewrite the packet parser to work in a
> re-entrant fashion so as to be able to span over many packets? It
> would save a lot of fiddling with the recbuf size.
> >
>
> You always have to keep buffer: either in receiver, either in parser.
>
> Easier to use when parser is stateless, but it requires more work
> when
> you parse again and again old data on receiving new.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Steve Vinoski-2
The underlying packet parser is capable of partial parsing, and that's
how decode_packet uses it. It's just that the inet driver doesn't seem
to call the packet parser the right way to take advantage of partial
parsing.

--steve

On Fri, Jun 24, 2011 at 9:27 AM, Robert Virding
<[hidden email]> wrote:

> Yes, of course you cannot get around some buffering. It is (should be) easier for the parser to keep track of its buffer as it can parse as it goes and only keep its already parsed data and perhaps that bit in the input buffer which was not enough for a parse, or partial parse.
>
> Robert
>
>
> ----- "Max Lapshin" <[hidden email]> wrote:
>
>> On Fri, Jun 24, 2011 at 12:10 AM, Robert Virding
>> <[hidden email]> wrote:
>> > Wouldn't it be better to rewrite the packet parser to work in a
>> re-entrant fashion so as to be able to span over many packets? It
>> would save a lot of fiddling with the recbuf size.
>> >
>>
>> You always have to keep buffer: either in receiver, either in parser.
>>
>> Easier to use when parser is stateless, but it requires more work
>> when
>> you parse again and again old data on receiving new.
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Max Lapshin-2
I have just thought, that proper web server should always use
packet_decode, rather than {mode, http}.

I'm currently trying to implement input of POST body as a stream and
understood very bad thing:
if client send second request very fast after sending first, it may
happens so, that post body will come to
us in the same packet with beginning of next header.

If Content-Length is 2900 bytes, it may be such situation:

1400 bytes of post body
1400 bytes of post body
100 bytes of post body + 1000 bytes of next request.

The only way to parse it is to keep buffer in web-server process and
use erlang:decode_packet, because the second way is to do blocking
gen_tcp:recv
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Steve Vinoski-2
On Fri, Jun 24, 2011 at 4:22 PM, Max Lapshin <[hidden email]> wrote:
> I have just thought, that proper web server should always use
> packet_decode, rather than {mode, http}.
>
> I'm currently trying to implement input of POST body as a stream and
> understood very bad thing:
> if client send second request very fast after sending first, it may
> happens so, that post body will come to
> us in the same packet with beginning of next header.

The client is broken in that case, since what you're describing is
HTTP pipelining and POST cannot be pipelined (nor can other
non-idempotent HTTP methods). The client has to wait for the POST
reply before sending the next request.

--steve
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_tcp and large HTTP headers

Max Lapshin-2
On Sat, Jun 25, 2011 at 12:35 AM, Steve Vinoski <[hidden email]> wrote:
>
> The client is broken in that case, since what you're describing is
> HTTP pipelining and POST cannot be pipelined (nor can other
> non-idempotent HTTP methods). The client has to wait for the POST
> reply before sending the next request.
>

So, I'm just frightened of a buggy behaviour, that shouldn't be
handled and there is no problem.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions