HTTP client with sessions support

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

HTTP client with sessions support

Mikl Kurkov
Hi!

I'm looking for a HTTP client with HTTP sessions support.
That means it should store and properly use session cookies and follow redirects.

Options:

1. httpc - standard app, has cookies store, redirects. Main problem is that it uses heavyweight profiles to isolate sessions,
So for each session I will need to create an atom (for name) and also a profile ETS table will be created. One possible workaround
is to use some kind of pool of predefined httpc profiles, flushing cookies on each session start. This will not work good for long running sessions though.
2. lhttpc - it seems to be closed
3. hackney - has redirects, but no cookie store, also definitely need some work on cookie processing code - looks like it uses outdated implementation
from some http _server_, so it has parser for 'cookie' header and builder for 'set-cookie'.
4. ibrowse - doesn't have session support (redirects, cookies)
5. gun - battle tested, low-level, no session support
6. xhttpc (https://github.com/seriyps/xhttpc) - not widely known, but looks promising,
uses layered middleware architecture.
Has already implemented middlewares for cookie store (parser based on last RFC6265), redirects, compression.
Supports different backends - httpc, hackney, lhttpc.

I plan to go with xhttpc, maybe implementing backend on top of gun.
Any suggestions, am I missing something?

--
Mikl

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Loïc Hoguin-3
On 06/11/2015 09:53 PM, Mikl Kurkov wrote:
> 5. gun - battle tested, low-level, no session support

Right we do want to have a cookie store later on. I am not sure how this
is supposed to look yet though. Suggestions welcome, I believe there is
a ticket for it.

Redirects, I am not too sure, it's fairly application-dependent (for
example I doubt you'd want to redirect automatically every time after a
successful POST to a REST API).

--
Loïc Hoguin
http://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Benoit Chesneau-2
Mikl what is missing in the cookie implementation of hackney? There is a new version coming next week so let's try to fix that... (any ticket is welcome).

Loic, only the 303 case could work with POST, other cases are ignored in hackney.

- benoît
On Fri 12 Jun 2015 at 00:39 Loïc Hoguin <[hidden email]> wrote:
On 06/11/2015 09:53 PM, Mikl Kurkov wrote:
> 5. gun - battle tested, low-level, no session support

Right we do want to have a cookie store later on. I am not sure how this
is supposed to look yet though. Suggestions welcome, I believe there is
a ticket for it.

Redirects, I am not too sure, it's fairly application-dependent (for
example I doubt you'd want to redirect automatically every time after a
successful POST to a REST API).

--
Loïc Hoguin
http://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Mikl Kurkov
In reply to this post by Loïc Hoguin-3
On Fri, Jun 12, 2015 at 1:39 AM, Loïc Hoguin <[hidden email]> wrote:
Right we do want to have a cookie store later on. I am not sure how this is supposed to look yet though. Suggestions welcome, I believe there is a ticket for it.

I'm afraid it will not be easy to find solution that will fit all user cases.
Someone need few huge client sessions, keeping cookies for many sites in ETS or even SQL database.
It seems httpc optimized for this use case.
In my case I need many tiny isolated sessions, so it is possible to have session store in state.
That's why I like xhttpc approach - you get some stable API and can use different implementations for
say cookie store that fits your needs.
 
Redirects, I am not too sure, it's fairly application-dependent (for example I doubt you'd want to redirect automatically every time after a successful POST to a REST API).

Regarding redirects, it of course has its caveats but it should just work for some main cases.

--
Mikl

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Mikl Kurkov
In reply to this post by Benoit Chesneau-2


On Fri, Jun 12, 2015 at 5:38 AM, Benoit Chesneau <[hidden email]> wrote:
Mikl what is missing in the cookie implementation of hackney? There is a new version coming next week so let's try to fix that... (any ticket is welcome).

Benoit,

First thing I stumbled upon on second minute of hackney usage is that it silently ignored cookies with Expires attribute,
I fixed this in my fork, but looking into code I understood that it was ported from some server codebase,
so it uses Cookie parser to parse Set-Cookie, and Set-Cookie builder to build Cookie. They have slightly different syntax
and constraints, so it need some refactoring or even full rewrite.
Also it seems several different representations for cookies are used inside lib,
like {Name, Value}, {Name, [{Name,Value}|Attrs]}, {Name, Value, Attrs}.
It would be great to have one unified structure for them.
Ticket added, see on github.

Another issue that bothers me is usage of process dictionary, I always try to avoid this.

--
Mikl

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Loïc Hoguin-3
In reply to this post by Mikl Kurkov
On 06/12/2015 02:19 PM, Mikl Kurkov wrote:

> On Fri, Jun 12, 2015 at 1:39 AM, Loïc Hoguin <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Right we do want to have a cookie store later on. I am not sure how
>     this is supposed to look yet though. Suggestions welcome, I believe
>     there is a ticket for it.
>
>
> I'm afraid it will not be easy to find solution that will fit all user
> cases.
> Someone need few huge client sessions, keeping cookies for many sites in
> ETS or even SQL database.
> It seems httpc optimized for this use case.
> In my case I need many tiny isolated sessions, so it is possible to have
> session store in state.
> That's why I like xhttpc approach - you get some stable API and can use
> different implementations for
> say cookie store that fits your needs.

The way I see it is that Gun needs to keep track of cookies for the
current connection *and* send cookies to the user process if any other
kind of tracking is required. Feeding cookies back into the Gun process
should be a function call or option away.

--
Loïc Hoguin
http://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Adrian Roe-3
One of the reasons I like (and extensively use) Gun is precisely because it does NOT automatically do stuff like track cookies for you or follow redirects.  

In the past we have used a seemingly endless train of Erlang HTTP clients and been tripped up when they try to be too helpful (retrying requests under the covers etc.)  There is plenty of room for helper libraries on top of things like gun (such as shotgun - internally we have one called spud_gun), but I’m always nervous when the low level library tries to “help”.  Already gun by default sends keep alive messages that break e.g. API calls to AWS (Amazon terminates the tcp connection when it notices it has inbound data that its API says should not be there).

I could just about see cookie handling as falling the right side of the “does too much for you” line, but have a strong preference that helper functionality  is clearly separated from the underlying core.

Big fan of Gun / Cowboy so absolutely no criticism intended - but you did say “suggestions welcome”.

Adrian

On Friday, Jun 12, 2015 at 1:44 pm, Loïc Hoguin <[hidden email]>, wrote:
On 06/12/2015 02:19 PM, Mikl Kurkov wrote:

> On Fri, Jun 12, 2015 at 1:39 AM, Loïc Hoguin <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Right we do want to have a cookie store later on. I am not sure how
> this is supposed to look yet though. Suggestions welcome, I believe
> there is a ticket for it.
>
>
> I'm afraid it will not be easy to find solution that will fit all user
> cases.
> Someone need few huge client sessions, keeping cookies for many sites in
> ETS or even SQL database.
> It seems httpc optimized for this use case.
> In my case I need many tiny isolated sessions, so it is possible to have
> session store in state.
> That's why I like xhttpc approach - you get some stable API and can use
> different implementations for
> say cookie store that fits your needs.

The way I see it is that Gun needs to keep track of cookies for the
current connection *and* send cookies to the user process if any other
kind of tracking is required. Feeding cookies back into the Gun process
should be a function call or option away.

--
Loïc Hoguin
http://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Benoit Chesneau-2
In reply to this post by Mikl Kurkov


On Fri, Jun 12, 2015 at 2:32 PM Mikl Kurkov <[hidden email]> wrote:
On Fri, Jun 12, 2015 at 5:38 AM, Benoit Chesneau <[hidden email]> wrote:
Mikl what is missing in the cookie implementation of hackney? There is a new version coming next week so let's try to fix that... (any ticket is welcome).

Benoit,

First thing I stumbled upon on second minute of hackney usage is that it silently ignored cookies with Expires attribute,
I fixed this in my fork, but looking into code I understood that it was ported from some server codebase,
so it uses Cookie parser to parse Set-Cookie, and Set-Cookie builder to build Cookie. They have slightly different syntax
and constraints, so it need some refactoring or even full rewrite.
Also it seems several different representations for cookies are used inside lib,
like {Name, Value}, {Name, [{Name,Value}|Attrs]}, {Name, Value, Attrs}.
It would be great to have one unified structure for them.
Ticket added, see on github.

Another issue that bothers me is usage of process dictionary, I always try to avoid this.

--
Mikl

Thanks! It's useful. I will implement changes for the next coming version aka 2.0.0 . 

I am not sure yet what should be the API. It depends if I still want to support older versions or not. 

- benoit 

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Benoit Chesneau-2
In reply to this post by Adrian Roe-3


On Fri, Jun 12, 2015 at 3:31 PM Adrian Roe <[hidden email]> wrote:
One of the reasons I like (and extensively use) Gun is precisely because it does NOT automatically do stuff like track cookies for you or follow redirects.  

Not sure if you know it, but Hackney as well doesn't handle redirection by default. You need to set the option to do it. Anyway the approach of hackney is to propose choice in the way you can handle easily any content over HTTP collecting and implementing some common patterns but still letting you the choice to ignore them.

- benoit

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Ingela Andin
In reply to this post by Mikl Kurkov
Hi!

2015-06-11 21:53 GMT+02:00 Mikl Kurkov <[hidden email]>:
Hi!

I'm looking for a HTTP client with HTTP sessions support.
That means it should store and properly use session cookies and follow redirects.

Options:

1. httpc - standard app, has cookies store, redirects. Main problem is that it uses heavyweight profiles to isolate sessions,
So for each session I will need to create an atom (for name) and also a profile ETS table will be created.

Creating a name for the profile is not that heavy?! The ets-table is an implementation detail not written in stone, it could use some other data structure like a dict or something. Feel free make a PR, if you make a good case it can be changed.

Regards Ingela Erlang/OTP team - Ericsson AB

 [...]

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Ulf Wiger-6

On 13 Jun 2015, at 21:35, Ingela Andin <[hidden email]> wrote:

Creating a name for the profile is not that heavy?! The ets-table is an implementation detail not written in stone, it could use some other data structure like a dict or something. Feel free make a PR, if you make a good case it can be changed.

Creating an ets table really isn’t a heavy operation.

Consider the following program:

-module(etsc).

-compile(export_all).


t(N) when N > 0 ->
    T = ets:new(t, []),
    ets:delete(T),
    t(N-1);
t(_) ->
    ok.


Eshell V5.10.4  (abort with ^G)
1> c(etsc).
{ok,etsc}
2> timer:tc(etsc,t,[1000]).
{2775,ok}
3> timer:tc(etsc,t,[1000]).
{2612,ok}
4> timer:tc(etsc,t,[1000]).
{2055,ok}
5> timer:tc(etsc,t,[1000]).
{1881,ok}

That is, on my old Macbook Air, it takes about a microsecond to create an ets table*.

Creating an atom is also obviously cheap, but I cringe a bit when list_to_atom/1 is used in generic code - although in this particular case, I guess creating memory leaks because of it is not very likely.

BR,
Ulf W

* Obviously, this test doesn’t accumulate ets tables. I removed the ets:delete/1 call, and things looked much worse:
- Creating the first 1000 tables took ca 13 ms.
- Creating 1000 tables when there were already 67K tables, took 6.2 seconds.
- However, creating an ets table from another process was still pretty fast - ca 10 usec
I conclude that having thousands of ets tables owned by a single process is a *bad* idea (probably not for this reason alone).

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Mikl Kurkov
On Sun, Jun 14, 2015 at 12:12 AM, Ulf Wiger <[hidden email]> wrote:
Creating an ets table really isn’t a heavy operation.

Agree, I'm mostly concerned about dynamic atom creation in runtime, isn't it recommended to avoid this on long running servers.

It would be great to be able to run httpc like any other linked server with start_link and then use its Pid to send requests without need of named profile. Or maybe let use some safe runtime data as profile name, like ref or string.
I see it is possible to run httpc in 'stand_alone' mode but this still requires atom profile name.

But well, for current task it seems I can try to use profile per user, not per session, so atoms list will not expand endlessly with time.

Thank you all for good points.

--
Mikl

 

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: HTTP client with sessions support

Ingela Andin
Hi!


2015-06-15 12:03 GMT+02:00 Mikl Kurkov <[hidden email]>:
On Sun, Jun 14, 2015 at 12:12 AM, Ulf Wiger <[hidden email]> wrote:
Creating an ets table really isn’t a heavy operation.

Agree, I'm mostly concerned about dynamic atom creation in runtime, isn't it recommended to avoid this on long running servers.


Well, profiles was designed as to be similar to inkognito windows in chrome, and to fit the API that was once contributed to us.
As such it was not expected that there would be lots and lots of profiles coming and going dynamically, and as the API was we needed
to have a registered name for the profile, hence the atom.

 
It would be great to be able to run httpc like any other linked server with start_link and then use its Pid to send requests without need of named profile. Or maybe let use some safe runtime data as profile name, like ref or string.

The profile concept could probably be extended to have "anonymous" profiles. 
 
I see it is possible to run httpc in 'stand_alone' mode but this still requires atom profile name.


The 'stand-alone' mode I think was a not so brilliant compromise for having a more light wight "HTTP GET". But if there is a demand you never know
what may happen.  I do not recommend the 'stand-alone' mode.

 
Regards Ingela Erlang/OTP team - Ericsson AB

[...] 


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions