Fire and Forget

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Fire and Forget

Steven Livingstone
Hi all, Erlang newbie here. So, hello all :-) Now,

I wish to do a fire and forget http post in Erlang.

I looked at the httpc module which seems to allow synchronous and
asynchronous calls but in the latter case i do not expect a return
value so don't need a request id .... and not sure whether that
behaviour influences what I want ... does it hold open a connection or
use up resources on a response that is never going to happen etc.

Does anyone know the best approach?

many thanks,
steven
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Fire and Forget

Paul Peregud-2
Low complexity solution: just spawn a process which will send request
and terminate.
More complex solution: spawn process under supervisor. Restart
strategy "temporary" is a good fit. Or "transient" if you want to
restart failed attempts.

As for "response never happens". Do you mean that server does not
return anything ever? Not even HTTP 200 OK? Just holds up connection?

On Fri, Apr 1, 2016 at 5:18 PM, Steven Livingstone <[hidden email]> wrote:

> Hi all, Erlang newbie here. So, hello all :-) Now,
>
> I wish to do a fire and forget http post in Erlang.
>
> I looked at the httpc module which seems to allow synchronous and
> asynchronous calls but in the latter case i do not expect a return
> value so don't need a request id .... and not sure whether that
> behaviour influences what I want ... does it hold open a connection or
> use up resources on a response that is never going to happen etc.
>
> Does anyone know the best approach?
>
> many thanks,
> steven
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions



--
Best regards,
Paul Peregud
+48602112091
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Fire and Forget

Steven Livingstone
Thanks Paul. It was more that as i would be just logging lots of data,
if some gets lost it is ok, hence it doesn't need to wait on a 200.

So, I guess it was how to optimise under those conditions, but perhaps
doing an async post and later getting a 200, even if it is just a "all
is ok" even if i don't intend to do anything beyond that may be a
sensible approach anyway.

Restart temporary under supervisor sounds good - i mainly don't want
to block anything. Unfortunately I come with the burden of years of
procedural programming so my mindset is still adjusting to the fact
spinning up many processes is a good thing in erlang.

It may be I should be passing the data using something other than an
HTTP POST and more erlang-like but one step at a time :-)

regards,
steven

On Fri, Apr 1, 2016 at 4:55 PM, Paul Peregud <[hidden email]> wrote:

> Low complexity solution: just spawn a process which will send request
> and terminate.
> More complex solution: spawn process under supervisor. Restart
> strategy "temporary" is a good fit. Or "transient" if you want to
> restart failed attempts.
>
> As for "response never happens". Do you mean that server does not
> return anything ever? Not even HTTP 200 OK? Just holds up connection?
>
> On Fri, Apr 1, 2016 at 5:18 PM, Steven Livingstone <[hidden email]> wrote:
>> Hi all, Erlang newbie here. So, hello all :-) Now,
>>
>> I wish to do a fire and forget http post in Erlang.
>>
>> I looked at the httpc module which seems to allow synchronous and
>> asynchronous calls but in the latter case i do not expect a return
>> value so don't need a request id .... and not sure whether that
>> behaviour influences what I want ... does it hold open a connection or
>> use up resources on a response that is never going to happen etc.
>>
>> Does anyone know the best approach?
>>
>> many thanks,
>> steven
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> --
> Best regards,
> Paul Peregud
> +48602112091
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Fire and Forget

Felix Gallo-2

I accidentally once destroyed a 50 machine php http api cluster by pointing a single four core erlang instance at it and making requests as fast as I could go. You may wish to build rate limiting and error handling in from the start instead of ignoring the response.

F.

On Apr 1, 2016 9:14 AM, "Steven Livingstone" <[hidden email]> wrote:
Thanks Paul. It was more that as i would be just logging lots of data,
if some gets lost it is ok, hence it doesn't need to wait on a 200.

So, I guess it was how to optimise under those conditions, but perhaps
doing an async post and later getting a 200, even if it is just a "all
is ok" even if i don't intend to do anything beyond that may be a
sensible approach anyway.

Restart temporary under supervisor sounds good - i mainly don't want
to block anything. Unfortunately I come with the burden of years of
procedural programming so my mindset is still adjusting to the fact
spinning up many processes is a good thing in erlang.

It may be I should be passing the data using something other than an
HTTP POST and more erlang-like but one step at a time :-)

regards,
steven

On Fri, Apr 1, 2016 at 4:55 PM, Paul Peregud <[hidden email]> wrote:
> Low complexity solution: just spawn a process which will send request
> and terminate.
> More complex solution: spawn process under supervisor. Restart
> strategy "temporary" is a good fit. Or "transient" if you want to
> restart failed attempts.
>
> As for "response never happens". Do you mean that server does not
> return anything ever? Not even HTTP 200 OK? Just holds up connection?
>
> On Fri, Apr 1, 2016 at 5:18 PM, Steven Livingstone <[hidden email]> wrote:
>> Hi all, Erlang newbie here. So, hello all :-) Now,
>>
>> I wish to do a fire and forget http post in Erlang.
>>
>> I looked at the httpc module which seems to allow synchronous and
>> asynchronous calls but in the latter case i do not expect a return
>> value so don't need a request id .... and not sure whether that
>> behaviour influences what I want ... does it hold open a connection or
>> use up resources on a response that is never going to happen etc.
>>
>> Does anyone know the best approach?
>>
>> many thanks,
>> steven
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> --
> Best regards,
> Paul Peregud
> <a href="tel:%2B48602112091" value="+48602112091">+48602112091
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Fire and Forget

Felix Gallo-2

Also, as far as resources go - http clients by themselves are unlikely to tax a box at any rate you are likely to do. At high rates, you may encounter file handle or socket starvation on (at least) essentially all Linux boxes owing to pathologically poor defaults out of the box.  You may need to tweak sysctl values on those systems if you need more than a few thousand http requests per second.

F.

On Apr 1, 2016 9:44 AM, "Felix Gallo" <[hidden email]> wrote:

I accidentally once destroyed a 50 machine php http api cluster by pointing a single four core erlang instance at it and making requests as fast as I could go. You may wish to build rate limiting and error handling in from the start instead of ignoring the response.

F.

On Apr 1, 2016 9:14 AM, "Steven Livingstone" <[hidden email]> wrote:
Thanks Paul. It was more that as i would be just logging lots of data,
if some gets lost it is ok, hence it doesn't need to wait on a 200.

So, I guess it was how to optimise under those conditions, but perhaps
doing an async post and later getting a 200, even if it is just a "all
is ok" even if i don't intend to do anything beyond that may be a
sensible approach anyway.

Restart temporary under supervisor sounds good - i mainly don't want
to block anything. Unfortunately I come with the burden of years of
procedural programming so my mindset is still adjusting to the fact
spinning up many processes is a good thing in erlang.

It may be I should be passing the data using something other than an
HTTP POST and more erlang-like but one step at a time :-)

regards,
steven

On Fri, Apr 1, 2016 at 4:55 PM, Paul Peregud <[hidden email]> wrote:
> Low complexity solution: just spawn a process which will send request
> and terminate.
> More complex solution: spawn process under supervisor. Restart
> strategy "temporary" is a good fit. Or "transient" if you want to
> restart failed attempts.
>
> As for "response never happens". Do you mean that server does not
> return anything ever? Not even HTTP 200 OK? Just holds up connection?
>
> On Fri, Apr 1, 2016 at 5:18 PM, Steven Livingstone <[hidden email]> wrote:
>> Hi all, Erlang newbie here. So, hello all :-) Now,
>>
>> I wish to do a fire and forget http post in Erlang.
>>
>> I looked at the httpc module which seems to allow synchronous and
>> asynchronous calls but in the latter case i do not expect a return
>> value so don't need a request id .... and not sure whether that
>> behaviour influences what I want ... does it hold open a connection or
>> use up resources on a response that is never going to happen etc.
>>
>> Does anyone know the best approach?
>>
>> many thanks,
>> steven
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> --
> Best regards,
> Paul Peregud
> <a href="tel:%2B48602112091" value="+48602112091" target="_blank">+48602112091
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Fire and Forget

Steven Livingstone
Interesting - thanks Felix. The 50 machine cluster destruction is
quite an achievement :-) I'll take heed.

I could be hooking some devices in and so there could be quite a lot
of data as they scale. I am considering putting something like
RabbitMQ in the middle if i need to scale but starting with the basics
just now.

I am new to Erlang but the entire approach sits well with me so i want
to persist with it. The defaults are something i'll research - some
endpoint will be on Docker instances and i know there could be
additional things to figure out there too.

Thanks again,
steven

On Fri, Apr 1, 2016 at 5:54 PM, Felix Gallo <[hidden email]> wrote:

> Also, as far as resources go - http clients by themselves are unlikely to
> tax a box at any rate you are likely to do. At high rates, you may encounter
> file handle or socket starvation on (at least) essentially all Linux boxes
> owing to pathologically poor defaults out of the box.  You may need to tweak
> sysctl values on those systems if you need more than a few thousand http
> requests per second.
>
> F.
>
> On Apr 1, 2016 9:44 AM, "Felix Gallo" <[hidden email]> wrote:
>>
>> I accidentally once destroyed a 50 machine php http api cluster by
>> pointing a single four core erlang instance at it and making requests as
>> fast as I could go. You may wish to build rate limiting and error handling
>> in from the start instead of ignoring the response.
>>
>> F.
>>
>> On Apr 1, 2016 9:14 AM, "Steven Livingstone" <[hidden email]> wrote:
>>>
>>> Thanks Paul. It was more that as i would be just logging lots of data,
>>> if some gets lost it is ok, hence it doesn't need to wait on a 200.
>>>
>>> So, I guess it was how to optimise under those conditions, but perhaps
>>> doing an async post and later getting a 200, even if it is just a "all
>>> is ok" even if i don't intend to do anything beyond that may be a
>>> sensible approach anyway.
>>>
>>> Restart temporary under supervisor sounds good - i mainly don't want
>>> to block anything. Unfortunately I come with the burden of years of
>>> procedural programming so my mindset is still adjusting to the fact
>>> spinning up many processes is a good thing in erlang.
>>>
>>> It may be I should be passing the data using something other than an
>>> HTTP POST and more erlang-like but one step at a time :-)
>>>
>>> regards,
>>> steven
>>>
>>> On Fri, Apr 1, 2016 at 4:55 PM, Paul Peregud <[hidden email]>
>>> wrote:
>>> > Low complexity solution: just spawn a process which will send request
>>> > and terminate.
>>> > More complex solution: spawn process under supervisor. Restart
>>> > strategy "temporary" is a good fit. Or "transient" if you want to
>>> > restart failed attempts.
>>> >
>>> > As for "response never happens". Do you mean that server does not
>>> > return anything ever? Not even HTTP 200 OK? Just holds up connection?
>>> >
>>> > On Fri, Apr 1, 2016 at 5:18 PM, Steven Livingstone <[hidden email]>
>>> > wrote:
>>> >> Hi all, Erlang newbie here. So, hello all :-) Now,
>>> >>
>>> >> I wish to do a fire and forget http post in Erlang.
>>> >>
>>> >> I looked at the httpc module which seems to allow synchronous and
>>> >> asynchronous calls but in the latter case i do not expect a return
>>> >> value so don't need a request id .... and not sure whether that
>>> >> behaviour influences what I want ... does it hold open a connection or
>>> >> use up resources on a response that is never going to happen etc.
>>> >>
>>> >> Does anyone know the best approach?
>>> >>
>>> >> many thanks,
>>> >> steven
>>> >> _______________________________________________
>>> >> erlang-questions mailing list
>>> >> [hidden email]
>>> >> http://erlang.org/mailman/listinfo/erlang-questions
>>> >
>>> >
>>> >
>>> > --
>>> > Best regards,
>>> > Paul Peregud
>>> > +48602112091
>>> _______________________________________________
>>> erlang-questions mailing list
>>> [hidden email]
>>> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions