What does "soft" real-time mean?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

What does "soft" real-time mean?

Avinash Dhumane-4
I have programmed high-frequency trading application in Erlang and comparing its reaction performance in the live market with other competing applications (mostly, in C).

It has been a consistent observation that my Erlang application lags substantially behind the competing applications. In fact, it never matched the reaction time of the so-called "slowest" C application in the market. 

The benchmark reaction time is about 5 microseconds. It is the time elapsed between tick to order. That is, the difference between the timestamps when I receive the tick (market event) from the mailbox of my process and when I submit the TCP send call on my order.

My process maintains the market depths of the securities and when my desired price (as a function of best buyers and/or sellers on multiple legs of the order) is observed, I compute the order terms and submit. 

The entire computation and communication (with the market) takes place within the single process and there are no collaborating processes (hence, no inter-process messaging; not even ETS). Still, the reaction time is in the order of 20 to 60 microseconds - that is, nowhere closer to the benchmark reaction time (5 microseconds).

What I wish to hear from the experienced people on this forum here is whether I am missing on something in my Erlang code, or should I just go ahead and program in C.

Thanks.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What does "soft" real-time mean?

Aleksander Nycz-2

Hello,


My private definition:

Hard real time -> 100% requests/jobs/processes must be completed before deadline

Soft real time -> N% requests/jobs/processes must be completed before deadline, where N < 100%,

    so we can/must accept that some jobs will not be finished on time.


And few question:

1. Erlang VM is running on Virtual Machine on bare metal?

2. What you erlang process really do? Can you show us some code?


Regards

Aleksander Nycz


W dniu 2017-10-20 o 14:00, Avinash Dhumane pisze:
I have programmed high-frequency trading application in Erlang and comparing its reaction performance in the live market with other competing applications (mostly, in C).

It has been a consistent observation that my Erlang application lags substantially behind the competing applications. In fact, it never matched the reaction time of the so-called "slowest" C application in the market. 

The benchmark reaction time is about 5 microseconds. It is the time elapsed between tick to order. That is, the difference between the timestamps when I receive the tick (market event) from the mailbox of my process and when I submit the TCP send call on my order.

My process maintains the market depths of the securities and when my desired price (as a function of best buyers and/or sellers on multiple legs of the order) is observed, I compute the order terms and submit. 

The entire computation and communication (with the market) takes place within the single process and there are no collaborating processes (hence, no inter-process messaging; not even ETS). Still, the reaction time is in the order of 20 to 60 microseconds - that is, nowhere closer to the benchmark reaction time (5 microseconds).

What I wish to hear from the experienced people on this forum here is whether I am missing on something in my Erlang code, or should I just go ahead and program in C.

Thanks.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

-- 
Aleksander Nycz
Chief Designer
Telco_021 BSS R&D
Comarch SA
Phone:  +48 17 785 5909
Mobile: +48 691 464 275
website: www.comarch.pl

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What does "soft" real-time mean?

Sergej Jurečko
In reply to this post by Avinash Dhumane-4
Erlang is the right tool for the job for a majority of server workloads. However when your requirements are a reaction time in microseconds, it most surely is not the right tool.

I would use Rust in that scenario if you’re picking languages.


Regards,
Sergej

> On 20 Oct 2017, at 14:00, Avinash Dhumane <[hidden email]> wrote:
>
> I have programmed high-frequency trading application in Erlang and comparing its reaction performance in the live market with other competing applications (mostly, in C).
>
> It has been a consistent observation that my Erlang application lags substantially behind the competing applications. In fact, it never matched the reaction time of the so-called "slowest" C application in the market.
>
> The benchmark reaction time is about 5 microseconds. It is the time elapsed between tick to order. That is, the difference between the timestamps when I receive the tick (market event) from the mailbox of my process and when I submit the TCP send call on my order.
>
> My process maintains the market depths of the securities and when my desired price (as a function of best buyers and/or sellers on multiple legs of the order) is observed, I compute the order terms and submit.
>
> The entire computation and communication (with the market) takes place within the single process and there are no collaborating processes (hence, no inter-process messaging; not even ETS). Still, the reaction time is in the order of 20 to 60 microseconds - that is, nowhere closer to the benchmark reaction time (5 microseconds).
>
> What I wish to hear from the experienced people on this forum here is whether I am missing on something in my Erlang code, or should I just go ahead and program in C.
>
> Thanks.
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What does "soft" real-time mean?

Hugo Mills-2
In reply to this post by Avinash Dhumane-4
On Fri, Oct 20, 2017 at 05:30:47PM +0530, Avinash Dhumane wrote:

> I have programmed high-frequency trading application in Erlang and
> comparing its reaction performance in the live market with other competing
> applications (mostly, in C).
>
> It has been a consistent observation that my Erlang application lags
> substantially behind the competing applications. In fact, it never matched
> the reaction time of the so-called "slowest" C application in the market.
>
> The benchmark reaction time is about 5 microseconds. It is the time elapsed
> between tick to order. That is, the difference between the timestamps when
> I receive the tick (market event) from the mailbox of my process and when I
> submit the TCP send call on my order.
   You're combining two measures here:

 - How fast does the process react once the message arrives in its
   mailbox? Not the time to getting a response out, but the delay
   between the message getting to the process, and the process being
   scheduled to actually start work on processing it.

 - Once the process starts work on it, how fast can it get the result
   out?

   Erlang's reputation is that's it's very good at the former measure,
even when there are many thousands of processes competing to be
scheduled.

   On the second measure, that's more about the efficiency of
executing the code once it starts. Depending on what you're doing in
the process, it will almost certainly be slower to do the actual work
than equivalent code in C, possibly _much_ slower.

   Note, however, that sometimes you can end up writing very poor
(inefficient) Erlang by following an efficient C implementation too
closely, and that a different implementation which takes account of
the functional nature of Erlang will run much faster.

   So the first thing I'd look at here would be benchmarking the
service code first (standalone, in a tight loop, without the message
passing part), to see how fast that actually runs. That should then at
least give you a good idea of which of the two measures above is the
slow one.

   Hugo.

> My process maintains the market depths of the securities and when my
> desired price (as a function of best buyers and/or sellers on multiple legs
> of the order) is observed, I compute the order terms and submit.
>
> The entire computation and communication (with the market) takes place
> within the single process and there are no collaborating processes (hence,
> no inter-process messaging; not even ETS). Still, the reaction time is in
> the order of 20 to 60 microseconds - that is, nowhere closer to the
> benchmark reaction time (5 microseconds).
>
> What I wish to hear from the experienced people on this forum here is
> whether I am missing on something in my Erlang code, or should I just go
> ahead and program in C.
>
> Thanks.

> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


--
Hugo Mills             | Great films about cricket: The Fantastic Four
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What does "soft" real-time mean?

Santiago Fernández
In reply to this post by Sergej Jurečko
In general Erlang is not the best option to do CPU intensive computations, especially with a single process.

I use Elixir/Erlang in financial apps to route incoming messages to websocket clients, process concurrent orders, etc. But, in my own benchmarks, it takes 5-10 microseconds only to parse a FIX incoming message (same process, no inter process comunication).


I think if you need Erlang for concurrency/scalability *and* you need to do some kind of HFT, so the way to go is by using a NIF or linked in driver (C, Rust?)

If you only need to do HFT, I think there is no reason to stick with Erlang, since there is no competition with the slowest C app.

HTH


--
Santiago

On Fri, Oct 20, 2017 at 10:46 AM, Sergej Jurečko <[hidden email]> wrote:
Erlang is the right tool for the job for a majority of server workloads. However when your requirements are a reaction time in microseconds, it most surely is not the right tool.

I would use Rust in that scenario if you’re picking languages.


Regards,
Sergej

> On 20 Oct 2017, at 14:00, Avinash Dhumane <[hidden email]> wrote:
>
> I have programmed high-frequency trading application in Erlang and comparing its reaction performance in the live market with other competing applications (mostly, in C).
>
> It has been a consistent observation that my Erlang application lags substantially behind the competing applications. In fact, it never matched the reaction time of the so-called "slowest" C application in the market.
>
> The benchmark reaction time is about 5 microseconds. It is the time elapsed between tick to order. That is, the difference between the timestamps when I receive the tick (market event) from the mailbox of my process and when I submit the TCP send call on my order.
>
> My process maintains the market depths of the securities and when my desired price (as a function of best buyers and/or sellers on multiple legs of the order) is observed, I compute the order terms and submit.
>
> The entire computation and communication (with the market) takes place within the single process and there are no collaborating processes (hence, no inter-process messaging; not even ETS). Still, the reaction time is in the order of 20 to 60 microseconds - that is, nowhere closer to the benchmark reaction time (5 microseconds).
>
> What I wish to hear from the experienced people on this forum here is whether I am missing on something in my Erlang code, or should I just go ahead and program in C.
>
> Thanks.
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What does "soft" real-time mean?

Avinash Dhumane-4
In reply to this post by Aleksander Nycz-2
Hello!

The application runs directly on latest Intel Xeon.

The source code of specific module in question is emailed to you separately.

There is a process per instance of "algorithm trading strategy", but no two processes share anything except the ordering socket to the market, which is accessed through synchronization (ETS). But, during the market test, we run only one or two strategies so there is no synchronization overhead.

The major computation is market depth update and price matching.

On Fri, Oct 20, 2017 at 6:55 PM, Aleksander Nycz <[hidden email]> wrote:

Hello,


My private definition:

Hard real time -> 100% requests/jobs/processes must be completed before deadline

Soft real time -> N% requests/jobs/processes must be completed before deadline, where N < 100%,

    so we can/must accept that some jobs will not be finished on time.


And few question:

1. Erlang VM is running on Virtual Machine on bare metal?

2. What you erlang process really do? Can you show us some code?


Regards

Aleksander Nycz


W dniu 2017-10-20 o 14:00, Avinash Dhumane pisze:
I have programmed high-frequency trading application in Erlang and comparing its reaction performance in the live market with other competing applications (mostly, in C).

It has been a consistent observation that my Erlang application lags substantially behind the competing applications. In fact, it never matched the reaction time of the so-called "slowest" C application in the market. 

The benchmark reaction time is about 5 microseconds. It is the time elapsed between tick to order. That is, the difference between the timestamps when I receive the tick (market event) from the mailbox of my process and when I submit the TCP send call on my order.

My process maintains the market depths of the securities and when my desired price (as a function of best buyers and/or sellers on multiple legs of the order) is observed, I compute the order terms and submit. 

The entire computation and communication (with the market) takes place within the single process and there are no collaborating processes (hence, no inter-process messaging; not even ETS). Still, the reaction time is in the order of 20 to 60 microseconds - that is, nowhere closer to the benchmark reaction time (5 microseconds).

What I wish to hear from the experienced people on this forum here is whether I am missing on something in my Erlang code, or should I just go ahead and program in C.

Thanks.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

-- 
Aleksander Nycz
Chief Designer
Telco_021 BSS R&D
Comarch SA
Phone:  +48 17 785 5909
Mobile: +48 691 464 275
website: www.comarch.pl

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What does "soft" real-time mean?

Joe Armstrong-2
Hard real time: All requests MUST be satisfied within a specified time.
Note this does not mean fast. If the deadline is five years and you respond
with a year it's hard real time :-)

Soft real time: Some requests might miss the deadlines.

Fast hard real time is really difficult. Here you'll need special
purpose hardware, a custom OS and a special programming language.

The programming language must be constructed so that you can prove
assertions about the worse case execution time of code. The OS must not
introduce unexpected delays, and the hardware must have predictable delays.

Hard real time is mostly associated with hardware control -  examples:
Braking systems in cars, control system in unstable aircraft, pulling
out fuel rods in a nuclear reactor if things go wrong.

In the software world, hard real time usually means "fast with
predictable delays" (but peoples idea of what is fast varies)

Erlang was designed for programming telephone exchanges - we wanted
response times of milliseconds for simple requests. We also didn't want
long lived computations to block computations that could be performed
immediately - so each process gets an equal time slice of the CPU.

Telecoms protocols are designed with failure in mind - so if the odd
deadline is missed the system just tries again.

My mental model of a telecomms system is that we have tens to hundreds
of thousands of long-lived connections. Each of these sleeps for a
very long
time (from the computer's point of view) then wakes up and requests a
computation which takes (from the computer's point of view) a very
small time.

This is what Erlang was designed for - hundreds of thousands of processes
that are largely inactive - then they wake up and request a small
amount of computation. The odd process might request a large amount of
processing
but this should not block the other processes. Also some processes will fail,
due to software errors, the errors should be logged and not damage the
unfailing parts of the system. Responses times should be of the order
of milliseconds.

This model works pretty well for large multiplexing web-servers (like
WhatsApp) - the VM has be designed for fast context switching,
fast process creation and fast message passing and have process
isolation and be non-blocking.

/Joe





On Fri, Oct 20, 2017 at 3:58 PM, Avinash Dhumane <[hidden email]> wrote:

> Hello!
>
> The application runs directly on latest Intel Xeon.
>
> The source code of specific module in question is emailed to you separately.
>
> There is a process per instance of "algorithm trading strategy", but no two
> processes share anything except the ordering socket to the market, which is
> accessed through synchronization (ETS). But, during the market test, we run
> only one or two strategies so there is no synchronization overhead.
>
> The major computation is market depth update and price matching.
>
> On Fri, Oct 20, 2017 at 6:55 PM, Aleksander Nycz
> <[hidden email]> wrote:
>>
>> Hello,
>>
>>
>> My private definition:
>>
>> Hard real time -> 100% requests/jobs/processes must be completed before
>> deadline
>>
>> Soft real time -> N% requests/jobs/processes must be completed before
>> deadline, where N < 100%,
>>
>>     so we can/must accept that some jobs will not be finished on time.
>>
>>
>> And few question:
>>
>> 1. Erlang VM is running on Virtual Machine on bare metal?
>>
>> 2. What you erlang process really do? Can you show us some code?
>>
>>
>> Regards
>>
>> Aleksander Nycz
>>
>>
>> W dniu 2017-10-20 o 14:00, Avinash Dhumane pisze:
>>
>> I have programmed high-frequency trading application in Erlang and
>> comparing its reaction performance in the live market with other competing
>> applications (mostly, in C).
>>
>> It has been a consistent observation that my Erlang application lags
>> substantially behind the competing applications. In fact, it never matched
>> the reaction time of the so-called "slowest" C application in the market.
>>
>> The benchmark reaction time is about 5 microseconds. It is the time
>> elapsed between tick to order. That is, the difference between the
>> timestamps when I receive the tick (market event) from the mailbox of my
>> process and when I submit the TCP send call on my order.
>>
>> My process maintains the market depths of the securities and when my
>> desired price (as a function of best buyers and/or sellers on multiple legs
>> of the order) is observed, I compute the order terms and submit.
>>
>> The entire computation and communication (with the market) takes place
>> within the single process and there are no collaborating processes (hence,
>> no inter-process messaging; not even ETS). Still, the reaction time is in
>> the order of 20 to 60 microseconds - that is, nowhere closer to the
>> benchmark reaction time (5 microseconds).
>>
>> What I wish to hear from the experienced people on this forum here is
>> whether I am missing on something in my Erlang code, or should I just go
>> ahead and program in C.
>>
>> Thanks.
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>> --
>> Aleksander Nycz
>> Chief Designer
>> Telco_021 BSS R&D
>> Comarch SA
>> Phone:  +48 17 785 5909
>> Mobile: +48 691 464 275
>> website: www.comarch.pl
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What does "soft" real-time mean?

Miles Fidelman
In reply to this post by Avinash Dhumane-4

"Hard real time" generally refers to things where exact timing matters - things like machine control, sensor-to-weapon linkages, autopilots, software-defined radio, and such.  As some definitions put it, missing a deadline is considered a system failure.  Generally, hard real time code involves direct hardware access, referenced to real-time clocks, and you can't miss any events.

"Soft real time" generally refers to things that are still "real-time" but where things are "looser."  For example, VoIP is time sensitive, but can tolerate the occasional lost packet.  Other kinds of soft real-time can tolerate a little jitter in the timing.

Another term you'll sometimes come across is "near real time" - which tends to imply that a little delay is tolerable.

More generally, "hard real time" is exactly that.  Everything else involves loosened constraints.

When it comes to languages & run-times, anything that relies on asynchronous operating system services is not going to support hard real-time.  Hard real-time generally requires that software run very close to bare iron, with complete control of the hardware (i.e, no interrupts, no pre-emption), and access to very accurate timing data.  Erlang is known for high performance, but is very far from able to support hard real-time.

Miles Fidelman


On 10/20/17 8:00 AM, Avinash Dhumane wrote:
I have programmed high-frequency trading application in Erlang and comparing its reaction performance in the live market with other competing applications (mostly, in C).

It has been a consistent observation that my Erlang application lags substantially behind the competing applications. In fact, it never matched the reaction time of the so-called "slowest" C application in the market. 

The benchmark reaction time is about 5 microseconds. It is the time elapsed between tick to order. That is, the difference between the timestamps when I receive the tick (market event) from the mailbox of my process and when I submit the TCP send call on my order.

My process maintains the market depths of the securities and when my desired price (as a function of best buyers and/or sellers on multiple legs of the order) is observed, I compute the order terms and submit. 

The entire computation and communication (with the market) takes place within the single process and there are no collaborating processes (hence, no inter-process messaging; not even ETS). Still, the reaction time is in the order of 20 to 60 microseconds - that is, nowhere closer to the benchmark reaction time (5 microseconds).

What I wish to hear from the experienced people on this forum here is whether I am missing on something in my Erlang code, or should I just go ahead and program in C.

Thanks.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

-- 
In theory, there is no difference between theory and practice.
In practice, there is.  .... Yogi Berra

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What does "soft" real-time mean?

Jesper Louis Andersen-2
In reply to this post by Avinash Dhumane-4
On Fri, Oct 20, 2017 at 3:12 PM Avinash Dhumane <[hidden email]> wrote:
What I wish to hear from the experienced people on this forum here is whether I am missing on something in my Erlang code, or should I just go ahead and program in C.


Your problem is one where you want to have a language which can execute computations quickly, and preferably well under 1ms. In that case, you often want explicit concurrency modeling as well as fast computation. Languages such a C, C++, Rust and OCaml are good candidates.

Soft realtime isn't that well-defined in the literature. But one can define it from hard realtime. Other people have already written about what hard realtime is, so I'm not going to bother with that. But I do note one thing about hard-realtime systems in the context of concurrency which I want to make:

If you have an existing schedule of processes, and you want to add a new one, you must say how often you want to be called, and for how long you will run. Now, the system can either accept your process, in which case it must fulfill the periodic schedulings you wanted. If you use too much time, it is valid for the system to kill your process. Importantly, if the scheduler cannot fit you into its schedule, it must reject starting your process. So if you ask that 10000 processes be started at the same point in time, the scheduler must reject you, unless it has access to 10000 processor cores.

Now, soft realtime is like hard realtime, but there is an SLA and certain objectives which must be met. For instance that 99th percentile of all requests are handled within 2ms. The last percent is ignored and we make no rules for those. Also note that "soft" realtime is a practical thing: you measure the system and fail it if it doesn't meet the SLA.

Erlang, as a system is good for handling such soft realtime requirements, provided that they tend to fall in the millisecond range and that you don't excessively load the system with work. Then you can stay within the SLA in practice, easily.

However, if your task is that you want to answer a query in less than a millisecond, you need something closer to the steel. C is one option, but note that if you want a nanosecond computation, you probably want to move from C to FPGAs. Often, you also want explicitly control over what you are doing. If you hog the core with something else, then even switching away from the current work to the newly arrived work can be enough to mess with your timing schedule. Likewise, if the operating system takes an interrupt at the wrong point in time, your timing will be off.

Hence, a good solution starts by defining the acceptable window of lag before answering what language to use. At, say, 5us, you have around 20 memory reads from DRAM into cache lines. Tight schedule which requires you have some control you wont have in a typical Erlang system, even if you factor out the computation to a NIF. Even in C, this will be hard to get close to. If your process is switched out, you are looking at TLB misses as well and it can easily prod you into a direction where you can only answer in 5us for synthetic benchmarks. The trick, of course, is to "cheat" by doing more than a single piece of work in say 50us and then divide down to claim that it is 5us. Or by keeping the data set small enough it fits into L1 or L2.

Erlang could be a nice tool for the orchestration on top of the C programs however, if you happen to need some massive concurrency there.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions