Making erlang more realtime?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Making erlang more realtime?

Vans S
I currently need to stream 60 frames per second of video and I made a simple test case using erlang.

I have 1 gen_server doing a erlang:send_after every 17ms, when the handle_info procs it calls

os:perf_counter(1000) then compares the last time to the current, if the difference is > 20,
print to console.

Iv noticed that about every 2-3 seconds, the send_after arrives around in 20-23ms, this is not
respecting the millisecond realtime nature that I thought Erlang can provide.

Is there any VM arguments that could be passed to increase the realtime properties or other
things that can be done?

The testcase is just a naked rebar app with 1 gen_server not doing anything else.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Dmytro Lytovchenko
Making it "more realtime" has little sense, as it is not realtime platform and never was, and never intended to be.

A few points to consider.
1. Erlang is not a realtime platform, i.e. it does not give you millisecond/microsecond guarantees. It is soft realtime, which guarantees that your code will eventually  and periodically get CPU time with some +/- 100 ms precision depending on the current load.

2. If you want such millisecond precision you could write the critical decoding/feeding part in native code with an appropriate tool such as C/C++/Rust/Ocaml/Java/C# etc while Erlang will do the rest of the management and orchestration in your system.

3. One more way would be to let Erlang decode video ahead of time, so that you have few frames ready and cached for such a hiccup moment. Then even if some frame is late, there is always another frame ready to cover for it. Then let video player smooth the hiccups. I think streaming services do this at the cost of significant caching ahead.

2017-03-02 21:17 GMT+01:00 Vans S <[hidden email]>:
I currently need to stream 60 frames per second of video and I made a simple test case using erlang.

I have 1 gen_server doing a erlang:send_after every 17ms, when the handle_info procs it calls

os:perf_counter(1000) then compares the last time to the current, if the difference is > 20,
print to console.

Iv noticed that about every 2-3 seconds, the send_after arrives around in 20-23ms, this is not
respecting the millisecond realtime nature that I thought Erlang can provide.

Is there any VM arguments that could be passed to increase the realtime properties or other
things that can be done?

The testcase is just a naked rebar app with 1 gen_server not doing anything else.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Vans S
Erlang is not a realtime platform, i.e. it does not give you millisecond/microsecond guarantees. It is soft realtime, which guarantees that your code will eventually  and periodically get CPU time with some +/- 100 ms precision 
> depending on the current load.

This was my misunderstanding. I thought Erlang guaranteed each process gets a 1ms (+/- 1) timeslice aka 2000 reductions and everything executes as realtime as possible to 1 millisecond.

And your #2 and #3, I already wiped up a test case in C and it seems il have to use Erlang as the auth/user management with more realtime app to handle the actual streaming.  Even caching the frames, with Erlang It seems I am getting similar realtime performance degradation just doing message passing and netio.


On Thursday, March 2, 2017 3:27 PM, Dmytro Lytovchenko <[hidden email]> wrote:


Making it "more realtime" has little sense, as it is not realtime platform and never was, and never intended to be.

A few points to consider.
1. Erlang is not a realtime platform, i.e. it does not give you millisecond/microsecond guarantees. It is soft realtime, which guarantees that your code will eventually  and periodically get CPU time with some +/- 100 ms precision depending on the current load.

2. If you want such millisecond precision you could write the critical decoding/feeding part in native code with an appropriate tool such as C/C++/Rust/Ocaml/Java/C# etc while Erlang will do the rest of the management and orchestration in your system.

3. One more way would be to let Erlang decode video ahead of time, so that you have few frames ready and cached for such a hiccup moment. Then even if some frame is late, there is always another frame ready to cover for it. Then let video player smooth the hiccups. I think streaming services do this at the cost of significant caching ahead.

2017-03-02 21:17 GMT+01:00 Vans S <[hidden email]>:
I currently need to stream 60 frames per second of video and I made a simple test case using erlang.

I have 1 gen_server doing a erlang:send_after every 17ms, when the handle_info procs it calls

os:perf_counter(1000) then compares the last time to the current, if the difference is > 20,
print to console.

Iv noticed that about every 2-3 seconds, the send_after arrives around in 20-23ms, this is not
respecting the millisecond realtime nature that I thought Erlang can provide.

Is there any VM arguments that could be passed to increase the realtime properties or other
things that can be done?

The testcase is just a naked rebar app with 1 gen_server not doing anything else.
______________________________ _________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/ listinfo/erlang-questions




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Ferenc Holzhauser
In reply to this post by Dmytro Lytovchenko
Probably it does not help an awful lot, but I recall an interesting project from couple of years ago called erlyvideo (later flussonic). I believe they have been busy with a similar challenge (video streaming with Erlang) and put a lot of work into tweaking performance. Perhaps you can still find interesting hints and discussions in the mailing list archive (or elsewhere) when searching for this project.

On 2 March 2017 at 21:26, Dmytro Lytovchenko <[hidden email]> wrote:
Making it "more realtime" has little sense, as it is not realtime platform and never was, and never intended to be.

A few points to consider.
1. Erlang is not a realtime platform, i.e. it does not give you millisecond/microsecond guarantees. It is soft realtime, which guarantees that your code will eventually  and periodically get CPU time with some +/- 100 ms precision depending on the current load.

2. If you want such millisecond precision you could write the critical decoding/feeding part in native code with an appropriate tool such as C/C++/Rust/Ocaml/Java/C# etc while Erlang will do the rest of the management and orchestration in your system.

3. One more way would be to let Erlang decode video ahead of time, so that you have few frames ready and cached for such a hiccup moment. Then even if some frame is late, there is always another frame ready to cover for it. Then let video player smooth the hiccups. I think streaming services do this at the cost of significant caching ahead.

2017-03-02 21:17 GMT+01:00 Vans S <[hidden email]>:
I currently need to stream 60 frames per second of video and I made a simple test case using erlang.

I have 1 gen_server doing a erlang:send_after every 17ms, when the handle_info procs it calls

os:perf_counter(1000) then compares the last time to the current, if the difference is > 20,
print to console.

Iv noticed that about every 2-3 seconds, the send_after arrives around in 20-23ms, this is not
respecting the millisecond realtime nature that I thought Erlang can provide.

Is there any VM arguments that could be passed to increase the realtime properties or other
things that can be done?

The testcase is just a naked rebar app with 1 gen_server not doing anything else.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Max Lapshin-2
main question here is:  what does requires sending frames exactly once per 17 ms?

Browser do not require it and erlang precision is enough.

C may become mandatory if you want to send about 500-800 mbit of video, mux several streams to single multi-stream and maintain strict constant bitrate via UDP to send all this to dumb IP -> DVB-C transmitter.

But usually erlang is enough: we are feeding satellite encoder from flussonic now and it is ok.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Ilya Shcherbak-2
hi, 

sorry, but the issue is unclear for me. can you describe a bit more specific a high level issue. why do you need to send frames in 17ms interval? it shouldn't be a requirement for a sender by design in my understanding.  
 
The constant delay doesn't look like a right solution for any reason:
- network spikes won't be resolved due to difference I/P/B frame sizes; (need to have a smart algorithm of a sender queue)
- playback sync won't be resolved due to network/receiver fluctuations; (need to have JB on a receiver)

You can check an implementation of a sender queue in WebRTC project: modules/pacing/paced_sender.cc for instance.

If you are transmitting high res video in a local network and facing an issue with transmitting you can try to tune socket buffer sizes for send/recv.

Probably i missed something and you can add more information about an initial task.





2017-03-03 22:23 GMT+07:00 Max Lapshin <[hidden email]>:
main question here is:  what does requires sending frames exactly once per 17 ms?

Browser do not require it and erlang precision is enough.

C may become mandatory if you want to send about 500-800 mbit of video, mux several streams to single multi-stream and maintain strict constant bitrate via UDP to send all this to dumb IP -> DVB-C transmitter.

But usually erlang is enough: we are feeding satellite encoder from flussonic now and it is ok.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Vans S
> Probably it does not help an awful lot, but I recall an interesting project from couple of years ago called erlyvideo (later flussonic)


Pretty interesting project I would have to dig into the erlyvideo source sometime.

> main question here is:  what does requires sending frames exactly once per 17 ms?

17 ms was just a random number I picked. Since ~16.777ms represents 60fps. I wanted to see how much extra delay could potentially happen per frame.
Realistically we want to forward frames as soon as we receive them.  Erlang is just routing, not actually encoding the frames. I wanted to see if I could
measure slowdowns between frames, for example if frame 1 spends ~6ms to be forwarded by erlang, we could arrive too late on the client end and the client might need to
skip a frame.

> Browser do not require it and Erlang precision is enough.
Im thinking so too now, I have also gotten better result (lower latency) setting process priority to max.


> - network spikes won't be resolved due to difference I/P/B frame sizes; (need to have a smart algorithm of a sender queue)
> You can check an implementation of a sender queue in WebRTC project: modules/pacing/paced_sender.cc for instance.
I need to consider this.  Im not sure if it is possible to drop x264 frames though if the end-to-end transmission is bottle necked.
I think the solution would be to drop to a lower bitrate.

> - playback sync won't be resolved due to network/receiver fluctuations; (need to have JB on a receiver)Not sure what JB is.

> Probably i missed something and you can add more information about an initial task.
I am getting clumps of frames arriving in the client using websockets, which is a NaCL web browser application for Google Chrome (localnet). I am trying
to narrow down the cause of it.  I think I have arrived that it might be because I am sending large ws frames (130kb+) and websocket in chrome chokes,
resulting in 5-8 frames being clumped up and presented to the decoder at the same time.

I am going to try using a basic TCP connection to the client to see if that fixes the issue.  This thought will unfortunately limit my application to
running as an extension only.



On Friday, March 3, 2017 12:44 PM, Ilya Shcherbak <[hidden email]> wrote:



hi,

sorry, but the issue is unclear for me. can you describe a bit more specific a high level issue. why do you need to send frames in 17ms interval? it shouldn't be a requirement for a sender by design in my understanding.  
 

The constant delay doesn't look like a right solution for any reason:
- network spikes won't be resolved due to difference I/P/B frame sizes; (need to have a smart algorithm of a sender queue)
- playback sync won't be resolved due to network/receiver fluctuations; (need to have JB on a receiver)

You can check an implementation of a sender queue in WebRTC project: modules/pacing/paced_sender.cc for instance.

If you are transmitting high res video in a local network and facing an issue with transmitting you can try to tune socket buffer sizes for send/recv.

Probably i missed something and you can add more information about an initial task.






2017-03-03 22:23 GMT+07:00 Max Lapshin <[hidden email]>:

main question here is:  what does requires sending frames exactly once per 17 ms?

>
>
>Browser do not require it and erlang precision is enough.
>
>
>C may become mandatory if you want to send about 500-800 mbit of video, mux several streams to single multi-stream and maintain strict constant bitrate via UDP to send all this to dumb IP -> DVB-C transmitter.
>
>
>But usually erlang is enough: we are feeding satellite encoder from flussonic now and it is ok.
>______________________________ _________________
>erlang-questions mailing list
>[hidden email]
>http://erlang.org/mailman/ listinfo/erlang-questions
>
>


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Max Lapshin-2
1) You need to start fixing your decoder that have to handle network delays.

2) if you write a proxy, then you don't need to worry about "realtime":  receive frame and forward it.  6ms will be spent inside erlang only if you write at least 100-300K of useless LOC that do nothing. In flussonic we measure microseconds.

3) you need to stop thinking about "latency and realtime" in erlang and start looking at tcpdump and understanding that TCP is a source of latency problem in p2p video streaming.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Tony Rogvall-2
In reply to this post by Vans S
FYI
>
>> Browser do not require it and Erlang precision is enough.
> Im thinking so too now, I have also gotten better result (lower latency) setting process priority to max.
>
From the Erlang documentation: http://erlang.org/doc/man/erlang.html#process_flag_priority
"Note
Priority level max is reserved for internal use in the Erlang runtime system, and is not to be used by others.”

/Tony


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Jesper Louis Andersen-2
In reply to this post by Vans S
On Thu, Mar 2, 2017 at 9:17 PM Vans S <[hidden email]> wrote:
I currently need to stream 60 frames per second of video and I made a simple test case using erlang.


Erlang is in many ways a "best effort" realtime system. It tries very hard to deliver timers and handle deadlines quickly, but it has no built-in provisioning to guarantee that they will be delivered on time.

Some systems, which people call "hard" realtime systems, work by you requesting a certain periodic schedule. For instance: I want to be called every 17ms and I will spend at most 2ms of processing time. The system then either accepts that schedule or rejects it based on current workload. Such a system can still fail: CPUs can go bad, and even if you run multiple CPUs in unison and vote on the correct result, the whole system can go bad due to too many failures.

Erlangs best-effort delivery can go bad because of numerous reasons:

* You requested 1,000,000 processes to be woken up on the same point in time. Some will have to wait and come later.
* You run on virtualized hardware, and some other task got the CPU.
* You are on battery and not AC on your laptop and the system runs at different clock rates.
* You are running OSX, which recently started saving power by piggybacking several wakeups together.
* You enabled another CPU, which has to become online and then filled with the relevant data before it can process.

Not all of these are the fault of the Erlang subsystem. In general, the only way to get a timely deadline is to overprovision the system enough that it has lots of free time to do what it wants.

That said, I'll say the same as Dmytro, Max, Ferenc and Ilya: try to build your system such that it starts processing as soon as a frame arrives and push it onwards. In a certain sense, what you should be worried about is the frame time, not the frame delay. That is, you should worry about how long it takes to process a frame, because if you start falling behind there, you are in trouble. You may find you don't need utter precision, but can extrapolate a bit around the actual point in time you were woken up. This is where some buffering can be nice. Many computer games toy with this idea: they create the illusion of a 16.7ms frame time, but internally things run on completely different schedules, and it runs with lag: whenever you have a frame in the graphics card buffer, the next one is being painted. This introduces one frame of "lag", but humans are able to learn how to ignore that when they frag people in CS:GO :P

In short: you are in principle requesting that Erlang does something it cannot do in its current form.

You can set up an SLA and then verify that the Erlang system keeps within that SLA. e.g., "for my system, timer delivery happens in the same millisecond that I requested 99.999% of the time". If it doesn't, you can try tuning the system in order to meet that requirement: do not run in the cloud, virtualized, eliminte other processes on the machine. Force the CPU clock cycle to be the same all the time. Do not sleep the CPU. Turn off the NIC so you don't get interference on the memory bus. Replace the OS scheduler with one that gives guaranteed schedules, and so on.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Making erlang more realtime?

Vans S
If it doesn't, you can try tuning the system in order to meet that requirement: do not run in the cloud, virtualized, eliminte other processes on the machine. Force the CPU clock cycle to be the same all the time.
> Do not sleep the CPU. Turn off the NIC so you don't get interference on the memory bus. Replace the OS scheduler with one that gives guaranteed schedules, and so on.

I started doing this to hunt down bottlenecks, I found issues all throughout my stack.

I am going to try again sometime soon.  Also I want to try tuning the kernel params for lower latency, I think now that its not Erlangs fault at all here and Erlang can be quite performant down to ~1ms consistent.
The send_after delayed timer resolution issues I was having I think are related to the kernel; proper isolation of erlang runtime from the rest of the OS and scheduler pinning I think will make erlang much more realtime.


On Saturday, March 4, 2017 4:42 PM, Jesper Louis Andersen <[hidden email]> wrote:


On Thu, Mar 2, 2017 at 9:17 PM Vans S <[hidden email]> wrote:
I currently need to stream 60 frames per second of video and I made a simple test case using erlang.



Erlang is in many ways a "best effort" realtime system. It tries very hard to deliver timers and handle deadlines quickly, but it has no built-in provisioning to guarantee that they will be delivered on time.

Some systems, which people call "hard" realtime systems, work by you requesting a certain periodic schedule. For instance: I want to be called every 17ms and I will spend at most 2ms of processing time. The system then either accepts that schedule or rejects it based on current workload. Such a system can still fail: CPUs can go bad, and even if you run multiple CPUs in unison and vote on the correct result, the whole system can go bad due to too many failures.

Erlangs best-effort delivery can go bad because of numerous reasons:

* You requested 1,000,000 processes to be woken up on the same point in time. Some will have to wait and come later.
* You run on virtualized hardware, and some other task got the CPU.
* You are on battery and not AC on your laptop and the system runs at different clock rates.
* You are running OSX, which recently started saving power by piggybacking several wakeups together.
* You enabled another CPU, which has to become online and then filled with the relevant data before it can process.

Not all of these are the fault of the Erlang subsystem. In general, the only way to get a timely deadline is to overprovision the system enough that it has lots of free time to do what it wants.

That said, I'll say the same as Dmytro, Max, Ferenc and Ilya: try to build your system such that it starts processing as soon as a frame arrives and push it onwards. In a certain sense, what you should be worried about is the frame time, not the frame delay. That is, you should worry about how long it takes to process a frame, because if you start falling behind there, you are in trouble. You may find you don't need utter precision, but can extrapolate a bit around the actual point in time you were woken up. This is where some buffering can be nice. Many computer games toy with this idea: they create the illusion of a 16.7ms frame time, but internally things run on completely different schedules, and it runs with lag: whenever you have a frame in the graphics card buffer, the next one is being painted. This introduces one frame of "lag", but humans are able to learn how to ignore that when they frag people in CS:GO :P

In short: you are in principle requesting that Erlang does something it cannot do in its current form.

You can set up an SLA and then verify that the Erlang system keeps within that SLA. e.g., "for my system, timer delivery happens in the same millisecond that I requested 99.999% of the time". If it doesn't, you can try tuning the system in order to meet that requirement: do not run in the cloud, virtualized, eliminte other processes on the machine. Force the CPU clock cycle to be the same all the time. Do not sleep the CPU. Turn off the NIC so you don't get interference on the memory bus. Replace the OS scheduler with one that gives guaranteed schedules, and so on.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...