View source for Doing your best to get rid of smudging in FFMPEG transcoding of RTSP streams

[[Various Learnings from getting Zomeminder 1.28.1 working well on Ubuntu Server 14.04|<Home]]



=== Deprecation Notice ===

* A lot of things have changed
* As of 2019, don't use Remote, use FFMPEG
* Don't put in any values for Max FPS and Alarm Max FPS - keep it empty. This always causes problems. Change the FPS settings inside the camera UI (not in ZM)
* Use "TCP" as a method inside FFMPEG, that usually takes care of smearing

Most of the other tips/etc may no longer be valid


===Background===

All my cameras are Foscam HD cameras (1280x960) and support RTSP. Further , 4 out of my 5 cameras are connected via WiFi. 
Zone minder has many options to capture frames, but for Foscam cameras it boils down to:
* Using the Foscam "snapshot2" URL to capture images instead of live streaming
* Using RTSP URL 
* Using MJPEG URL (needs a configuration change to Foscam)

I investigated all 3 options. 
* In terms of quality, Option 1 (image snapshot) is the best. No artifacts, no smudging nothing. The downside is there are a high amount of connection drops, even if I dropped the HD stream to 720p.
* Foscam seems to only stream MJPEG as 640x480. That was not acceptable (and the quality of the stream seemed to have a lot of artifacting)
* So really, the only option left was RTSP


===RTSP via FFMPEG/VLC/Remote ===

Now that RTSP was the only viable option, there are 3 methods you can use (well, maybe 4, not sure if Curl supports RTSP - never tried)
* You can use source type as FFMPEG, VLC or (a new addition with ZM 1.28) Remote


====Source Type: Remote====

Based on [https://github.com/ZoneMinder/ZoneMinder/issues/734 this] issue in github, knnniggett writes that starting ZM 1.28, they have built in their own RTSP handler that does not rely on FFMPEG.
This option may or may not work. For you. It works well for me -no smudging. But if this mode does not work for you, read on

====Source Type: libVLC====

VLC is supposed to be a drop in replacement for FFMPEG. It works, but here is the rub:
* With 5 cameras capturing HD streams, using FFMPEG, my CPU consumption always hovered between 12% - 30%
* The moment I changed ONE camera to VLC, my CPU consumption shot up to 67% and stayed there
* The moment I changed TWO cameras to VLC, I'd hit 97% and then I'd start getting spurts of green screens and other errors

So obviously, VLC was a fail for me.

====Source Type: FFMPEG====

FFMPEG worked, but:
* I noticed a lot of smearing and smudging while FFMPEG transcoded RTSP to images. The smearing would be random and quite frequent. The problem was that this made ZM think there were large changes in frames and it would trigger all sorts of alarms

This is what I mean by smearing:

[[File:Smearing_example.jpeg]]

===Strategies to solve smearing===
(credit: PacoLM, source: http://www.zoneminder.com/forums/viewtopic.php?f=32&t=23031)

So there are a few aspects to addressing this problem, based on various other threads I read, and advice from forum members:
* Make sure RTSP is streaming over TCP not UDP (for whatever reason, there are high packet losses over UDP with Foscam, even within a LAN)
* Make sure you reduce frame rate of the camera directly inside the camera (via its web interface)
* Make sure you send key frame rates more frequently (RTSP reduces bandwidth by sending differential image data. Once it a while it sends "key frames" which is the full frame from the camera to resync)
* <del>Don't mess with FPS inside ZM. Set what you need inside the camera. Apparently, </del> Based on testing done by PacoLM, changing FPS at ZM and having the camera send a different FPS causes problems, so:
* For each camera, I cleared FPS and Alarm FPS inside ZM. <del>Kept them blank</del>. Inside ZM put in an FPS that is greater than the frame rate of the camera. This is important. You don't want ZM to reduce FPS - it causes issues. Instead reduce at camera. The reason I recommend you put in a higher FPS Max in ZM is because when the camera goes offline, ZM takes up max CPU trying to keep connecting to the camera and Max FPS helps curtail this.


Lets take them up one by one:

====RTSP streaming over TCP====

Based on threads I read over at Foscam forums, there is no way to "tell" foscam to stream over TCP if UDP is also available. There are various strategies to work around it, but before you try the various options, 
if you set remote method as "RTSP/RTP" in ZoneMinder, ZM forces a TCP connection. You can validate this by checking with netstat while you are streaming. So chances are if you are using this mode, you are already enforcing TCP
(Note: other mechanisms like routing the source host via you external public IP and setting port forwarding to TCP is unnecessary as I said above, ZM is enforcing TCP. Also adding "?tcp" to the URL caused complete havoc in my system - ZM hung up, DB stopped working due to too many connections)

====Reducing Frame Rate & increasing Key frames in-camera====
I directly accessed my camera admin web interface, and reduced FPS to 3 and key frame rate interval to 10, Frankly, for the purpose of capturing motion alarms, I think an FPS of 1,2,3 are perfectly fine

==== Not messing with FPS inside ZM ====
For each camera, I cleared FPS and Alarm FPS inside ZM. <del>Kept them blank</del>. Inside ZM put in an FPS that is greater than the frame rate of the camera. This is important. You don't want ZM to reduce FPS - it causes issues. Instead reduce at camera. The reason I recommend you put in a higher FPS Max in ZM is because when the camera goes offline, ZM takes up max CPU trying to keep connecting to the camera and Max FPS helps curtail this.

===Outcome===
I was at a stage where I was getting smearing once every few minutes. I now reached a stage where I was getting a few smears maybe once in a few hours. BIG IMPROVEMENT

===Further tuning===
To eliminate the spurious alarms generated by the occasional smear, I changed Misc->Alarm Frame count -  I put in a minimum number of frames that ZM will count for alarms before actually raising an alarm. I analyzed my alarm stats and noticed that whenever smearing happens, it lasts for a max of 2 frames. So I put in a minimum alarm frame count as 3. This completely eliminated false events due to smearing. Obviously, this value may change based on your in-camera FPS. (credit: user kkroft suggested I use this feature to not record events on Signal loss [http://www.zoneminder.com/forums/viewtopic.php?f=32&t=23036 here], which I thought I could apply for the occasional smear too)

Finally, as of April 2015, I stopped using FFMPEG and switched to native RTSP (only av. in 1.28). If it works, it does not smudge at all. But it may not work for you - it seems temperamental; in which case the tips suggested above may help you with FFMPEG

Best of luck!