The export speed performance is killing me yet again.
I had an event occur last night and the police are investigating. I have footage that they need, but I've been waiting over 3 hours for the first of 3 exports to complete. Looks like it'll take at least 4 hours, if not 5, for each export so I won't be able to get them all the footage until tomorrow. Which is ridiculous!
My NVR is running on a 12th gen Core i7 with 10 cores / 16 threads, 32G RAM, and pure SSD storage. It is asleep running 10 cameras. It has plenty of resources available to help with the export. I know others have run into this, but for this export performance to be so terrible is very frustrating.
I'm currently using the default BVR storage format, but without an export solution, what would I lose by changing to MP4 file storage? I can do that, can't I? As far as I can tell, BVR files are only causing me great pain! If I was able to just copy the files and edit them with standard video tools, that would be a breath of fresh air right now!
Export speed tricks??
Re: Export speed tricks??
I just exported a video clip and it took just a few minutes, the clip was only 4 minutes long, how big is the file you are trying to export and how are you doing the export, I just right clicked on a file in the all clips list and selected Convert/export!
Re: Export speed tricks??
Well sure, exporting a 4min clip should take seconds. lol
These are 30 min to 1hr clips. Which is “long”, sure, but that same clip after export can be ripped through with ffmpeg or clip champ on that same computer in 30 minutes or less. (Speeding up or resizing for example.)
The difference is that when BI is exporting, the CPU and GPU stay basically idle. 20-30% load while just recording and it MIGHT increase by 5% while exporting, but sometimes even goes DOWN. When using any other video tool on this PC, both the CPU and GPU go to 100%. BI just isn’t using the resources that are available.
Any input on what I would lose by switching to just recording to mp4 and being able to skip the export re-encoding process?
These are 30 min to 1hr clips. Which is “long”, sure, but that same clip after export can be ripped through with ffmpeg or clip champ on that same computer in 30 minutes or less. (Speeding up or resizing for example.)
The difference is that when BI is exporting, the CPU and GPU stay basically idle. 20-30% load while just recording and it MIGHT increase by 5% while exporting, but sometimes even goes DOWN. When using any other video tool on this PC, both the CPU and GPU go to 100%. BI just isn’t using the resources that are available.
Any input on what I would lose by switching to just recording to mp4 and being able to skip the export re-encoding process?
Re: Export speed tricks??
Well my experiment with MP4 native recording is a bust at this point. While using MP4 as a file type for recording reviewing footage broke entirely. Reviewing footage that was only a few minutes ago produces an error that says you can only do that with BVR and reviewing footage from a couple hours ago just fails with a slew of errors. The local UI mentions needing a K-Lite codec so maybe I'll go down that path at some point, but with MP4 set as the recoding format the clips are all chaotic. With BVR I got nice consistent 1h clips, which is what I want. My settings are set to 1hr / 3G clip splits. With MP4 however, it's like it doesn't know that setting exists. I get 5-10 clips per hour with no rhyme or reason. 6min clips, 7min clips, 15 min clips, 55min clips, with file sizes all over the place of 200M, 300M, 700M, and 2G. Even if I got the footage reviewing working, those chaotic clips are bonkers.
In the file format settings in the video compression section there is a re-encode option there. I can't find documentation on what exactly that does though. The File (container) format setting is different, and that's now set back to BVR for me. In the compression section I had things set to direct-to-disk, but I'm wondering if using the re-encode option there will produce TWO files per clip or what? Really disappointed in the documentation for BI for things like this. Reading every page that mentions the term, watching 20 videos on the topic, and while some of them mention particular features, and a few might even suggest settings, almost none of them explain WHAT something like this IS or why one might want to use it or what the impact of using it would be. For this re-encode feature, the best I've found so far is a video that says "just leave it at BVR, that's best, and leave it set to direct unless you want to re-encode". I don't know if I want to re-encode... how about 2 more seconds of saying "re-encoding does xyz and replaces your BVR with another format, or adds a second file or whatever".
In the file format settings in the video compression section there is a re-encode option there. I can't find documentation on what exactly that does though. The File (container) format setting is different, and that's now set back to BVR for me. In the compression section I had things set to direct-to-disk, but I'm wondering if using the re-encode option there will produce TWO files per clip or what? Really disappointed in the documentation for BI for things like this. Reading every page that mentions the term, watching 20 videos on the topic, and while some of them mention particular features, and a few might even suggest settings, almost none of them explain WHAT something like this IS or why one might want to use it or what the impact of using it would be. For this re-encode feature, the best I've found so far is a video that says "just leave it at BVR, that's best, and leave it set to direct unless you want to re-encode". I don't know if I want to re-encode... how about 2 more seconds of saying "re-encoding does xyz and replaces your BVR with another format, or adds a second file or whatever".
Re: Export speed tricks??
I did some further testing on a much larger file, the BVR file is 3,738,510 KB in size and is 1:30:26 hours of video.
The export was processing at about 26.3 FPS, the CPU usage was 14-15%, this system has no separate GPU; only the built in AMD graphics on the chip.
It took my system 57 minutes to complete the export to MP4 the exported video file is 15,150,487 KB
The computer is an ASRock DeskMeet x300 with a AMD socket AM4 Ryzen 7 5700G and 64GB of RAM
The export was processing at about 26.3 FPS, the CPU usage was 14-15%, this system has no separate GPU; only the built in AMD graphics on the chip.
It took my system 57 minutes to complete the export to MP4 the exported video file is 15,150,487 KB
The computer is an ASRock DeskMeet x300 with a AMD socket AM4 Ryzen 7 5700G and 64GB of RAM
Re: Export speed tricks??
Thank you for that!
What resolution is the source video?
All of my cameras are 5MP and I'm recording them at 5MP and do not resize any of them during export. I do however need to re-encode to preserve the date/time overlay as that is paramount in most cases to tie the visuals to the time they occurred. I've read many suggestions about using the camera settings to produce that overlay, but I haven't found a camera yet that can keep reliable time even with NTP configured (including handling time zones correctly), or that can produce a nice looking overlay. To me, having a single source of truth for the timestamps (the NVR itself) makes the most logical sense, but since that adds overhead in BI, I can get why people shy away from that. It's too important to me to be able to have consistent and accurate timestamps though to rely on the cameras.
Why I can't "bake in" the overlays during recording is a question though. That would be the ideal compromise. Baking it into the video as it was being recorded would remove the need to re-encode on export.
What resolution is the source video?
All of my cameras are 5MP and I'm recording them at 5MP and do not resize any of them during export. I do however need to re-encode to preserve the date/time overlay as that is paramount in most cases to tie the visuals to the time they occurred. I've read many suggestions about using the camera settings to produce that overlay, but I haven't found a camera yet that can keep reliable time even with NTP configured (including handling time zones correctly), or that can produce a nice looking overlay. To me, having a single source of truth for the timestamps (the NVR itself) makes the most logical sense, but since that adds overhead in BI, I can get why people shy away from that. It's too important to me to be able to have consistent and accurate timestamps though to rely on the cameras.
Why I can't "bake in" the overlays during recording is a question though. That would be the ideal compromise. Baking it into the video as it was being recorded would remove the need to re-encode on export.
Re: Export speed tricks??
The video main stream is 1280 x 720
I have all of my cameras set to sync to a local NTP server on my router, they always stay set to the correct time and are each individually set to maintain DST in the camera settings. I have read that putting the time stamp on using BI can result in claims that the video has been edited and the evidence thrown out in court!
I am going to try doing the export with different setting other than what ever the default is in BI to see what happens!
I have all of my cameras set to sync to a local NTP server on my router, they always stay set to the correct time and are each individually set to maintain DST in the camera settings. I have read that putting the time stamp on using BI can result in claims that the video has been edited and the evidence thrown out in court!
I am going to try doing the export with different setting other than what ever the default is in BI to see what happens!
Re: Export speed tricks??
That helps. Your resolution is a lot less than mine. ~1MP versus 5MP. That may be a major factor in my performance issues. Not because the PC isn't up to it, as it is mostly asleep, but because the export process seems limited to 4 threads no matter what setting you put in the threads option. I have at times seen 4 cores maxed out. Never more than 4 and most times even while 4 are busy, they're not maxed out. Maybe the export would be impressive at 1080, but it just bogs down at 5MP?
I may explore dropping the resolution down as part of my performance efforts. I may also create a copy of certain cameras and have those copies save their video as MP4. Doesn't make sense to do that for all, but there are a few cameras that it could make sense for. Along those lines though, all of my cameras have sub-streams coming in. I wonder if there is a way to save both the main and sub streams? If I could choose to export the sub-stream in certain cases, that would be very helpful. For example, for this recent event, I exported 2 of my camera views purely for their audio content. The sub-stream quality would have been fine for those as the audio was just as good in the sub-stream.
I've configured NTP on every camera I've used over the years, and it's never been consistent. I've used Dahua, Reolink, and now Amcrest cameras (and several other random knock-offs that are exactly the same hardware and software). They SEEM to work upon initial setup, but many have manual DST settings versus automatic for one, so twice a year I have to touch them manually, or some have been buggy where even though they claim to sync with NTP every hour, the time drifts inside that hour by up to 5 minutes. Aside from poor timekeeping, the size and quality of the overlays built into cameras have left me unsatisfied. Most don't have any settings for font type or size, or if they do, it's very limited. And even on my current 8 cameras, 7 of which are Amcrest, the same overlay settings don't always result in identical looking overlays. This is nitpicky for sure but being able to have a consistent and perfectly matching overlay on all channels is a breath of fresh air. Being able to make a change to one and copy it to the others is really nice as well versus having to use the often clunky camera interface on each camera to change the overlays. As one example of my use case, I have a weather camera where I've added an overlay for the cardinal directions to give context to what you're seeing. That isn't possible with the built-in camera overlay without typing a bunch of dashes to force the placement of 3 compass letters where they need to be.
On the overlay point, I've been in IT for banks for 20+ years and ALL bank video systems add overlays, always have even with analog video systems, and the overlay has never really been challenged in court as far as being manipulated. Being the wrong time, sure, but that's easy to verify by any clock caught in a view or opening and closing activity caught on video. Along those lines, the overlay added by the camera could be claimed to be a modification of the "pure" video stream as well. In court cases, simple claims like "it's been modified!" of course I'm sure happen all the time, but the key is not the claim, but what happens next. In probably hundreds of cases over the years, I've only had 1 challenge come back to me where an investigation was done to substantiate or discredit a claim. In that case, the defendant claimed that what occurred was on an entirely different day. The police wanted to see how the timestamp was added to the video and in about 15 min we were able to show them that it's done automatically, and we were also able to show them events from multiple days that provided support for the timestamp being correct. (There was a parade the day prior and a funeral the day after that matched up with our video timestamps if I recall correctly on that one.)
What I've learned by dealing with courts and video over the years is the real world is not like CSI. For example, several camera systems I've deployed have come with these fancy certifications for their video authenticity. The basic gist is that if you use their video format and their player, they promise with all of their might that they can prove in a court of law that the video has not be modified. Which is great, and maybe with a trial of the century on the level of OJ, maybe someone would care about that. However, in EVERY case of real world court in KS, OK, PA, TX, and NE that I've been involved with, they always want us to convert the video to MP4 or AVI or burn it to a DVD for maximum compatibility with the courtroom TV they'll be using. Not once has anyone used the proprietary player because then they'd have to do tech stuff in the courtroom (you know, "hard" things like have a PC with the right software and hooked up to a TV correctly, etc...). Nobody except OJ is paying millions of dollars for a forensic video manipulation expert to discredit the video. Which does beg the question... I wonder how much nefarious editing gets through the court system... because every court I've worked with would have been pretty easy to fool if you tried even a little. But that's a terrifying thought to entertain, so we'll just leave that there for now...
I may explore dropping the resolution down as part of my performance efforts. I may also create a copy of certain cameras and have those copies save their video as MP4. Doesn't make sense to do that for all, but there are a few cameras that it could make sense for. Along those lines though, all of my cameras have sub-streams coming in. I wonder if there is a way to save both the main and sub streams? If I could choose to export the sub-stream in certain cases, that would be very helpful. For example, for this recent event, I exported 2 of my camera views purely for their audio content. The sub-stream quality would have been fine for those as the audio was just as good in the sub-stream.
I've configured NTP on every camera I've used over the years, and it's never been consistent. I've used Dahua, Reolink, and now Amcrest cameras (and several other random knock-offs that are exactly the same hardware and software). They SEEM to work upon initial setup, but many have manual DST settings versus automatic for one, so twice a year I have to touch them manually, or some have been buggy where even though they claim to sync with NTP every hour, the time drifts inside that hour by up to 5 minutes. Aside from poor timekeeping, the size and quality of the overlays built into cameras have left me unsatisfied. Most don't have any settings for font type or size, or if they do, it's very limited. And even on my current 8 cameras, 7 of which are Amcrest, the same overlay settings don't always result in identical looking overlays. This is nitpicky for sure but being able to have a consistent and perfectly matching overlay on all channels is a breath of fresh air. Being able to make a change to one and copy it to the others is really nice as well versus having to use the often clunky camera interface on each camera to change the overlays. As one example of my use case, I have a weather camera where I've added an overlay for the cardinal directions to give context to what you're seeing. That isn't possible with the built-in camera overlay without typing a bunch of dashes to force the placement of 3 compass letters where they need to be.
On the overlay point, I've been in IT for banks for 20+ years and ALL bank video systems add overlays, always have even with analog video systems, and the overlay has never really been challenged in court as far as being manipulated. Being the wrong time, sure, but that's easy to verify by any clock caught in a view or opening and closing activity caught on video. Along those lines, the overlay added by the camera could be claimed to be a modification of the "pure" video stream as well. In court cases, simple claims like "it's been modified!" of course I'm sure happen all the time, but the key is not the claim, but what happens next. In probably hundreds of cases over the years, I've only had 1 challenge come back to me where an investigation was done to substantiate or discredit a claim. In that case, the defendant claimed that what occurred was on an entirely different day. The police wanted to see how the timestamp was added to the video and in about 15 min we were able to show them that it's done automatically, and we were also able to show them events from multiple days that provided support for the timestamp being correct. (There was a parade the day prior and a funeral the day after that matched up with our video timestamps if I recall correctly on that one.)
What I've learned by dealing with courts and video over the years is the real world is not like CSI. For example, several camera systems I've deployed have come with these fancy certifications for their video authenticity. The basic gist is that if you use their video format and their player, they promise with all of their might that they can prove in a court of law that the video has not be modified. Which is great, and maybe with a trial of the century on the level of OJ, maybe someone would care about that. However, in EVERY case of real world court in KS, OK, PA, TX, and NE that I've been involved with, they always want us to convert the video to MP4 or AVI or burn it to a DVD for maximum compatibility with the courtroom TV they'll be using. Not once has anyone used the proprietary player because then they'd have to do tech stuff in the courtroom (you know, "hard" things like have a PC with the right software and hooked up to a TV correctly, etc...). Nobody except OJ is paying millions of dollars for a forensic video manipulation expert to discredit the video. Which does beg the question... I wonder how much nefarious editing gets through the court system... because every court I've worked with would have been pretty easy to fool if you tried even a little. But that's a terrifying thought to entertain, so we'll just leave that there for now...
Re: Export speed tricks??
I read your issues and had to replicate as I haven't exported any video file in many versions. And was a bit worried. I came to the conclusion you are using a potatoe and/or have ignored all best practices, Like NTP. Those 30 years of IT do not help today. How do I know; the same. My 30 years of IT allows me to understand the situation but does not help me to get there with out the help of others and best practices. Here is an example of what it did in about a minute. 4.9GB 
quality of the host site sucks, but I never did this before so have no idea what I should use. https://imgdrop.io/image/0KFtK
This is on a Unraid Win 11 VM assigned 4 normal core and 12 efficiency cores, assigned 32 GBS ram I also have a NVIDIA RTX 3090 GPU pass through. 11 Full time recording main stream. I'm not using any advance 265/264 codec from cameras. Limit camera streams to 15 FPS. Not transcoding audio on exports as I have an issue with the audio card, plays fine locally to VM but not able to transcode right now. It is a virtual machine issue. During export I did not notice any real uptick in hardware usage, both using direct and no direct. The no direct might have been a little higher CPU usage. It acts like a straight copy of a file from one directory to another.
quality of the host site sucks, but I never did this before so have no idea what I should use. https://imgdrop.io/image/0KFtKThis is on a Unraid Win 11 VM assigned 4 normal core and 12 efficiency cores, assigned 32 GBS ram I also have a NVIDIA RTX 3090 GPU pass through. 11 Full time recording main stream. I'm not using any advance 265/264 codec from cameras. Limit camera streams to 15 FPS. Not transcoding audio on exports as I have an issue with the audio card, plays fine locally to VM but not able to transcode right now. It is a virtual machine issue. During export I did not notice any real uptick in hardware usage, both using direct and no direct. The no direct might have been a little higher CPU usage. It acts like a straight copy of a file from one directory to another.