Memory Leak when enabling CodeProject.AI

[Mace]

After several years of running Sighthound because of its rather solid AI-based object recognition, I decided to move to Blue Iris in combination with CodeProject.AI. So far I'm okay with the results (still working on tweaking the AI part). However I discovered that my system was suffering from a memory leak. After doing several tests, it turned out that either CodeProject.AI, or the combination of CodeProject.AI and Blue Iris is causing the memory leak. With CodeProject.AI enabled, memory usage will start around 40 % and in the course of a day or two this will creep up to 95-99 %, grinding the system to a halt. By disabling AI (disabling CodeProject.AI service, disabling all AI-related items from the Blue Iris setup), memory consumption remains steady at around 36 %. The weird thing is that none of the Windows memory metrics (Memory - active private working set, Commit size, ...) indicate anything near the suggested percentage of excessive memory usage when AI is enabled. Both Blue Iris and CodeProject.AI (including Python) will take up around 1 GB each.

I could schedule a daily reboot of the laptop as a workaround, but of course the root cause needs to be addressed. For now, I've simply disabled CodeProject.AI and am relying on Blue Iris's default motion detection (which seems to work okay).

I'm running Windows 11 Pro on a Lenovo LOQ 15APH8 laptop with 16 GB of RAM, an AMD 7840HS and an RTX 4060. In the BIOS I've selected 'Discrete graphics only' to make sure the RTX 4060 is used at all times.
Blue Iris version is 5.8.2.2 and CodeProject.AI version is 2.3.4.

Anyone else having similar issues regarding the memory leak and can anyone confirm whether it's CodeProject.AI and/or the combination with Blue Iris that is causing it?

Cheers

[MikeBwca]

...

[Mace]

BI 5.8.2.4, and CP 2.2.4. Working well.

I'll try CP 2.2.4 and see what happens. So what's the reason you're running CP 2.2.4 rather than 2.3.4? Did you run into my issue?

[MikeBwca]

...

[TimG]

Running CPAI v2.4.6 RC2 here since yesterday and I haven't had memory leaks with this or any of the previous intermediate versions.

Attempted latest NVIDIA GTX1650 video driver (Up to CUDA 12.3) with CUDA 11.8 tools at the same time. ALPR now doesn't work, but I can live without that until things catch up. AI with custom models is working well.

Secret of my success: Good back ups

FWIW: The two things I have seen cause memory issues were Anti Virus exclusions, and the anti-ransomware component of 'Acronis True Image' which brought down all of my pc's until I disabled it.

[Jaker788]

Have you figured out a solution to this? I have this exact issue with using CPAI. I did learn that you can view this hidden memory usage with a program called rammap, and what it's showing is a TON of "mapped files" that are almost entirely standby memory but for some reason never released, all of the major mapped file memory is the .bvr files. The drive is not the bottleneck in writing, it's idle time is 96-99% all the time. It just seems like between the 2 programs, these files were forgotten and never released from memory. I've tried BI updates, CPAI updates, switching from YOLOv5.net to YOLOv8 CUDA, nothing has changed. Restarting either program also does not release the memory, so it seems like they really forgot and the files just pile up over days. The only thing that works is disabling CPAI...

[TimG]

Have you informed Support ? That needs looking at. I still don't appear to be having any problems though.

[avatar42]

I should note this is only happening on 1 of my 5 servers despite them all being pretty close to the same setups other than cameras. It continues to happen even if I revert all the way back to 5.4.9.18. This makes me suspicious that it is related to some Windows or .Net update.

For example, attached screenshots are of server running ~12 hours.

Screenshot 2024-09-25 103301.png (34.1 KiB) Viewed 3101 times

Screenshot 2024-09-25 103301.png (34.1 KiB) Viewed 3101 times