Summary
The main character, the leaky program, is a legacy rust application. We are migrating it to Kubernetes. The migration went smoothly - we had a few hiccups, but nothing major. And then the hell broke loose.
In my previous blog post I wrote about heaptrack - a tool that can trace heap allocations. I’ve used it before and it has been an excellent tool for finding memory leaks in c/rust/C++ applications. I wanted to pinpoint the exact location of the leak. And fix it as soon as possible.
The application was running on a server at 127.0.0%. The data was in a file called /tmp/leaks. It was being used to create a heap profile and check for leaks or excessive memory usage.
Problems with the application are being fixed, but not yet fixed. The problem is with the way the data is calculated and the way it is displayed.
Do not trust your instincts. Don’t jump to conclusions. Don't give up - the problem is there; you must find it.
