Hi,
thanks for the insightful information. Indeed, if we are allocating much more memory then we need somewhere, we would consider this a bug and fix it. At the moment I do not know where that might be. Hints are welcome :)
Keep in mind that IBR-DTN is mostly C++, so it is not a straightforward application of "malloc", but instead memory allocations are "hidden" behind "new" or some fancy copy constructors that might get used when moving stuff around in STL collections. So it is not that easy to profile. However, for some malloc strategies fragmentation might lead to an increase of virtual memory size, if the allocator takes the easy way out as long as possible allocating higher addresses you would also see the observed behavior when doing a lot of small allocations, even when there are freed in time. To get some circumstantial evidence whether this is the case here, It would be interesting to see whether the behavior is different when using another compiler/C library or operating system.
Sebastian
Am 12.06.2014 um 16:24 schrieb Romeu Monteiro <romeumonteiro7@gmail.com>:
> Hi,
>
> We did some research on virtual memory and we believe what is happening is that dtnd is allocating much more memory than it is actually using, and this is what leads to such high values of virtual memory. Do you have any idea about which part(s) of the code may be causing such excessive memory allocation? (You can find more details in the e-mail bellow, keep in mind that we are working and talking about a small system - like an embedded system - with 64MB of RAM.)
>
> Thank you,
>
> Romeu
>
> ***************************************************
>
> Hi all,
>
> I was reading about the Linux Virtual Memory system and I got some interesting answers.
>
> The LVM uses the RAM and flash (and swap, which our system don't have any..) as its pages, but the interest thing is how the process allocates memory. From malloc man page:
>
> "By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory
> really is available. In case it turns out that the system is out of memory, one or more processes will be killed by the OOM killer."
>
> So if your process is calling the malloc function and not really using this memory (it will only really allocate the memory at the first read/write operation is requested by the process), you can got over 100% of the memory. This strange strategy can be tested with the following code:
>
>
>
>
> #include <stdio.h>
>
>
>
> #include <stdlib.h>
>
>
>
> #include <string.h>
>
>
>
>
>
>
>
> #define MALLOC_SIZE (1024*1024*10)
>
>
>
>
>
>
>
> int main(void) {
>
>
>
> int i = 0;
>
>
>
> while(1) {
>
>
>
> char *foo = (char*)malloc(MALLOC_SIZE);
>
>
>
> //memset(foo, 0xaa, MALLOC_SIZE);
>
>
>
> printf("Allocation No. %d\n", i++);
>
>
>
> if(foo == NULL) {
>
>
>
> printf("Couldn't alloc\n");
>
>
>
> fflush(stdout);
>
>
>
> return 0;
>
>
>
> }
>
>
>
> }
>
>
>
> return 0;
>
>
>
> }
>
>
>
>
>
>
>
>
>
>
> I compiled it calling its binary mem_test. Following you can check an instant I caught using the top tool, first with the memset line commented (no memory access) and then with the memset line uncommented (memory access).
>
>
>
>
>
>
>
> On the first case you can actually see the VM size increasing (I saw it over 4000%! where top calculates it as a percentage of the RAM size), but the free memory available is not decreasing.
>
>
>
> On the second case the free memory available decreases to zero.
>
> mem_test without accessing the allocated memory:
>
> Mem: 33012K used, 28168K free, 0K shrd, 0K buff, 13824K cached
> CPU: 0% usr 2% sys 0% nic 96% idle 0% io 0% irq 0% sirq
> Load average: 0.21 0.11 0.07 1/50 14343
> PID PPID USER STAT VSZ %VSZ %CPU COMMAND
> 13335 13314 root R 1548 3% 1% top -d 1
> 14041 18904 root S 1361m2276% 0% ./mem_test
>
> mem_test accessing the allocated memory:
>
> Mem: 52272K used, 8908K free, 0K shrd, 0K buff, 3544K cached
> CPU: 2% usr 6% sys 0% nic 90% idle 0% io 0% irq 0% sirq
> Load average: 0.22 0.13 0.08 1/52 14954
> PID PPID USER STAT VSZ %VSZ %CPU COMMAND
> 14937 18904 root S 31500 51% 5% ./mem_test
>
> Of course it's not a critical issue, but it means the dtnd is allocating more memory than what it's using, and since the system kills the process if it goes out of memory, we can live with that. In my opinion, if it will be a Veniam application, you need to understand where it's doing this.
>
>
> On Wed, Jun 11, 2014 at 12:28 AM, Romeu Monteiro <romeumonteiro7@gmail.com> wrote:
> Hi Sebastian,
>
> Thank you very much for this additional information.
> It will definitely help us address the issue. :)
>
> Romeu
>
>
> On Mon, Jun 9, 2014 at 6:03 PM, Sebastian Schildt <schildt@ibr.cs.tu-bs.de> wrote:
> Hello
>
> Am 09.06.2014 um 02:05 schrieb Romeu Monteiro <romeumonteiro7@gmail.com>:
>
> > 2) When you say dtnd was memleak checked, that refers only to real memory, right? Did you make some measurements regarding virtual memory as well? We are currently working with industry partners on a huge testbed, with hundreds of vehicles equipped with small memory devices that can host IBR-DTN. It is an extremely sensitive testbed running multiple services and experiments, and so we are required to be extremely careful with everything that may cause any kind of malfunction of the devices. We were alerted of the very high values of virtual memory by dtnd and asked if we could check and solve that issue. It may not be really an issue, but neither we nor our partners are familiar enough with issues of virtual memory to guarantee that such high values will not pose a risk to the testbed.
>
> According to my understanding, the max virtual memory is the maximum addressable space per process. So as a rule of thumb on a 32 bit architecture you will hit the limit between 3 and 4 GB (depending how much is reserved for kernel address space). Theoretically (I have never tested this), if a process requests more, it will be killed (or the allocation fails, and if it is not programmed carefully the program will die a horrible death shortly after). This should not pose a risk to other processes. So if the virtual memory stays well below the address space limitations of the machine you should have no problem whatsoever (How much IBR-DTN will take over a longer time I do not know, we never consciously checked this). Also the 3-4 GB is sort of a lower border, for example a x86 machine with PAE kernel would be able to use almost 16 GB per process. 64 bit system will handle "enough".
>
>
> There might be per-process limits for virtual memory, check
>
> ulimit -v
>
> if it sys "unlimited", it goes as far as the machine can go. You might set it to a lower (too low) value to check how IBR-DTN behaves, and whether other processes are affected, see
> http://unix.stackexchange.com/questions/28678/how-to-limit-available-virtual-memory-per-process
>
>
>
> Good luck and don't destroy you testbed :)
>
> Sebastian
> --
> !! This message is brought to you via the `ibr-dtn' mailing list.
> !! Please do not reply to this message to unsubscribe. To unsubscribe or adjust
> !! your settings, send a mail message to <ibr-dtn-request@ibr.cs.tu-bs.de>
> !! or look at https://mail.ibr.cs.tu-bs.de/listinfo/ibr-dtn.
>
>