Hi Sebastian,
I have several moving single boards with IBRDTN collection and sending information each minute to a server through a few gateways. The dtnd deamon several times goes above 70% of CPU, which increases the load average (top or uptime) for values much higher than 1. It seems that sending packets increases a lot the CPU usage when compared with dtnd deamon running without any tool. Do you have any hit why this is happening? We cannot continue using the ibrdtn software with this really high values.
Thanks, Tiago
On 19 Jun 2014, at 10:13, Sebastian Schildt schildt@ibr.cs.tu-bs.de wrote:
As far as I know,we have never checked reducing the Thread stack size. However, considering that the RSS size is always significantly smaller than virtual memory, it seems, it could work. It is hard to influence the number of threads, because for every connection a couple of threads (3?) are spun up. So, the more neighbors you are connected to, the more threads you get. Basically the multi-threaded model is in the DNA of IBR-DTN because only these architecture will allow making use of multicore architectures. . However, according to our experience there is not much overhead from those threads. We have seen MUCH more threads than 25 on OpenWRT based MIPS boards without any problems.
Also, if you remove a neighbor, after some while the number of threads should go down again (there should not be any "thread-leakage", if there is, it is a bug)
Here are some more information regarding the maximum number of threads and influence on virtual memory:
http://stackoverflow.com/questions/344203/maximum-number-of-threads-per-proc...
But anyway, if you hit any walls, let us know :)
Sebastian
Am 19.06.2014 um 10:55 schrieb Tiago Condeixa tscondeixa@ua.pt:
Hi,
I install ibrdtn software in single board computers running OpenWrt with MIPS architecture and just 128M of RAM Memory. Because of this, I reduce the thread stack size in Thread.cpp file from 2M to 256K and virtual memory reduces a lot, and at least during the last day it didn’t crash. Do you tested the ibrdtn with smaller value or do you know if I will have a segmentation fault for larger scenarios? What is the operation requiring more memory by dtnd? I defined a path for the bundle storage.
I also check dtnd process and it uses around 25 threads, which is a lot for our embedded system with other programs running. There is any change to reduce the number of threads? and how can I do it?
thanks a lot, Tiago -- !! This message is brought to you via the `ibr-dtn' mailing list. !! Please do not reply to this message to unsubscribe. To unsubscribe or adjust !! your settings, send a mail message to ibr-dtn-request@ibr.cs.tu-bs.de !! or look at https://mail.ibr.cs.tu-bs.de/listinfo/ibr-dtn.
Hello,
this is hard to diagnose… Generally speaking: If the DTN dameon is idling in the background and has no contact to neighbors, CPU usage schould be neglibile. If it connects and has a lot of bundles to transmit you might see it spiking. Can you describe more clearly, for how long CPU usage spikes are, and what happens in the network during that time?
Your configuration regarding storage, and if you use it, security might help shed some light onto this. For example, one known reason for high CPU usgae could also be a corrupted SQLite storage. Another reasons might be a very, very slow storage (then you should see lots of IOWait in top). To check whether this is the reason, you could also start the deamon without any storage option set (it will work in RAM then).
As a short-time fix, it should be no problem to set the niceness of IBR-DTN to 19 (man renice). This will not reduce load, but it will make sure that IBR-DTN does not take so much ressources from other processes away. However, in any case, in the low-traffic situation you lined out, IBR-DTN should not take that much ressources. Short spikes with high/full CPU load on contact can be expected, but they should not be that long that they increase your load average.
Lastly, can you tell, what kind of processor you use? Is it a MIPS4k (like the good old TI AR7 found in old routers) or 24K (like the Atheros AR7xx family) variant? Maybe I can find something similar in our stack of hardware to get a better feeling what to expect :)
MfG
Sebastian
On 23 Jun 2014, at 15:40, Tiago Condeixa tscondeixa@ua.pt wrote:
Hi Sebastian,
I have several moving single boards with IBRDTN collection and sending information each minute to a server through a few gateways. The dtnd deamon several times goes above 70% of CPU, which increases the load average (top or uptime) for values much higher than 1. It seems that sending packets increases a lot the CPU usage when compared with dtnd deamon running without any tool. Do you have any hit why this is happening? We cannot continue using the ibrdtn software with this really high values.
Thanks, Tiago
On 19 Jun 2014, at 10:13, Sebastian Schildt schildt@ibr.cs.tu-bs.de wrote:
As far as I know,we have never checked reducing the Thread stack size. However, considering that the RSS size is always significantly smaller than virtual memory, it seems, it could work. It is hard to influence the number of threads, because for every connection a couple of threads (3?) are spun up. So, the more neighbors you are connected to, the more threads you get. Basically the multi-threaded model is in the DNA of IBR-DTN because only these architecture will allow making use of multicore architectures. . However, according to our experience there is not much overhead from those threads. We have seen MUCH more threads than 25 on OpenWRT based MIPS boards without any problems.
Also, if you remove a neighbor, after some while the number of threads should go down again (there should not be any "thread-leakage", if there is, it is a bug)
Here are some more information regarding the maximum number of threads and influence on virtual memory:
http://stackoverflow.com/questions/344203/maximum-number-of-threads-per-proc...
But anyway, if you hit any walls, let us know :)
Sebastian
Am 19.06.2014 um 10:55 schrieb Tiago Condeixa tscondeixa@ua.pt:
Hi,
I install ibrdtn software in single board computers running OpenWrt with MIPS architecture and just 128M of RAM Memory. Because of this, I reduce the thread stack size in Thread.cpp file from 2M to 256K and virtual memory reduces a lot, and at least during the last day it didn’t crash. Do you tested the ibrdtn with smaller value or do you know if I will have a segmentation fault for larger scenarios? What is the operation requiring more memory by dtnd? I defined a path for the bundle storage.
I also check dtnd process and it uses around 25 threads, which is a lot for our embedded system with other programs running. There is any change to reduce the number of threads? and how can I do it?
thanks a lot, Tiago -- !! This message is brought to you via the `ibr-dtn' mailing list. !! Please do not reply to this message to unsubscribe. To unsubscribe or adjust !! your settings, send a mail message to ibr-dtn-request@ibr.cs.tu-bs.de !! or look at https://mail.ibr.cs.tu-bs.de/listinfo/ibr-dtn.
participants (2)
-
Sebastian Schildt
-
Tiago Condeixa