Ok, but it still seems like it could be a bug. CPU-wise you should be pretty save, it seems it is the CPU as in the old Ubiqiti RouterStation Pro, and we have several of those running with IBR-DTN.

Below you said, you saw nearly " 3K of bundles per minute“ every minute. Did this change with epidemic, or is the CPU usage just lower? Are those bundles from your application, that get replicated over and over, or are this routing handshakes (they are sent from endpoint <EID>/routing, so they can be spotted in the log easily, i.e. if the dameon is started with -v or) _Normally_ I think static connections should not interfere with Prophet. However, what exaclty do you mean with static connections:

1. An always on connection with another DTN node (i.e. ethernet cable) -> That works fine 2. A static neighbor you declared in the configuration? That should also work. Maybe some useless routing handshake packets are generated if the neighbor s configured but not available, but since they have a short lifetime, that should not matter… 3. A static route in the configuration file? If so, what ist the route exactly?

The weird thing here really is, that you expereince lower CPU load with Epidemic than with Prophet. That is suspicious, because Prophet does not do so much more. The more heavey parts such as creating and exchanging BloomFilters are in Epidemic as well…

Next week our IBR-DTN archmage Johannes will be back on board. Maybe he has some additional ideas.

While bundles might remain in storage until they are expired, this is similar in the worst case for Prophet to (i.e. if there is no path). I do not know, whether that is a problem with your storage, however, even Epidemic makes sure, that a bundle will only be transferred once to one peer, so at least bandwidth-wise it is not so much worse*

Sorry, that most of this is guesswork. I am a bit out of ideas atm. But please post whatever details/findings you can come up with.

Sebastian

* However, retransmits might occur, if a node delivered a bundle directly, deletes it from its storage and afterwards looses his Bloomfilter state, because it is powered down. When it comes up again, it could get the bundles again, if others still have them in storage

On 25 Jun 2014, at 18:45, Tiago Condeixa tscondeixa@ua.pt wrote:

Hi again Sebastian,

I changed the routing protocol from prophet to epidemic and the CPU usage and load average decreased a lot, so It seems there is a problem with prophet or at least when It is mixed with static connections. My main problem right now it that there isn’t ack in epidemic, and the bundles remain in the defined folder until the lifetime defined, even when they are delivered to the destination.

thanks, Tiago

Begin forwarded message:

...

From: Tiago Condeixa tscondeixa@ua.pt Subject: Re: [ibr-dtn] threads Date: 25 Jun 2014 15:03:33 GMT+1 To: Sebastian Schildt schildt@ibr.cs.tu-bs.de

Hi Sebastian,

I’m using the new ibrdtn version 0.12.1 in all boards. You can find the board specifications in (http://routerboard.com/RB411UAHR), with CPU AR7161-BC1A. The bundles are stored in flash with default storage. The configuration file has the following configurations, the ones not shown are commented. This configurations are for the moving boards, while in the gateways I also have another interface with ibrdtn (wired) and a static connection to my server also running dtnd. fragmentation = no storage_path = /root/DTN/ibrdtn/bundles limit_storage = 3M discovery_address = 224.0.0.142 discovery_timeout = 5 discovery_crosslayer = yes net_interfaces = lan0 net_lan0_type = tcp net_lan0_interface = wlan1 net_lan0_port = 4556 routing = prophet routing_forwarding = yes routing_prefer_direct = yes dht_enabled = no

I measured the gateway load average during almost a day, please see the attached pdf file. The three moving boards (now stopped in lab ensuring connectivity) and the gateway generate nearly 3K of bundles per minute, and all this data goes through the gateway until the server. The values seems quite high just for 4 boards with few bundles, so I assumed that in a dynamic scenarios with 50 or 100 boards and more information being collected I will have much more CPU usage and load average.

I already know the neighbors and their IP addresses at the kernel level, so do you if I disable neighbor advertisements and routing and feed it directly from my kernel information I can reduce a lot the CPU usage, load average and memory? Do you think this high values are due to the tcp sessions transferring the bundles or they can also be generated by the control messages/processes? Do you have any more hits on how can I reduce the overall resources consumption of ibrdtn?

I will continue measuring the load average changing some things in ibrdtn configuration.

Thanks, Tiago

<uptime_737.pdf>

...

On 25 Jun 2014, at 09:11, Sebastian Schildt schildt@ibr.cs.tu-bs.de wrote:

...

Hello,

this is hard to diagnose… Generally speaking: If the DTN dameon is idling in the background and has no contact to neighbors, CPU usage schould be neglibile. If it connects and has a lot of bundles to transmit you might see it spiking. Can you describe more clearly, for how long CPU usage spikes are, and what happens in the network during that time?

Your configuration regarding storage, and if you use it, security might help shed some light onto this. For example, one known reason for high CPU usgae could also be a corrupted SQLite storage. Another reasons might be a very, very slow storage (then you should see lots of IOWait in top). To check whether this is the reason, you could also start the deamon without any storage option set (it will work in RAM then).

As a short-time fix, it should be no problem to set the niceness of IBR-DTN to 19 (man renice). This will not reduce load, but it will make sure that IBR-DTN does not take so much ressources from other processes away. However, in any case, in the low-traffic situation you lined out, IBR-DTN should not take that much ressources. Short spikes with high/full CPU load on contact can be expected, but they should not be that long that they increase your load average.

Lastly, can you tell, what kind of processor you use? Is it a MIPS4k (like the good old TI AR7 found in old routers) or 24K (like the Atheros AR7xx family) variant? Maybe I can find something similar in our stack of hardware to get a better feeling what to expect :)

MfG

Sebastian

On 23 Jun 2014, at 15:40, Tiago Condeixa tscondeixa@ua.pt wrote:

...

Hi Sebastian,

I have several moving single boards with IBRDTN collection and sending information each minute to a server through a few gateways. The dtnd deamon several times goes above 70% of CPU, which increases the load average (top or uptime) for values much higher than 1. It seems that sending packets increases a lot the CPU usage when compared with dtnd deamon running without any tool. Do you have any hit why this is happening? We cannot continue using the ibrdtn software with this really high values.

Thanks, Tiago

On 19 Jun 2014, at 10:13, Sebastian Schildt schildt@ibr.cs.tu-bs.de wrote:

...

As far as I know,we have never checked reducing the Thread stack size. However, considering that the RSS size is always significantly smaller than virtual memory, it seems, it could work. It is hard to influence the number of threads, because for every connection a couple of threads (3?) are spun up. So, the more neighbors you are connected to, the more threads you get. Basically the multi-threaded model is in the DNA of IBR-DTN because only these architecture will allow making use of multicore architectures. . However, according to our experience there is not much overhead from those threads. We have seen MUCH more threads than 25 on OpenWRT based MIPS boards without any problems.

Also, if you remove a neighbor, after some while the number of threads should go down again (there should not be any "thread-leakage", if there is, it is a bug)

Here are some more information regarding the maximum number of threads and influence on virtual memory:

http://stackoverflow.com/questions/344203/maximum-number-of-threads-per-proc...

But anyway, if you hit any walls, let us know :)

Sebastian

Am 19.06.2014 um 10:55 schrieb Tiago Condeixa tscondeixa@ua.pt:

...

Hi,

I install ibrdtn software in single board computers running OpenWrt with MIPS architecture and just 128M of RAM Memory. Because of this, I reduce the thread stack size in Thread.cpp file from 2M to 256K and virtual memory reduces a lot, and at least during the last day it didn’t crash. Do you tested the ibrdtn with smaller value or do you know if I will have a segmentation fault for larger scenarios? What is the operation requiring more memory by dtnd? I defined a path for the bundle storage.

I also check dtnd process and it uses around 25 threads, which is a lot for our embedded system with other programs running. There is any change to reduce the number of threads? and how can I do it?

thanks a lot, Tiago -- !! This message is brought to you via the `ibr-dtn' mailing list. !! Please do not reply to this message to unsubscribe. To unsubscribe or adjust !! your settings, send a mail message to ibr-dtn-request@ibr.cs.tu-bs.de !! or look at https://mail.ibr.cs.tu-bs.de/listinfo/ibr-dtn.