Excessive file copies in IBR-DTN?
Hello, IBR-DTN developers and users!
I suspect that IBR-DRN does excessive work on copying files while doing dtnrecv.
Here is what I do. 1. On the node2 I start dtnrecv --file /var/tmp/1.iso --name file 2. On the node1 I start dtnsend dtn://node-2.dtn/file /media/1.iso
Then in another console on node2 I look, what does IBR-DTN do. 1. It stores the data, received, from node1 to a blob file in /var/spool/dtnd/blobs/ 2. Than it copies that blob file to a bundle file in /var/spool/dtnd/bundles/ 3. Than it copies the bundle file to the destination file /var/tmp/1.iso
Steps 1 and 2 look ok to me. But why use bit-by-bit copy on step 3 instead of just move (mv) the bundle file to the destination file without the expensive data transfer? This excessive data copying consumes memory to store the whole bundle and takes more time.
My node2 has 2Gb RAM. The file that I send (1.iso) is 1.2Gb. RAM gets exhausted on the step 3, and dtnrecv fails with error message "Aborted".
So I have two proposals. 1. Implement better error reporting, e.g. "Not enough memory to copy file." instead of "Aborted." 2. Move bundle file to the destination file instead of copying it.
------------------------------------------------------------------------------------------------------------------------------- My IBR-DTN config: local_uri = dtn://node-2.dtn logfile = /var/log/ibrdtn.log timezone = +4 limit_blocksize = 0 user = dtnd blob_path = /var/spool/dtnd/blobs storage_path = /var/spool/dtnd/bundles limit_storage = 5G discovery_announce = 0 net_interfaces = eth0 net_rebind = yes net_autoconnect = 60 net_eth0_type = tcp net_eth0_interface = eth0 net_eth0_port = 4556 net_eth0_discovery = yes routing_forwarding = yes static1_address = 192.168.150.38 static1_port = 4556 static1_uri = dtn://node-1.dtn static1_proto = tcp static1_immediately = yes dht_enabled = yes dht_bootstrapping = yes
Best Regards, Sergey Syreskin
An addition to my previous post.
I have added 1Gb more RAM to node2 and retried my test. Dtnrecv still terminates with the "Aborted." message. It seems that a memory leak takes place.
Is there any way I can debug this case?
Thu, 04 Oct 2012 14:06:27 +0400 от Sergey Sireskin sire@mail.ru:
Hello, IBR-DTN developers and users!
I suspect that IBR-DRN does excessive work on copying files while doing dtnrecv. ....
Best Regards, Sergey Syreskin
According to top dtnrecv eats 35% of memory right before it terminates.
Here are the last lines of strace output for dtnrecv: ---------------------------------------------------------------------------------------------------------- recvfrom(8, "dtn!\3\1\0\n\20dtn://node-2.dtn", 5120, 0, NULL, NULL) = 25 mmap(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f0ec28fd000 mprotect(0x7f0ec28fd000, 4096, PROT_NONE) = 0 clone(child_stack=0x7f0ec2afcf10, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f0ec2afd9d0, tls=0x7f0ec2afd700, child_tidptr=0x7f0ec2afd9d0) = 3456 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0ec2b10000 write(1, "Wait for incoming bundle... \n", 29Wait for incoming bundle... ) = 29 open("/var/tmp/1.iso", O_RDWR|O_CREAT|O_TRUNC, 0666) = 9 futex(0x7fffadbc8174, FUTEX_WAIT_PRIVATE, 1, NULL ) = 0 futex(0x7fffadbc8140, FUTEX_WAKE_PRIVATE, 1) = 0 close(9) = 0 write(4, "i", 1) = 1 write(6, "i", 1) = 1 close(8) = 0 close(3) = 0 close(4) = 0 close(5) = 0 close(6) = 0 write(2, "Aborted.", 8Aborted.) = 8 write(2, "\n", 1 ) = 1 exit_group(1) = ? ----------------------------------------------------------------------------------------------------------
Is there any debug information or test results, that I can supply?
Thu, 04 Oct 2012 17:41:30 +0400 от Sergey Sireskin sire@mail.ru:
An addition to my previous post.
I have added 1Gb more RAM to node2 and retried my test. Dtnrecv still terminates with the "Aborted." message. It seems that a memory leak takes place.
Is there any way I can debug this case?
Thu, 04 Oct 2012 14:06:27 +0400 от Sergey Sireskin sire@mail.ru:
Hello, IBR-DTN developers and users!
I suspect that IBR-DRN does excessive work on copying files while doing dtnrecv. ....
Best Regards, Sergey Syreskin
--
!! This message is brought to you via the `ibr-dtn' mailing list.
!! Please do not reply to this message to unsubscribe. To unsubscribe or adjust
!! your settings, send a mail message to ibr-dtn-request@ibr.cs.tu-bs.de
!! or look at https://www.ibr.cs.tu-bs.de/mailman/listinfo/ibr-dtn.
Hello Sergey,
to understand your issue completely I need some more details.
Is /var/tmp a harddrive or a virtual drive based on RAM?
How much space is available in these paths? - /var/tmp - /var/spool/dtnd
You are right regarding the copy behavior in IBR-DTN. There are several reasons to process the bundles that way. But first you have to understand the classical client-server model [1] which is also valid for DTN daemon and DTN clients (such as dtnrecv).
[1] http://en.wikipedia.org/wiki/Client%E2%80%93server_model
Each client is connected to the daemon using a TCP channel. There is no assumption of shared disk space. That allows to split-up daemon and client on different machines if necessary and (much more important) a clean software design without permission issues during deployment on various platforms. Further that means, it is impossible to just move the bundle in the last delivery step to another location because you cannot just move a file through a TCP connection. Even if that would be possible, it would break the possibility to deliver multiple copies of the same bundle to different clients.
The copying inside of the daemon between blob and bundles path is also required, because the daemon needs copies of the bundles to work with. Think of workspace if there are bundles in the "blob"-path and bundles in the "bundles"-path are in some kind of long-term storage. Each time the daemon should deliver a bundle to a client or receive some data a bundle is copied to the volatile BLOB storage. If the store process put a bundle into the storage and no process owns a pointer to the bundle the BLOB file will disappear.
However, the SQLite version of the storage already has an improved mechanism which works with hardlinks instead copies which might be much faster in your setup.
Kind regards, Johannes Morgenroth
Am 04.10.2012 12:06, schrieb Sergey Sireskin:
Hello, IBR-DTN developers and users!
I suspect that IBR-DRN does excessive work on copying files while doing dtnrecv.
Here is what I do.
- On the node2 I start dtnrecv --file /var/tmp/1.iso --name file
- On the node1 I start dtnsend dtn://node-2.dtn/file /media/1.iso
Then in another console on node2 I look, what does IBR-DTN do.
- It stores the data, received, from node1 to a blob file in
/var/spool/dtnd/blobs/ 2. Than it copies that blob file to a bundle file in /var/spool/dtnd/bundles/ 3. Than it copies the bundle file to the destination file /var/tmp/1.iso
Steps 1 and 2 look ok to me. But why use bit-by-bit copy on step 3 instead of just move (mv) the bundle file to the destination file without the expensive data transfer? This excessive data copying consumes memory to store the whole bundle and takes more time.
My node2 has 2Gb RAM. The file that I send (1.iso) is 1.2Gb. RAM gets exhausted on the step 3, and dtnrecv fails with error message "Aborted".
So I have two proposals.
- Implement better error reporting, e.g. "Not enough memory to copy
file." instead of "Aborted." 2. Move bundle file to the destination file instead of copying it.
My IBR-DTN config: local_uri = dtn://node-2.dtn logfile = /var/log/ibrdtn.log timezone = +4 limit_blocksize = 0 user = dtnd blob_path = /var/spool/dtnd/blobs storage_path = /var/spool/dtnd/bundles limit_storage = 5G discovery_announce = 0 net_interfaces = eth0 net_rebind = yes net_autoconnect = 60 net_eth0_type = tcp net_eth0_interface = eth0 net_eth0_port = 4556 net_eth0_discovery = yes routing_forwarding = yes static1_address = 192.168.150.38 static1_port = 4556 static1_uri = dtn://node-1.dtn static1_proto = tcp static1_immediately = yes dht_enabled = yes dht_bootstrapping = yes
Best Regards, Sergey Syreskin
-- !! This message is brought to you via the `ibr-dtn' mailing list. !! Please do not reply to this message to unsubscribe. To unsubscribe or adjust !! your settings, send a mail message to ibr-dtn-request@ibr.cs.tu-bs.de !! or look at https://www.ibr.cs.tu-bs.de/mailman/listinfo/ibr-dtn.
Hello Johannes,
Thanks a lot for your detailed explanation.
I didn't know that client and daemon can work on different machines. How do client applications discover server address in case if DTN daemon is on another machine? Among dtnsend and dtnrecv command line options I see only "-U" for UNIX domain sockets, which as far as I know work only on localhost.
Both /var/tmp and /var/spool/dtnd are on the same disk partition which has 6.4Gb free.
Best Regards, Sergey Syreskin
Mon, 08 Oct 2012 09:36:41 +0200 от Johannes Morgenroth morgenro@ibr.cs.tu-bs.de:
Hello Sergey,
to understand your issue completely I need some more details.
Is /var/tmp a harddrive or a virtual drive based on RAM?
How much space is available in these paths?
- /var/tmp
- /var/spool/dtnd
You are right regarding the copy behavior in IBR-DTN. There are several
reasons to process the bundles that way. But first you have to
understand the classical client-server model [1] which is also valid for
DTN daemon and DTN clients (such as dtnrecv).
[1] http://en.wikipedia.org/wiki/Client%E2%80%93server_model
Each client is connected to the daemon using a TCP channel. There is no
assumption of shared disk space. That allows to split-up daemon and
client on different machines if necessary and (much more important) a
clean software design without permission issues during deployment on
various platforms. Further that means, it is impossible to just move the
bundle in the last delivery step to another location because you cannot
just move a file through a TCP connection. Even if that would be
possible, it would break the possibility to deliver multiple copies of
the same bundle to different clients.
The copying inside of the daemon between blob and bundles path is also
required, because the daemon needs copies of the bundles to work with.
Think of workspace if there are bundles in the "blob"-path and bundles
in the "bundles"-path are in some kind of long-term storage. Each time
the daemon should deliver a bundle to a client or receive some data a
bundle is copied to the volatile BLOB storage. If the store process put
a bundle into the storage and no process owns a pointer to the bundle
the BLOB file will disappear.
However, the SQLite version of the storage already has an improved
mechanism which works with hardlinks instead copies which might be much
faster in your setup.
Kind regards,
Johannes Morgenroth
Am 04.10.2012 12:06, schrieb Sergey Sireskin:
Hello, IBR-DTN developers and users!
I suspect that IBR-DRN does excessive work on copying files while doing
dtnrecv.
Here is what I do.
On the node2 I start dtnrecv --file /var/tmp/1.iso --name file
On the node1 I start dtnsend dtn://node-2.dtn/file /media/1.iso
Then in another console on node2 I look, what does IBR-DTN do.
- It stores the data, received, from node1 to a blob file in
/var/spool/dtnd/blobs/
- Than it copies that blob file to a bundle file in
/var/spool/dtnd/bundles/
- Than it copies the bundle file to the destination file /var/tmp/1.iso
Steps 1 and 2 look ok to me. But why use bit-by-bit copy on step 3
instead of just move (mv)
the bundle file to the destination file without the expensive data transfer?
This excessive data copying consumes memory to store the whole bundle
and takes more time.
My node2 has 2Gb RAM. The file that I send (1.iso) is 1.2Gb. RAM gets
exhausted on the step 3,
and dtnrecv fails with error message "Aborted".
So I have two proposals.
- Implement better error reporting, e.g. "Not enough memory to copy
file." instead of "Aborted."
- Move bundle file to the destination file instead of copying it.
My IBR-DTN config:
local_uri = dtn://node-2.dtn
logfile = /var/log/ibrdtn.log
timezone = +4
limit_blocksize = 0
user = dtnd
blob_path = /var/spool/dtnd/blobs
storage_path = /var/spool/dtnd/bundles
limit_storage = 5G
discovery_announce = 0
net_interfaces = eth0
net_rebind = yes
net_autoconnect = 60
net_eth0_type = tcp
net_eth0_interface = eth0
net_eth0_port = 4556
net_eth0_discovery = yes
routing_forwarding = yes
static1_address = 192.168.150.38
static1_port = 4556
static1_uri = dtn://node-1.dtn
static1_proto = tcp
static1_immediately = yes
dht_enabled = yes
dht_bootstrapping = yes
Best Regards,
Sergey Syreskin
--
!! This message is brought to you via the `ibr-dtn' mailing list.
!! Please do not reply to this message to unsubscribe. To unsubscribe or adjust
!! your settings, send a mail message to ibr-dtn-request@ibr.cs.tu-bs.de
!! or look at https://www.ibr.cs.tu-bs.de/mailman/listinfo/ibr-dtn.
Hello Johannes,
I have conducted a series of tests with different file sizes. The tests show that transfer of of files larger than 1Gb (1073741824 bytes) fail. Files of 1Gb and smaller size are transmitted successfully.
Best regards, Sergey Syreskin
Mon, 08 Oct 2012 09:36:41 +0200 от Johannes Morgenroth morgenro@ibr.cs.tu-bs.de:
Hello Sergey,
to understand your issue completely I need some more details.
Is /var/tmp a harddrive or a virtual drive based on RAM?
How much space is available in these paths?
- /var/tmp
- /var/spool/dtnd
You are right regarding the copy behavior in IBR-DTN. There are several
reasons to process the bundles that way. But first you have to
understand the classical client-server model [1] which is also valid for
DTN daemon and DTN clients (such as dtnrecv).
[1] http://en.wikipedia.org/wiki/Client%E2%80%93server_model
Each client is connected to the daemon using a TCP channel. There is no
assumption of shared disk space. That allows to split-up daemon and
client on different machines if necessary and (much more important) a
clean software design without permission issues during deployment on
various platforms. Further that means, it is impossible to just move the
bundle in the last delivery step to another location because you cannot
just move a file through a TCP connection. Even if that would be
possible, it would break the possibility to deliver multiple copies of
the same bundle to different clients.
The copying inside of the daemon between blob and bundles path is also
required, because the daemon needs copies of the bundles to work with.
Think of workspace if there are bundles in the "blob"-path and bundles
in the "bundles"-path are in some kind of long-term storage. Each time
the daemon should deliver a bundle to a client or receive some data a
bundle is copied to the volatile BLOB storage. If the store process put
a bundle into the storage and no process owns a pointer to the bundle
the BLOB file will disappear.
However, the SQLite version of the storage already has an improved
mechanism which works with hardlinks instead copies which might be much
faster in your setup.
Kind regards,
Johannes Morgenroth
Am 04.10.2012 12:06, schrieb Sergey Sireskin:
Hello, IBR-DTN developers and users!
I suspect that IBR-DRN does excessive work on copying files while doing
dtnrecv.
Here is what I do.
On the node2 I start dtnrecv --file /var/tmp/1.iso --name file
On the node1 I start dtnsend dtn://node-2.dtn/file /media/1.iso
Then in another console on node2 I look, what does IBR-DTN do.
- It stores the data, received, from node1 to a blob file in
/var/spool/dtnd/blobs/
- Than it copies that blob file to a bundle file in
/var/spool/dtnd/bundles/
- Than it copies the bundle file to the destination file /var/tmp/1.iso
Steps 1 and 2 look ok to me. But why use bit-by-bit copy on step 3
instead of just move (mv)
the bundle file to the destination file without the expensive data transfer?
This excessive data copying consumes memory to store the whole bundle
and takes more time.
My node2 has 2Gb RAM. The file that I send (1.iso) is 1.2Gb. RAM gets
exhausted on the step 3,
and dtnrecv fails with error message "Aborted".
So I have two proposals.
- Implement better error reporting, e.g. "Not enough memory to copy
file." instead of "Aborted."
- Move bundle file to the destination file instead of copying it.
My IBR-DTN config:
local_uri = dtn://node-2.dtn
logfile = /var/log/ibrdtn.log
timezone = +4
limit_blocksize = 0
user = dtnd
blob_path = /var/spool/dtnd/blobs
storage_path = /var/spool/dtnd/bundles
limit_storage = 5G
discovery_announce = 0
net_interfaces = eth0
net_rebind = yes
net_autoconnect = 60
net_eth0_type = tcp
net_eth0_interface = eth0
net_eth0_port = 4556
net_eth0_discovery = yes
routing_forwarding = yes
static1_address = 192.168.150.38
static1_port = 4556
static1_uri = dtn://node-1.dtn
static1_proto = tcp
static1_immediately = yes
dht_enabled = yes
dht_bootstrapping = yes
Best Regards,
Sergey Syreskin
--
!! This message is brought to you via the `ibr-dtn' mailing list.
!! Please do not reply to this message to unsubscribe. To unsubscribe or adjust
!! your settings, send a mail message to ibr-dtn-request@ibr.cs.tu-bs.de
!! or look at https://www.ibr.cs.tu-bs.de/mailman/listinfo/ibr-dtn.
Hello again,
indeed here exists an issue with dtnrecv. This program receives all the payload first and put it into memory. Then the content gets written to the file. That means you can not receive bundles larger than your memory when using dtnrecv. This isn't an issue of the API, it is a limitation of this tool and might be improved in future releases.
Kind regards, Johannes
Am 08.10.2012 15:58, schrieb Sergey Sireskin:
Hello Johannes,
I have conducted a series of tests with different file sizes. The tests show that transfer of of files larger than 1Gb (1073741824 bytes) fail. Files of 1Gb and smaller size are transmitted successfully.
Best regards, Sergey Syreskin
participants (2)
-
Johannes Morgenroth -
Sergey Sireskin