[R-Forge #1458] Add fast insert() function

[arunsrinivasan]

Submitted by: Matt Dowle; Assigned to: Nobody; R-Forge link

insert(DT, list(...))
and
insert(DT, ...)

This would grow DT, and memcpy the value in.
Possibly, even respecting keys, and doing a memmove to make space.

Related, tables() could report length and truelength separately. There could be an allocate(DT,n) function which increases truelength but not the length (allowing many quick insert()s if user knows that is about to happen).

[jangorecki]

+1
This would be useful to use data.table as reporting engine. Using Rserve any other language can outsource the job to data.table engine, insert allows to constant access without the need to "reload" data.table to put new rows.

[ssssman]

Fast insertion will be a very nice feature to have. We can use data.table as an in-memory database and build a R server, just like KDB+ and q.

http://code.kx.com/wiki/Cookbook/PerformanceTips
q)load :trade q)row: first trade q)\t do[1000000; insert[trade; row]]
1968

[ssssman]

Appending a list to the nested list is quite fast. If data.table is the list of list, fast insertion is possible?

runBenchmark <- function(n) {
microbenchmark(times = 5,
list_= {
a <- list(NA)
for(i in 1:n) {a <- list(a, list(i))}
}

                     )

}

runBenchmark(1e6)
Unit: seconds
expr min lq mean median uq max neval
list_ 1.026988 1.123105 1.276299 1.166307 1.39488 1.670215 5

[jangorecki]

@ssssman data.table is column oriented, so when you append to list, you append columns, not rows.

[ssssman]

@jangorecki understood. Do you think it is possible to add one-row data.table to another much bigger one quickly at microsecond level? rbind will take about 0.2 millisecond.

[jangorecki]

@ssssman depends what you expect. You still need to pre-allocate "empty" (full of NAs probably) rows. Without pre-allocating memory I don't think it would be possible to insert in-place. Then you need to handle those "empty" rows in your application, and insert values there when needed. It is not perfect as you most likely will need to filter out those rows for most operations, and you still need to pre-allocate them. Also maintaining key is getting tricky (if inserted key columns are always bigger than existing one, then setting sorted attribute manually should be enough).

[MichaelChirico]

@jangorecki This is for inserting rows? Is there a difference to #635? Reprex would help as I don't quite follow

[jangorecki]

It is different but it requires same changes in underlying structure to over-allocate rows. This need to be changed the very lowest level in our C code, and will need to changes in multiple places.

[jangorecki]

in linked issue another api has been proposed. API that works inserting into data.frame

dt[nrow(dt) + 1, ] <- c(5, 5, 3)