improve vague allocation error message

[zachokeeffe]

When one has not set alloc.col to be sufficiently high explicitly and attempts to create more columns than max(100L,ncol(DT)+64L), the following error is printed:

Internal logical error. DT passed to assign has not been allocated enough column slots.

I did not know what this meant, but I found an answer on Stack Overflow. It would be helpful if the package would explicitly direct the user to help(truelength).

Also, for some reason I only encountered this issue with set(), but not :=.

[arunsrinivasan]

@zachokeeffe thanks for filing here.

On your last line (between set() and :=), could you provide a minimal reproducible example (filed as a separate issue)?

[groceryheist]

I encountered such a bug when using ":="
I'm attaching a small dataset where i encounter the problem. The data is edit metadata from wikia wikis.

data.table.memory.bug.zip

library(data.table)
load("data.table.memory.bug.RData")

reverted.edits[talk.page.edits,
          ":="(reverter.initiated.discussion = TRUE,
               first.time.reverter.discussed = min(i.date.time),
               reverter.discussed.revid=min(i.revid)
               ),
          on=.(reverted.by,
               wiki.name,
               talk,date.time<date.time,
               first.time.editor.discussed>=date.time),
          by=.EACHI]

adding alloc.col(reverted.edits,3) fixes the problem on one machine. Interestingly, I found the bug in another environment with an older R version. In this environment there is a segfault instead of an error message, and the alloc.col fix does not work.
`

[jangorecki]

I am not able to reproduce the problem, maybe you forget about setDT after loading data from RData file?

load("reproduces.segfault.RData")
ls()
#[1] "reverted.edits"  "talk.page.edits"
tables()
#              NAME   NROW NCOL MB
#1:  reverted.edits  5,393   39  2
#2: talk.page.edits 38,223   35 13
#                                                COLS KEY
#1: revid,anon,articleid,date.time,deleted,editor,...    
#2: revid,anon,articleid,date.time,deleted,editor,...    
#Total: 15MB
setDT(reverted.edits)
setDT(talk.page.edits)
reverted.edits[talk.page.edits,
          ":="(reverter.initiated.discussion = TRUE,
               first.time.reverter.discussed = min(i.date.time),
               reverter.discussed.revid=min(i.revid)
               ),
          on=.(reverted.by,
               wiki.name,
               talk,date.time<date.time,
               first.time.editor.discussed>=date.time),
          by=.EACHI]
reverted.edits[1:6,1:6]
#   revid  anon articleid           date.time deleted       editor
#1:  3869  TRUE      1111 2005-05-20 14:21:31   FALSE 62.234.15.61
#2:  6340  TRUE      2075 2005-07-28 23:59:04   FALSE  66.20.28.21
#3:  6379 FALSE       938 2005-07-31 08:21:07   FALSE       WhatIf
#4:  6390 FALSE       938 2005-07-31 08:21:31   FALSE       WhatIf
#5:  6391 FALSE       938 2005-08-02 08:42:06   FALSE       WhatIf
#6:  6632  TRUE      2075 2005-08-31 02:09:23   FALSE  66.20.28.21

No minimal example to put to unit tests unfortunately so closing without test. Please let us know if problem still persists for you.