Came across this when writing an answer to this SO post. Most definitely a bug.
require(data.table) ## 1.9.5
X = data.table(x=c(1,1,1,2,2), y=1:5, key="x")
Y = data.table(x=2:5, y=letters[1:4], key="x")
X[Y]
# x y i.y
#1: 2 4 a
#2: 2 5 a
#3: 3 NA b
#4: 4 NA c
#5: 5 NA d
X[Y, .N]
# [1] 2
X[Y, .N, nomatch=0L]
# [1] 2
The first result should be 5, since the count is happening after performing the join. For example, sum(y) works as intended.
X[Y, sum(y)] # [1] NA - correct result
X[Y, sum(y, na.rm=TRUE)] # correct result