data.table spark/databases interface

[ysgit]

data.table is awesome but most people don't have 100GB memory in order to handle really large data sets in memory.

Big progress has been made making the Apache Spark framework available through R in the last couple of years. Two such projects are Apache's sparkr and Rstudio's sparklyr. Both of these provide a dplyr style interface to spark's data processing engine.

As a heavy data.table user it would be amazing if there were to be a data.table interface for spark. That would make it incredibly easy for data scientists to migrate their projects from the smaller CSV style data sets to the huge data sets that can be processed by spark.

A classic data pipeline for me is

1. Bring the data into R by CSV
2. Do some pre-processing (filters, joins, aggregation, feature extraction) of the data using data.table
3. Build a model using one of R's many machine learning packages

I want to be able to migrate this to

1. Connect to data on hadoop cluster
2. Do some pre-processing (filters, joins, aggregation, feature extraction) of the data using data.table's spark interface
3. Build a model using one of spark's many machine learning algorithm.

[mattdowle]

Thanks for the encouragement. Fully agree.
Yes there are lots of efforts in this space. Note that I (and now Jan Gorecki) work at H2O.
I gave a recent presentation here: https://www.youtube.com/watch?v=5X7h1rZGVs0
The slides are here : https://github.com/Rdatatable/data.table/wiki/Presentations
Just to check whether you had seen these before? Before discussing further.

[ysgit]

Hi @mattdowle thanks for pointing those out. Very impressive results!
So would you say that H2O already enables that pipeline?

[mattdowle]

A lot depends on details of your data and types of feature engineering. Definitely test H2O's ML as well as MLlib. We don't have data.table syntax for either Spark or H2O yet. So short answer is no because of that. But yes we'd like to add it.

[LeonardAukea]

Fully agree, this would be great. Hope that data.table will be made available in the near future.

[sskarkhanis]

I really hope DT syntax is available someday. Personally, I really prefer the DT syntax and would like to use it consistently in Spark rather than cringe with dplyr.
Fingers crossed...

[fbala]

something new?

[aadler]

Adding my voice to a desire to be able to use data.table instead of dplyr on distributed data in Spark/H2O

[petalatuy]

Using data.table in Spark would be AMAZING!

[MichaelChirico]

I ran into an issue caused by SparkR's using rJava (and all its endless flaws) today, would love to have been able to just use data.table. directly instead of the magical file i/o gymnastics I found myself resorting to

…

On Dec 1, 2017 9:56 PM, "petalatuy" ***@***.***> wrote: Using data.table in Spark would be AMAZING! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1828 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHQQdXbs22uCw1Zxyg2zWqkUXGWjzuifks5s8AWqgaJpZM4JuUyv> .

[saptarshiguha]

My plus 1000

[sunsiyu]

+1 any updates on the future plan of data.table package?

[MichaelChirico]

I'm curious what ppl want out of this exactly.

Just to be able to use [ on an RDD like you would on a data.table (namely, i/j/by)?

certainly the full functionality is a ways away, but i imagine it wouldn't be too earth shaking to make an idiom for filtering, grouping, even joining by sending syntax within [] to the corresponding operations in sparkR.

in particular, this would just amount to (in essence) aliasing sparkR functions in a syntax friendlier for data.table regulars.

is this what people have in mind?

[jangorecki]

No updates, there is a lot to dev around data.table itself, so external interfacing is not that high priority now. Instead of just spark integration it makes more sense to integrate to dplyr, something like dplyr.table (inverse of dtplyr). Then any dplyr backend will work on data.table syntax.

[harryprince]

indeed, dplyr SQL interface is good enough.

[DavidArenburg]

@jangorecki The problem with that is that will also be slow/full of bugs/ and very unstable as dplyr interface changes on almost daily basis and sometimes they change the whole idiom at once like they did with lazyeval > rlang > tidyeval and G-d knows else what as I long time lost track. Not to mention that hadley, who once stated (with a "tongue in cheeck") that data.table uses "cryptic shortcuts" now masks few of these shortcuts and suddenly don't consider them so cryptic anymore. In short, creating such an API would be a full time job IMO.

I think migrating a few main functionalities from data.table and keep adding them when there is time would be much safer/easier.