[cbindlist/mergelist] Add some C utilities: NROW/NCOL/isDataTable/isDataList/perhapsDataTable by MichaelChirico · Pull Request #6433 · Rdatatable/data.table

MichaelChirico · 2024-08-29T17:25:19Z

Towards #4370. None of this is exposed in the public API yet, so not tested here.

MichaelChirico · 2024-08-29T17:25:33Z

[cbindlist/mergelist] Add some C utilities: NROW/NCOL/isDataTable/isDataList/perhapsDataTable #6433 : 2 dependent PRs (#6434 , #6435 ) 👈 (View in Graphite)
[cbindlist/mergelist] Add ERR to the mult enum in bmerge.c #6432
[cbindlist/mergelist] Whitespace changes #6431
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2024-08-29T17:43:08Z

No obvious timing issues in HEAD=cbind-merge-list-utils

Generated via commit 9825d83

Download link for the artifact containing the test results: ↓ atime-results.zip

Task	Duration
R setup and installing dependencies	4 minutes and 58 seconds
Installing different package versions	9 minutes and 29 seconds
Running and plotting the test cases	2 minutes and 32 seconds

src/utils.c

@jan-glx

add cbind by reference, timing R prototype of mergelist wording use lower overhead funs stick to int32 for now, correct R_alloc bmerge C refactor for codecov and one loop for speed address revealed codecov gaps refactor vecseq for codecov seqexp helper, some alloccol export on C bmerge codecov, types handled in R bmerge already better comment seqexp bmerge mult=error #655 multiple new C utils swap if branches explain new C utils comments mostly reduce conflicts to PR #4386 comment C code address multiple matches during update-on-join #3747 Revert "address multiple matches during update-on-join #3747" This reverts commit b64c0c3. merge.dt has temporarily mult arg, for testing minor changes to cbindlist c dev mergelist, for single pair now add quiet option to cc() mergelist tests add check for names to perhaps.dt rm mult from merge.dt method rework, clean, polish multer, fix righ and full joins make full join symmetric mergepair inner function to loop on extra check for symmetric mergelist manual ensure no df-dt passed where list expected comments and manual handle 0 cols tables more tests more tests and debugging move more logic closer to bmerge, simplify mergepair more tests revert not used changes reduce not needed checks, cleanup copy arg behavior, manual, no tests yet cbindlist manual, export both cleanup processing bmerge to dtmatch test function match order for easier preview vecseq gets short-circuit batch test allow browser big cleanup remmove unneeded stuff, reduce diff more cleanup, minor manual fixes add proper test scripts Merge branch 'master' into cbind-merge-list comment out not used code for coverage more tests, some nocopy opts rename sql test script, should fix codecov simplify dtmatch inner branch more precise copy, now copy only T or F unused arg not yet in api, wording comments and refer issues codecov hasindex coverage codecov gap tests for join using key, cols argument fix missing import forderv more tests, improve missing on handling more tests for order of inner and full join for long keys new allow.cartesian option, #4383, #914 reduce diff, improve codecov reduce diff, comments need more DT, not lists, mergelist 3+ tbls proper escape heavy check unit tests more tests, address overalloc failure mergelist and cbindlist retain index manual, examples fix manual minor clarify in manual retain keys, right outer join for snowflake schema joins duplicates in cbindlist recycling in cbindlist escape 0 input in copyCols empty input handling closing cbindlist vectorized _on_ and _join.many_ arg rename dtmatch to dtmerge vectorized args: how, mult push down input validation add support for cross join, semi join, anti join full join, reduce overhead for mult=error mult default value dynamic fix manual add "see details" to Rd mention shared on in arg description amend feedback from Michael semi and anti joins will not reorder x columns Merge branch 'master' into cbind-merge-list spelling, thx to @jan-glx check all new funs used and add comments bugfix, sort=T needed for now Merge branch 'master' into cbind-merge-list Update NEWS.md Merge branch 'master' into cbind-merge-list Merge branch 'master' into cbind-merge-list NEWS placement numbering ascArg->order Merge remote-tracking branch 'origin/cbind-merge-list' into cbind-merge-list attempt to restore from master Update to stopf() error style Need isFrame for now More quality checks: any(!x)->!all(x); use vapply_1{b,c,i} really restore from master try to PROTECT() before duplicate() update error message in test appease the rchk gods extraneous space missing ';' use catf simplify perhapsDataTableR move sqlite.Rraw.manual into other.Rraw simplify for loop Merge remote-tracking branch 'origin/cbind-merge-list' into cbind-merge-list

codecov · 2024-09-30T06:14:54Z

Codecov Report

Attention: Patch coverage is 0% with 28 lines in your changes missing coverage. Please review.

Project coverage is 98.50%. Comparing base (67856ae) to head (9825d83).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/utils.c	0.00%	28 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6433      +/-   ##
==========================================
- Coverage   98.68%   98.50%   -0.19%     
==========================================
  Files          79       79              
  Lines       14726    14754      +28     
==========================================
  Hits        14533    14533              
- Misses        193      221      +28

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jan-glx

add cbind by reference, timing R prototype of mergelist wording use lower overhead funs stick to int32 for now, correct R_alloc bmerge C refactor for codecov and one loop for speed address revealed codecov gaps refactor vecseq for codecov seqexp helper, some alloccol export on C bmerge codecov, types handled in R bmerge already better comment seqexp bmerge mult=error #655 multiple new C utils swap if branches explain new C utils comments mostly reduce conflicts to PR #4386 comment C code address multiple matches during update-on-join #3747 Revert "address multiple matches during update-on-join #3747" This reverts commit b64c0c3. merge.dt has temporarily mult arg, for testing minor changes to cbindlist c dev mergelist, for single pair now add quiet option to cc() mergelist tests add check for names to perhaps.dt rm mult from merge.dt method rework, clean, polish multer, fix righ and full joins make full join symmetric mergepair inner function to loop on extra check for symmetric mergelist manual ensure no df-dt passed where list expected comments and manual handle 0 cols tables more tests more tests and debugging move more logic closer to bmerge, simplify mergepair more tests revert not used changes reduce not needed checks, cleanup copy arg behavior, manual, no tests yet cbindlist manual, export both cleanup processing bmerge to dtmatch test function match order for easier preview vecseq gets short-circuit batch test allow browser big cleanup remmove unneeded stuff, reduce diff more cleanup, minor manual fixes add proper test scripts Merge branch 'master' into cbind-merge-list comment out not used code for coverage more tests, some nocopy opts rename sql test script, should fix codecov simplify dtmatch inner branch more precise copy, now copy only T or F unused arg not yet in api, wording comments and refer issues codecov hasindex coverage codecov gap tests for join using key, cols argument fix missing import forderv more tests, improve missing on handling more tests for order of inner and full join for long keys new allow.cartesian option, #4383, #914 reduce diff, improve codecov reduce diff, comments need more DT, not lists, mergelist 3+ tbls proper escape heavy check unit tests more tests, address overalloc failure mergelist and cbindlist retain index manual, examples fix manual minor clarify in manual retain keys, right outer join for snowflake schema joins duplicates in cbindlist recycling in cbindlist escape 0 input in copyCols empty input handling closing cbindlist vectorized _on_ and _join.many_ arg rename dtmatch to dtmerge vectorized args: how, mult push down input validation add support for cross join, semi join, anti join full join, reduce overhead for mult=error mult default value dynamic fix manual add "see details" to Rd mention shared on in arg description amend feedback from Michael semi and anti joins will not reorder x columns Merge branch 'master' into cbind-merge-list spelling, thx to @jan-glx check all new funs used and add comments bugfix, sort=T needed for now Merge branch 'master' into cbind-merge-list Update NEWS.md Merge branch 'master' into cbind-merge-list Merge branch 'master' into cbind-merge-list NEWS placement numbering ascArg->order Merge remote-tracking branch 'origin/cbind-merge-list' into cbind-merge-list attempt to restore from master Update to stopf() error style Need isFrame for now More quality checks: any(!x)->!all(x); use vapply_1{b,c,i} really restore from master try to PROTECT() before duplicate() update error message in test appease the rchk gods extraneous space missing ';' use catf simplify perhapsDataTableR move sqlite.Rraw.manual into other.Rraw simplify for loop Merge remote-tracking branch 'origin/cbind-merge-list' into cbind-merge-list

MichaelChirico · 2024-10-01T16:59:17Z

OK, this PR LGTM.

@jan-glx

add cbind by reference, timing R prototype of mergelist wording use lower overhead funs stick to int32 for now, correct R_alloc bmerge C refactor for codecov and one loop for speed address revealed codecov gaps refactor vecseq for codecov seqexp helper, some alloccol export on C bmerge codecov, types handled in R bmerge already better comment seqexp bmerge mult=error #655 multiple new C utils swap if branches explain new C utils comments mostly reduce conflicts to PR #4386 comment C code address multiple matches during update-on-join #3747 Revert "address multiple matches during update-on-join #3747" This reverts commit b64c0c3. merge.dt has temporarily mult arg, for testing minor changes to cbindlist c dev mergelist, for single pair now add quiet option to cc() mergelist tests add check for names to perhaps.dt rm mult from merge.dt method rework, clean, polish multer, fix righ and full joins make full join symmetric mergepair inner function to loop on extra check for symmetric mergelist manual ensure no df-dt passed where list expected comments and manual handle 0 cols tables more tests more tests and debugging move more logic closer to bmerge, simplify mergepair more tests revert not used changes reduce not needed checks, cleanup copy arg behavior, manual, no tests yet cbindlist manual, export both cleanup processing bmerge to dtmatch test function match order for easier preview vecseq gets short-circuit batch test allow browser big cleanup remmove unneeded stuff, reduce diff more cleanup, minor manual fixes add proper test scripts Merge branch 'master' into cbind-merge-list comment out not used code for coverage more tests, some nocopy opts rename sql test script, should fix codecov simplify dtmatch inner branch more precise copy, now copy only T or F unused arg not yet in api, wording comments and refer issues codecov hasindex coverage codecov gap tests for join using key, cols argument fix missing import forderv more tests, improve missing on handling more tests for order of inner and full join for long keys new allow.cartesian option, #4383, #914 reduce diff, improve codecov reduce diff, comments need more DT, not lists, mergelist 3+ tbls proper escape heavy check unit tests more tests, address overalloc failure mergelist and cbindlist retain index manual, examples fix manual minor clarify in manual retain keys, right outer join for snowflake schema joins duplicates in cbindlist recycling in cbindlist escape 0 input in copyCols empty input handling closing cbindlist vectorized _on_ and _join.many_ arg rename dtmatch to dtmerge vectorized args: how, mult push down input validation add support for cross join, semi join, anti join full join, reduce overhead for mult=error mult default value dynamic fix manual add "see details" to Rd mention shared on in arg description amend feedback from Michael semi and anti joins will not reorder x columns Merge branch 'master' into cbind-merge-list spelling, thx to @jan-glx check all new funs used and add comments bugfix, sort=T needed for now Merge branch 'master' into cbind-merge-list Update NEWS.md Merge branch 'master' into cbind-merge-list Merge branch 'master' into cbind-merge-list NEWS placement numbering ascArg->order Merge remote-tracking branch 'origin/cbind-merge-list' into cbind-merge-list attempt to restore from master Update to stopf() error style Need isFrame for now More quality checks: any(!x)->!all(x); use vapply_1{b,c,i} really restore from master try to PROTECT() before duplicate() update error message in test appease the rchk gods extraneous space missing ';' use catf simplify perhapsDataTableR move sqlite.Rraw.manual into other.Rraw simplify for loop Merge remote-tracking branch 'origin/cbind-merge-list' into cbind-merge-list

ben-schwen

LGTM

MichaelChirico mentioned this pull request Aug 29, 2024

[cbindlist/mergelist] Whitespace changes #6431

Merged

MichaelChirico force-pushed the cbind-merge-list-mult-err branch from 8a519b4 to 275c386 Compare August 29, 2024 17:31

MichaelChirico force-pushed the cbind-merge-list-utils branch from 63141a6 to 627e77d Compare August 29, 2024 17:31

MichaelChirico changed the title ~~cbindlist~~ [cbindlist/mergelist] Add some C utilities: NROW/NCOL/isDataTable/isDataList/perhapsDataTable Aug 29, 2024

MichaelChirico added the graphite-ready label Aug 29, 2024

MichaelChirico commented Aug 29, 2024

View reviewed changes

src/utils.c Outdated Show resolved Hide resolved

MichaelChirico commented Aug 29, 2024

View reviewed changes

src/utils.c Outdated Show resolved Hide resolved

MichaelChirico commented Aug 29, 2024

View reviewed changes

src/utils.c Outdated Show resolved Hide resolved

MichaelChirico commented Aug 29, 2024

View reviewed changes

src/utils.c Outdated Show resolved Hide resolved

MichaelChirico force-pushed the cbind-merge-list-mult-err branch from 275c386 to 82b940a Compare August 30, 2024 17:28

MichaelChirico force-pushed the cbind-merge-list-utils branch from bcc1d77 to 5e53c78 Compare August 30, 2024 17:28

MichaelChirico force-pushed the cbind-merge-list-mult-err branch from 82b940a to cb6358d Compare August 31, 2024 17:34

MichaelChirico force-pushed the cbind-merge-list-utils branch from 5e53c78 to 88b1e3c Compare August 31, 2024 17:34

MichaelChirico force-pushed the cbind-merge-list-mult-err branch from cb6358d to 1b06b9c Compare September 30, 2024 05:24

MichaelChirico force-pushed the cbind-merge-list-utils branch from 88b1e3c to 90a04e3 Compare September 30, 2024 05:25

MichaelChirico force-pushed the cbind-merge-list-mult-err branch from 1b06b9c to 80ec084 Compare September 30, 2024 05:50

MichaelChirico force-pushed the cbind-merge-list-utils branch from 90a04e3 to 4478411 Compare September 30, 2024 05:50

MichaelChirico force-pushed the cbind-merge-list-mult-err branch from 80ec084 to 39fc30e Compare September 30, 2024 06:07

MichaelChirico force-pushed the cbind-merge-list-utils branch from 4478411 to f9d245f Compare September 30, 2024 06:07

MichaelChirico force-pushed the cbind-merge-list-mult-err branch from 39fc30e to 0ab0f5e Compare October 1, 2024 16:39

MichaelChirico force-pushed the cbind-merge-list-utils branch from f9d245f to 8bba6af Compare October 1, 2024 16:39

restore ws change

7342791

MichaelChirico marked this pull request as ready for review October 1, 2024 16:41

MichaelChirico requested a review from HughParsonage as a code owner October 1, 2024 16:41

MichaelChirico force-pushed the cbind-merge-list-utils branch 2 times, most recently from 0c42ca9 to e6971cf Compare October 1, 2024 16:58

MichaelChirico force-pushed the cbind-merge-list-utils branch from e6971cf to 3ef33d7 Compare October 1, 2024 17:02

ben-schwen approved these changes Nov 17, 2024

View reviewed changes

MichaelChirico added this to the 1.18.0 milestone Jan 29, 2025

Base automatically changed from cbind-merge-list-mult-err to master June 30, 2025 18:43

Merge branch 'master' into cbind-merge-list-utils

9825d83

MichaelChirico merged commit 1b3998e into master Jun 30, 2025
11 of 13 checks passed

MichaelChirico deleted the cbind-merge-list-utils branch June 30, 2025 18:53

aitap mentioned this pull request Jul 19, 2025

mergelist: isFrame -> isDataFrame #7194

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cbindlist/mergelist] Add some C utilities: NROW/NCOL/isDataTable/isDataList/perhapsDataTable#6433

[cbindlist/mergelist] Add some C utilities: NROW/NCOL/isDataTable/isDataList/perhapsDataTable#6433
MichaelChirico merged 5 commits intomasterfrom
cbind-merge-list-utils

MichaelChirico commented Aug 29, 2024 •

edited

Loading

Uh oh!

MichaelChirico commented Aug 29, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Aug 29, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Sep 30, 2024 •

edited

Loading

Uh oh!

MichaelChirico commented Oct 1, 2024

Uh oh!

ben-schwen left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MichaelChirico commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelChirico commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Sep 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MichaelChirico commented Oct 1, 2024

Uh oh!

ben-schwen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MichaelChirico commented Aug 29, 2024 •

edited

Loading

MichaelChirico commented Aug 29, 2024 •

edited

Loading

github-actions bot commented Aug 29, 2024 •

edited

Loading

codecov bot commented Sep 30, 2024 •

edited

Loading