fix: escape underscores when simplifying `starts_with` by willemv · Pull Request #19077 · apache/datafusion

willemv · 2025-12-03T20:26:16Z

Which issue does this PR close?

Closes starts_with function are simplified incorrectly when the prefix contains an underscore #19076.

What changes are included in this PR?

The simplify implementation for StartsWithFunc is adjusted to also escape underscores instead of only percent signs.

Are these changes tested?

Yes, new sql logic tests were added testing starts_with functions with string literals and patterns that contain an underscore.

Similar tests were added for ends_with, even though that function has no simplification and is not affected by this bug. If simplification is ever introduced there, escaping the underscore cannot be overlooked then.

Are there any user-facing changes?

No

willemv · 2025-12-03T21:04:19Z

@alamb / @andygrove :
Hi, I'm a new contributor. I'm pinging you to trigger the CI tasks, as documented in your contributor guide.

pepijnve · 2025-12-03T21:17:08Z

I've reviewed the changes. Looks correct to me.
FYI, this was introduced by #14119

willemv · 2025-12-04T06:29:43Z

Fixed the formatting, and rebased on the latest main

bert-beyondloops · 2025-12-04T09:57:53Z

Should you replace ‘\’ as well ?

‘%’, ‘_’ and ‘\’ are the 3 special characters within a like expression

See 'predicate.rs'

fn contains_like_pattern(pattern: &str) -> bool {
    memchr3(b'%', b'_', b'\\', pattern.as_bytes()).is_some()
}

willemv · 2025-12-04T11:31:20Z

Should you replace ‘\’ as well ?

‘%’, ‘_’ and ‘\’ are the 3 special characters within a like expression

See 'predicate.rs'
fn contains_like_pattern(pattern: &str) -> bool {
    memchr3(b'%', b'_', b'\\', pattern.as_bytes()).is_some()
}

Agh, but of course ...

willemv · 2025-12-04T11:35:44Z

Backslashes are now also escaped.

I also adjusted the SLT's: they were not failing before the fix, because the simplification for starts_with is not invoked when simply select starts_with('literal', 'anotherliteral'). That's probably because constant expressions are evaluated before the simplification.

alamb

This makes sense to me -- thank you @willemv

I think this was initially introduced here (fyi @jatin510 )

#14119

Thank you @pepijnve for the assist

alamb · 2025-12-04T12:03:04Z

datafusion/sqllogictest/test_files/string/string_view.slt

 ----
 logical_plan
-01)Projection: test.column1_utf8 LIKE Utf8("foo\%%") AS c1, test.column1_large_utf8 LIKE LargeUtf8("foo\%%") AS c2, test.column1_utf8view LIKE Utf8View("foo\%%") AS c3, test.column1_utf8 LIKE Utf8("f_o%") AS c4, test.column1_large_utf8 LIKE LargeUtf8("f_o%") AS c5, test.column1_utf8view LIKE Utf8View("f_o%") AS c6
+01)Projection: test.column1_utf8 LIKE Utf8("foo\%%") AS c1, test.column1_large_utf8 LIKE LargeUtf8("foo\%%") AS c2, test.column1_utf8view LIKE Utf8View("foo\%%") AS c3, test.column1_utf8 LIKE Utf8("f\_o%") AS c4, test.column1_large_utf8 LIKE LargeUtf8("f\_o%") AS c5, test.column1_utf8view LIKE Utf8View("f\_o%") AS c6


🤦 -- _ matches exactly one character -- so escaping them is the right thing to do

bert-beyondloops · 2025-12-04T13:52:59Z

datafusion/functions/src/string/starts_with.rs

-            //   1. 'ja%'         (input pattern)
-            //   2. 'ja\%'        (escape special char '%')
-            //   3. 'ja\%%'       (add suffix for starts_with)
+            // Example: starts_with(col, 'j_a%') -> col LIKE 'j\_a\%%'


Example: starts_with(col, 'j\_a%') -> col LIKE 'j\\\_a\%%' ?

Indeed, I adjusted the steps below it but not this line ...

Fixed now

alamb · 2025-12-05T08:45:37Z

fmt test failure is unrelated. Merging up to get fix from main

alamb · 2025-12-05T21:39:54Z

Thanks again @willemv @pepijnve and @bert-beyondloops

## Which issue does this PR close? - Closes apache#19076. ## What changes are included in this PR? The `simplify` implementation for `StartsWithFunc` is adjusted to also escape underscores instead of only percent signs. ## Are these changes tested? Yes, new sql logic tests were added testing `starts_with` functions with string literals and patterns that contain an underscore. Similar tests were added for `ends_with`, even though that function has no simplification and is not affected by this bug. If simplification is ever introduced there, escaping the underscore cannot be overlooked then. ## Are there any user-facing changes? No Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> (cherry picked from commit 340a28c)

Co-authored-by: zjregee <zjregee@gmail.com>

## Which issue does this PR close? - Closes apache#19076. ## What changes are included in this PR? The `simplify` implementation for `StartsWithFunc` is adjusted to also escape underscores instead of only percent signs. ## Are these changes tested? Yes, new sql logic tests were added testing `starts_with` functions with string literals and patterns that contain an underscore. Similar tests were added for `ends_with`, even though that function has no simplification and is not affected by this bug. If simplification is ever introduced there, escaping the underscore cannot be overlooked then. ## Are there any user-facing changes? No Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> (cherry picked from commit 340a28c)

…#392) v52 ## Which issue does this PR close? - Closes apache#19076. ## What changes are included in this PR? The `simplify` implementation for `StartsWithFunc` is adjusted to also escape underscores instead of only percent signs. ## Are these changes tested? Yes, new sql logic tests were added testing `starts_with` functions with string literals and patterns that contain an underscore. Similar tests were added for `ends_with`, even though that function has no simplification and is not affected by this bug. If simplification is ever introduced there, escaping the underscore cannot be overlooked then. ## Are there any user-facing changes? No Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> (cherry picked from commit 340a28c) Co-authored-by: Willem Verstraeten <willem.verstraeten@gmail.com>

…#392) v52 --- [Cherry-pick summary: v46→v47] Source commit: b4e6d5f (fix: escape underscores when simplifying starts_with (apache#19077) (#392) v52) Strategy: cherry-picked cleanly Upstream PR: apache#19077 (not in v47) Test coverage: adequate (adds slt tests for starts_with and ends_with with underscore patterns) Tests: cargo nextest run -p datafusion-functions passed

github-actions bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Dec 3, 2025

willemv force-pushed the fix/19076-starts-with-simplification branch from bb36713 to 352bbff Compare December 3, 2025 20:30

willemv force-pushed the fix/19076-starts-with-simplification branch from 352bbff to d41adfe Compare December 3, 2025 21:11

willemv force-pushed the fix/19076-starts-with-simplification branch 2 times, most recently from 30507bd to c55f53d Compare December 4, 2025 06:28

willemv force-pushed the fix/19076-starts-with-simplification branch 2 times, most recently from 42f63c6 to f59f039 Compare December 4, 2025 11:33

alamb approved these changes Dec 4, 2025

View reviewed changes

bert-beyondloops reviewed Dec 4, 2025

View reviewed changes

fix: escape underscores when simplifying starts_with (apache#19076)

dd01c71

willemv force-pushed the fix/19076-starts-with-simplification branch from f59f039 to dd01c71 Compare December 5, 2025 07:16

Merge branch 'main' into fix/19076-starts-with-simplification

5de894f

alamb added this pull request to the merge queue Dec 5, 2025

Merged via the queue into apache:main with commit 340a28c Dec 5, 2025
27 checks passed

avantgardnerio pushed a commit to coralogix/arrow-datafusion that referenced this pull request Jan 9, 2026

fix starts_with (apache#14812, apache#19077)

8469666

avantgardnerio added a commit to coralogix/arrow-datafusion that referenced this pull request Jan 9, 2026

fix starts_with (apache#14812, apache#19077) (#391)

886e7ca

Co-authored-by: zjregee <zjregee@gmail.com>

dispanser mentioned this pull request Feb 26, 2026

Datafusion upgrade to DF 47 coralogix/arrow-datafusion#397

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: escape underscores when simplifying `starts_with`#19077

fix: escape underscores when simplifying `starts_with`#19077
alamb merged 2 commits intoapache:mainfrom
willemv:fix/19076-starts-with-simplification

willemv commented Dec 3, 2025

Uh oh!

willemv commented Dec 3, 2025

Uh oh!

pepijnve commented Dec 3, 2025 •

edited

Loading

Uh oh!

willemv commented Dec 4, 2025

Uh oh!

bert-beyondloops commented Dec 4, 2025

Uh oh!

willemv commented Dec 4, 2025

Uh oh!

willemv commented Dec 4, 2025 •

edited

Loading

Uh oh!

alamb left a comment

Uh oh!

alamb Dec 4, 2025

Uh oh!

bert-beyondloops Dec 4, 2025

Uh oh!

willemv Dec 5, 2025

Uh oh!

alamb commented Dec 5, 2025

Uh oh!

alamb commented Dec 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

willemv commented Dec 3, 2025

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

willemv commented Dec 3, 2025

Uh oh!

pepijnve commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

willemv commented Dec 4, 2025

Uh oh!

bert-beyondloops commented Dec 4, 2025

Uh oh!

willemv commented Dec 4, 2025

Uh oh!

willemv commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

bert-beyondloops Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

willemv Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Dec 5, 2025

Uh oh!

alamb commented Dec 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pepijnve commented Dec 3, 2025 •

edited

Loading

willemv commented Dec 4, 2025 •

edited

Loading