ASAN error in fread.c

[MichaelChirico]

Here are the logs of the error:

https://gitlab.com/Rdatatable/data.table/-/jobs/10696837289

I think we should fix the bug (presumably introduced by #7138) before continuing to dig a hole with further edits to the file.

Originally posted by @MichaelChirico in #7181 (comment)

[aitap]

The failing test is 1869.6, for files whose size is a multiple of the memory page size in the operating system: https://github.com/Rdatatable/data.table/blob/32e5aea1681398335f1bcedb4fe1563906701634/inst/tests/tests.Rraw#L12582-L12587 1869.7 may fail as well.

[aitap]

The out-of-bounds read can be turned into a real crash if a file that fits in a large but exact number of pages is left mapped into memory, not copied like in test 1869.6, and detect_types() tries parse_double_hexadecimal(), which reads memory past the last page containing the file contents: # edited below: make sure the crash happens on Windows too f <- file("foo.txt", "wb") writeLines( c("a234567", rep("0x1.56p", 4096*1000/8-3), "0x1.5p", "0x1.5p", "0"), f ) close(f) data.table::fread("foo.txt", verbose = TRUE) <...> [02] Opening the file Opening file foo.txt File opened, size = 3.906MiB (4096000 bytes). Memory mapped ok [03] Detect and skip BOM [04] Arrange mmap to be \0 terminated \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal. <...> [07] Detect column types, dec, good nrow estimate and whether first row is column names Number of sampling jump points = 100 because (4095999 bytes from row 1 to eof) / (2 * 792 jump0size) == 2585 dec='.' detected based on a balance of 0 parsed fields Type codes (jump 000) : B Quote rule 0 *** caught segfault *** address 0x7f46627e8000, cause 'memory not mapped' Traceback: 1: data.table::fread("foo.txt", verbose = TRUE) An irrecoverable exception occurred. R is aborting now ... This is one of the places where the code implicitly relies on the invariant that *eof is always addressable and equal to '\0' <#7037 (comment)>. If you read the bytes one by one, you hit that zero and stop. If you skip ahead without a bounds check, you can fall off the end of the buffer. @badasahog, would you like to work on fixing this? This is relatively important, both because a known crash, even an unlikely one, is pretty bad in itself, and because ASAN errors prevent us from releasing a new version, even when we know they are de-facto harmless. Such code would either fail CRAN pre-tests or result in us getting a CRAN deadline when discovered.

[badasahog]

This is quite unfortunate; I think rolling back may be the best option.

My concern is that it's related to the const_cast, and something rocked the compiler into tripping on undefined behavior.

Maybe it's best to just not rock the boat until this is settled, I may have a better solution in mind to avoid the issue entirely.

If anyone else has a better theory as to what broke, I'm all ears.

[aitap]

For a Windows reproducer that doesn't need ASAN and an explanation of the mechanism, see the comment above: #7183 (comment)

[MichaelChirico]

If anyone else has a better theory as to what broke, I'm all ears.

My best guess (I could be wrong) is that the claim that x[2] will be optimized away by the compiler does not hold under all settings, compilers. If x[2] is read sometimes where it wouldn't have previously (because the tests on x[0] / x[1] would have failed after encountering the \0 end-of-string), that would explain the crash.

[badasahog]

Damnit, just confirmed this. You warned me and I didn't listen. GG.