instance of 'std::overflow_error' - robin_hood::map overflow

[BiotechPedro]

Hi!

I've been playing with the powerful splitcode for weeks but I have recently struggled with an apparent problem of the hashing map for a FASTQ file of ~300M reads. To check I was doing everything correctly, I run the same command and config file in a different FASTQ file of ~25M - it worked fine.

The full log with the common error on the big FASTQ file is the following:

* Using a list of 3073 tags (vector size: 3073; map size: 58,345; num elements in map: 115,177)
* will process sample 1: my_input.fastq
65M reads processed (79.1% assigned)terminate called after throwing an instance of 'std::overflow_error'
  what():  robin_hood::map overflow
Aborted (core dumped)

My config file includes a barcode, a linker and another barcode with an allowed hamming distance of 1, 2 and 1.

My command is the following: splitcode -c config.txt --nFastqs=1 --assign --outb=02_splitcode/final_barcodes.fastq --mapping=02_splitcode/mapping.txt -o 02_splitcode/output_R1.fastq -t 64 my_input.fastq.

I also tried to reduce the Hamming distance in order to make the map size lower, but got the same error. It normally happens in the moment when 25-50% of the reads are processed, independently of parallelizing with -t or not. Finally, I cloned the repo and changed the line 1210 of the file splitcode/src/ProcessReads.cpp in order to write bufadd += l[i] + nl[i] + (comments && seq[i]->comment.l != 0 ? 1 : 0); // includes name, qual, and the extra space instead of bufadd += l[i] + nl[i]; // includes name and qual. It was suggested by an LLM, but I got the same error.

In case it helps, the RAM capacity in the computer I'm working on is of 252G, although splitcode seems to use it minimally.

Thanks in advance for your time!!!

Pedro

[Yenaled]

Hello, what version of splitcode are you using and do you remember how you installed it? I'd recommend either directly downloading the binaries (via the GitHub releases page) or using conda.

The root of the issue is that, if you're compiling from source, in older versions of C++, we resort to using an older hash table data structure (robin_hood) whereas in newer versions (>=2017), we'll use an updated one (unordered_dense) which is able to handle a larger load without crashing.

The load is not due to the allowable hamming distance; it's because of the hash map created by using --assign; you're probably finding too many barcode combinations so it's overloading the map.

[BiotechPedro]

Thanks for the info!
I installed it last week with conda, so the last version.
Which would be the solution? Increasing —bclen or another thing?

[Yenaled]

Hmm, strange that it doesn’t work on conda.

Ok try this binary then:

https://github.com/pachterlab/splitcode/releases/download/v0.31.2/splitcode_linux-v0.31.2.tar.gz

[BiotechPedro]

It happens the same error if running the downloaded binary :/

~/software/binary/splitcode/splitcode -c config.txt --nFastqs=1 --assign --outb=02_splitcode/DNA/final_barcodes.fastq --mapping=02_splitcode/DNA/mapping.txt -o 02_splitcode/DNA/output_R1.fastq -t 64 my_input.fastq.gz 
* Using a list of 3073 tags (vector size: 3073; map size: 58,345; num elements in map: 115,177)
* will process sample 1: 1256_2025_S1_R1_001.fastq.gz
64M reads processed (79.2% assigned)terminate called after throwing an instance of 'std::overflow_error'
  what():  robin_hood::map overflow
Aborted (core dumped)

Both the versions of the binary and the conda installation is for version 0.31.2. Moreover, in case it help my gcc --version is gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 and my system is Ubuntu 20.04.6 LTS with architecture x86_64. I also tried in another computer (Rocky Linux 8.7; x86_64) with gcc --version being gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16) and splitcode version bein 0.31.2 installed with conda - but got the exact same error.

To my understanding, the hash map should not be that big since the whitelist of barcodes A is ~1500 and the one of barcode B is ~1750. I write down the config file in case it helps with debugging this:

tag	id	group	distance	location	minFindsG	next	exclude
barcodeA.txt$	bc1	BC1	1	0:0:8	1	-	0
ACGTACGCGTGTAT	linker	linker	2	0:8:25	1	{{BC2}}	1
barcodeB.txt$	bc2	BC2	1	0:22:32	1	-	0

[Yenaled]

I've figured out the issue. You can use the following fix:

Go to the file CMakeLists.txt (note: There are multiple CMakeLists.txt files -- look at the one at the root of the codebase).

And replace the following:

if(${CMAKE_VERSION} VERSION_LESS 3.1)
    add_compile_options(-std=c++11)
else()
    set(CMAKE_CXX_STANDARD 11)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)
    set(CMAKE_CXX_EXTENSIONS OFF)
endif()

With:

if(${CMAKE_VERSION} VERSION_LESS 3.1)
    add_compile_options(-std=c++17)
else()
    set(CMAKE_CXX_STANDARD 17)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)
    set(CMAKE_CXX_EXTENSIONS OFF)
endif()

And then compile from source.

Essentially, my cmake file is forcing an older version of C++ (C++11) to be used even when a newer compiler version is available. I'll fix this in the next release and ping you again here when I do. Thanks for your help in identifying the problem :)

[BiotechPedro]

Thank YOU very much!

It run perfectly for the big FASTQ now.

Just a little note in case this is useful to you or anyone else. When I did cmake .. with version 4.0.3, the following error was printed.

CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.
  
  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.
  
  Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.

Solution was to downgrade my cmake version. I installed splitcode from source changing the above mentioned lines in the rooted CMakeLists.txt and using cmake versions 3.27.0-rc4 and 3.28.0 in two different computers.

Please, let me know in this thread when you fix this in the next release so the conda installation doesn't lead to the issue.

Again, many thanks,
Pedro

[Yenaled]

Made a new official release just now that should fix this (apologies for the delay; got busy).