Spring changes summer regression - Part 1.

[tyuldashev]

Possible values:

• Project: Maven, Gradle, Intellij
• JDK: 1.8, 11, 17
• Framework: JUnit 4, JUnit 5, TestNG
• Parametrization: On, Off

Suggestion:

Let's do independent validation of "JDK <-> Framework" and "JDK <-> Framework <-> Parametrization". Project type will equally distributed between testers.

JDK <-> Framework Pair:

All possible combinations of JDK and Framework are selected.
Project type is iterated independently.
Parametrization is OFF.

Scenario:

1. Crete new project with specified project type, JDK.
2. Remove default test dependency from Maven or Gradle configuration and refresh project structure.
3. Copy paste org.utbot.examples.exceptions.ExceptionExamples and org.utbot.examples.controlflow.Conditions from UTBot project into source code folder.
4. Invoke test generation for org.utbot.examples.exceptions.ExceptionExamples file with specific Framework - all test frameworks suggested to be installed.
5. Generate tests - tests are generated without error with proper test Framework.
6. Run tests - tests run without error.
7. Invoke test generation again for org.utbot.examples.controlflow.Conditions, all except installed framework suggested to be installed.
8. Generate and run tests with the same framework - tests are generated and executed without errors.

Project	JDK	Framework	Status	Comment
Gradle	1.8	JUnit 5	✅
Maven	1.8	TestNG	✅	7.5 added
Gradle	11	JUnit 5	✅
Maven	11	TestNG	✅	7.6 added
Maven	17	JUnit 5	✅
Gradle	17	TestNG	✅	7.6 added

JDK <-> Framework <-> Parametrization

All possible combinations of Parametrization and Framework need to be tested. But actually all cases with Parametrization OFF were tested in previous section, so we select only case Parametrization is ON.
Additionally Parametrization is possible only with JUnit5 and TestNG frameworks, so we leave only with 6 test cases:

JDK	Framework	Parametrization	Status	Comment
1.8	JUnit 4	On	➖	Not applicable
1.8	JUnit 5	On	✅
1.8	TestNG	On	➖	Not applicable
11	JUnit 4	On	➖	Not applicable
11	JUnit 5	On	✅
11	TestNG	On	✅
17	JUnit 4	On	➖	Not applicable
17	JUnit 5	On	✅
17	TestNG	On	✅

JDK	Fuzzing/Symbolic	Symbolic 100%	Fuzzing 100%
1.8	casts	casts	exceptions
11	mock	mock	casts
17	mock	exceptions	exceptions

Test Generation Timeout	Fuzzing/Symbolic	Symbolic 100%	Fuzzing 100%
60	strings	structures	lambda
10	x	x	algorithms/CorrectBracketSequences
1	x	x	x

Tests used during verification:

• casts
• exceptions
• mock
• strings
• structures
• lambda
• objects
• stdlib
• algorithms/CorrectBracketSequences
• arrays/ArrayOfObjects
• arrays/FinalStaticFieldArray
• enums/ClassWithEnum
• exceptions/JvmCrashExamples
• collections/Optionals
• collections/QueueUsages
• controlflow/CycleDependedCondition - to check summaries
• controlflow/Switch
• types/Generics
• nested
• invokes
• make.symbolic/ClassWithComplicatedMethods
• mixed/LoggerExample - to check force mocking
• natives
• reflection
• stream
• thirdparty/numbers/ArithmeticUtils
• threads
• unsafe
• wrappers
• annotations/NotNullAnnotation

[tyuldashev]

Still actual:

• No assert in test for method which returns Exception object #1870
• @utbot.invokes {@link soot.dummy.InvokeDynamic in JavaDoc for test method #1869
• No assert in test for method which returns Exception object #1870
• Hanging tests duplicates are not minimized #2130

Opened defects:

• Identical or contradictory@utbot.executesCondition for assert in user code #2414
• Concrete execution error on usage of stream .toArray #2413
• UTBot cannot generate test for simple function when user sets small timeout #2410
• Parametrized test with contradictory asserts: assertNotNull and assertNull #2390
• Mock static methods enables on every change of Test framework or Mocking strategy #2400
• TestNG parametrized test generation broken #2384

[alisevych]

@tyuldashev Could you please add the sample package name for each tested mode?
Also use samples from different packages. Especially first part of this list :

• Manual testing of Release 2023.3 Part 4 Samples list #2012
• casts
• exceptions
• mock

Testing of Symbolic 100% and Fuzzing 100% modes needs to be made on several different samples.
Also please check the logs in your user folder:
~\AppData\Local\Temp\UTBot\rdEngineProcessLogs\utbot-engine-current.log

[tyuldashev]

Currently (and in 2022.12.1 plugin version) even for simple function like return a / b; UTbot unable to generate tests with small timeouts (< ~10s) nor by Symbolic neither by Fuzzer.

[alisevych]

@tyuldashev Please create an issue for this case.
As it is actual for Symbolic and Fuzzer both, it seems to be about timeout handling.