Introduce e2e testing with testcontainers

[gopidesupavan]

Introducing End to End testing to test against real environment with dags.

We can write all the possible scenario dags and test on real environment.
For now i have added a simple dag that triggers dag and validates xcom value.
Extend this to more complex scenario to validate.

My plan is it have a schedule run frequently similar to canary to validate all the tests thats under e2e.
edit: Default it runs in canary after prod image built.

We can utilise this one to test on RC releases likely provide input to image version and run all the tests. So once release manager cuts RC and released then he can trigger this test suite. IMHO very useful to test.

• Add breeze support
• Add documentation

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[gopidesupavan]

Should we have e2e testing separate repo and keep all the tests there and uses them part of workflow? 🤔

[potiuk]

Should we have e2e testing separate repo and keep all the tests there and uses them part of workflow? 🤔

I think separate repo is generally a bad idea. Monorepo has it's drawbacks - but it has also benefits - for example much easier syncing of dependencie with workspace, keeping single contribuition workflow, being able to stop PR from merging when the tests fail etc. I can't see any real benefit of having separate repo for tests that should essentially be run as part of the same workflow as all other tests (especially when we have uv sync and workspaces.

[gopidesupavan]

Should we have e2e testing separate repo and keep all the tests there and uses them part of workflow? 🤔

I think separate repo is generally a bad idea. Monorepo has it's drawbacks - but it has also benefits - for example much easier syncing of dependencie with workspace, keeping single contribuition workflow, being able to stop PR from merging when the tests fail etc. I can't see any real benefit of having separate repo for tests that should essentially be run as part of the same workflow as all other tests (especially when we have uv sync and workspaces.

agree make sense :) do you think we should run these e2e tests part PR's, i am thinking of we can do it in schedule, i remember @kaxil was mentioning there were around 200+ test scenario combinations for xcoms itself so i think it would fit in schedule run? WDYT?

[potiuk]

for xcoms itself so i think it would fit in schedule run? WDYT?

a) we can improve selective tests to trigger a relevant subset of those
b) we can run full set in canary runs

[gopidesupavan]

for xcoms itself so i think it would fit in schedule run? WDYT?

a) we can improve selective tests to trigger a relevant subset of those b) we can run full set in canary runs

yes that also works good idea ;)

[gopidesupavan]

getting this one ready today, we should be able to catch more errors running actual dag scenarios, like the recent secrets issue.

[gopidesupavan]

@vatsrahul1001 am adding e2e test suite to CI and this can be triggered part of regular CI or after rc release it helps detecting any issues by running e2e tests.

Could you please help me what test suite you regularly run and the all the example dags you have setup for testing? i will add it to test suite.

@kaxil please add your thoughts on this if you have anything in mind to cover specifically combination of scenarios. i see you mentioned here around 254 combinations it would be good to know those :) #53130 (comment)

[kaxil]

@vatsrahul1001 am adding e2e test suite to CI and this can be triggered part of regular CI or after rc release it helps detecting any issues by running e2e tests.

Could you please help me what test suite you regularly run and the all the example dags you have setup for testing? i will add it to test suite.

@kaxil please add your thoughts on this if you have anything in mind to cover specifically combination of scenarios. i see you mentioned here around 254 combinations it would be good to know those :) #53130 (comment)

Hey @gopidesupavan 👋

It is

airflow/task-sdk/tests/task_sdk/execution_time/test_task_runner.py

Lines 1391 to 1426 in 00b3b14

	@pytest.mark.parametrize(
	"map_indexes",
	[
	pytest.param(-1, id="not_mapped_index"),
	pytest.param(1, id="single_map_index"),
	pytest.param([0, 1], id="multiple_map_indexes"),
	pytest.param((0, 1), id="any_iterable_multi_indexes"),
	pytest.param(None, id="index_none"),
	pytest.param(NOTSET, id="index_not_set"),
	],
	)
	@pytest.mark.parametrize(
	"task_ids",
	[
	pytest.param("push_task", id="single_task"),
	pytest.param(["push_task1", "push_task2"], id="tid_multiple_tasks"),
	pytest.param({"push_task1", "push_task2"}, id="tid_any_iterable"),
	pytest.param(None, id="tid_none"),
	pytest.param(NOTSET, id="tid_not_set"),
	],
	)
	@pytest.mark.parametrize(
	"xcom_values",
	[
	pytest.param("hello", id="string_value"),
	pytest.param("'hello'", id="quoted_string_value"),
	pytest.param({"key": "value"}, id="json_value"),
	pytest.param([], id="empty_list_no_xcoms_found"),
	pytest.param((1, 2, 3), id="tuple_int_value"),
	pytest.param([1, 2, 3], id="list_int_value"),
	pytest.param(42, id="int_value"),
	pytest.param(True, id="boolean_value"),
	pytest.param(pd.DataFrame({"col1": [1, 2], "col2": [3, 4]}), id="dataframe_value"),
	],
	)
	def test_xcom_pull(

root@6cb799c71804:/opt/airflow# pytest task-sdk/tests/task_sdk/execution_time/test_task_runner.py::TestRuntimeTaskInstance::test_xcom_pull --collect-only

...
...
==== 270 tests collected in 2.69s ======

[gopidesupavan]

@vatsrahul1001 am adding e2e test suite to CI and this can be triggered part of regular CI or after rc release it helps detecting any issues by running e2e tests.
Could you please help me what test suite you regularly run and the all the example dags you have setup for testing? i will add it to test suite.
@kaxil please add your thoughts on this if you have anything in mind to cover specifically combination of scenarios. i see you mentioned here around 254 combinations it would be good to know those :) #53130 (comment)

Hey @gopidesupavan 👋

It is

airflow/task-sdk/tests/task_sdk/execution_time/test_task_runner.py

Lines 1391 to 1426 in 00b3b14

	@pytest.mark.parametrize(
	"map_indexes",
	[
	pytest.param(-1, id="not_mapped_index"),
	pytest.param(1, id="single_map_index"),
	pytest.param([0, 1], id="multiple_map_indexes"),
	pytest.param((0, 1), id="any_iterable_multi_indexes"),
	pytest.param(None, id="index_none"),
	pytest.param(NOTSET, id="index_not_set"),
	],
	)
	@pytest.mark.parametrize(
	"task_ids",
	[
	pytest.param("push_task", id="single_task"),
	pytest.param(["push_task1", "push_task2"], id="tid_multiple_tasks"),
	pytest.param({"push_task1", "push_task2"}, id="tid_any_iterable"),
	pytest.param(None, id="tid_none"),
	pytest.param(NOTSET, id="tid_not_set"),
	],
	)
	@pytest.mark.parametrize(
	"xcom_values",
	[
	pytest.param("hello", id="string_value"),
	pytest.param("'hello'", id="quoted_string_value"),
	pytest.param({"key": "value"}, id="json_value"),
	pytest.param([], id="empty_list_no_xcoms_found"),
	pytest.param((1, 2, 3), id="tuple_int_value"),
	pytest.param([1, 2, 3], id="list_int_value"),
	pytest.param(42, id="int_value"),
	pytest.param(True, id="boolean_value"),
	pytest.param(pd.DataFrame({"col1": [1, 2], "col2": [3, 4]}), id="dataframe_value"),
	],
	)
	def test_xcom_pull(

root@6cb799c71804:/opt/airflow# pytest task-sdk/tests/task_sdk/execution_time/test_task_runner.py::TestRuntimeTaskInstance::test_xcom_pull --collect-only

...
...
==== 270 tests collected in 2.69s ======

cool thanks will make these combinations to execute on real dag scenario.

[vatsrahul1001]

@vatsrahul1001 am adding e2e test suite to CI and this can be triggered part of regular CI or after rc release it helps detecting any issues by running e2e tests.

Could you please help me what test suite you regularly run and the all the example dags you have setup for testing? i will add it to test suite.

@kaxil please add your thoughts on this if you have anything in mind to cover specifically combination of scenarios. i see you mentioned here around 254 combinations it would be good to know those :) #53130 (comment)

This is great work and will be very helpful during the RC testing phase! A good starting point could be running all the example DAGs that already exist, as they cover a wide range of features. From there, our team can also look into contributing additional DAGs that reflect some others common use cases.

[potiuk]

This is great work and will be very helpful during the RC testing phase! A good starting point could be running all the example DAGs that already exist, as they cover a wide range of features. From there, our team can also look into contributing additional DAGs that reflect some others common use cases.

Agreed :)

[gopidesupavan]

This is great work and will be very helpful during the RC testing phase! A good starting point could be running all the example DAGs that already exist, as they cover a wide range of features. From there, our team can also look into contributing additional DAGs that reflect some others common use cases.

Agreed :)

Yeah have already plan to include them part of this. working on them currently.

[gopidesupavan]

Oh github actions is not allowing some patterns in file paths.

With the provided path, there will be 352 files uploaded
Artifact name is valid!
Root directory input is valid!
Error: The path for one of the files in artifact is not valid: /dag_id=example_bash_decorator/run_id=manual__2025-08-23T10:27:47.958128+00:00/task_id=also_run_this/attempt=1.log. Contains the following character:  Colon :
          
Invalid characters include:  Double quote ", Colon :, Less than <, Greater than >, Vertical bar |, Asterisk *, Question mark ?, Carriage return \r, Line feed \n
          
The following characters are not allowed in files that are uploaded due to limitations with certain file systems such as NTFS. To maintain file system agnostic behavior, these characters are intentionally not allowed to prevent potential problems with downloads on different file systems.

[gopidesupavan]

Finally ready for review.

[gopidesupavan]

Would like to get this first pass simple and added basic test and test to trigger example dags. will extend this for further xcoms/connections etc;

@vatsrahul1001 can also extend on top of this, feel free to add your suite in this :)

[potiuk]

Fantastic job @gopidesupavan . Just some nit comments.

[potiuk]

Not sure if we intend to backport it, but It might be possible actually, so I left the label,.

[gopidesupavan]

Not sure if we intend to backport it, but It might be possible actually, so I left the label,.

🤔 yeah not sure, these test runs in canary anyway it hints if anything breaks.

[gopidesupavan]

Finally green :) merging it now.

[github-actions]

Backport failed to create: v3-0-test. View the failure log Run details

Status	Branch	Result
❌	v3-0-test

You can attempt to backport this manually by running:

cherry_picker 0954496 v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue