Add endpoint to watch dag run until finish by uranusjr · Pull Request #51920 · apache/airflow

uranusjr · 2025-06-19T08:01:40Z

I initially wanted to just enhance the trigger endpoint to optionally stream until the run finishes, but it seems that FastAPI does not like this optionally stream idea. You can do it of course, but would loose a lot of the auto annotation reflection feature. So I opted to have a separate streaming endpoint instead.

This endpoint repeatedly emits a JSON object at the specified interval, until the dag reaches a finished state.

Tests to come.

potiuk · 2025-06-19T10:32:29Z

FYI. Not sure if it helps but OpenAPI 3.2.0 will have support for application/json-seq OAI/OpenAPI-Specification#3730 (planned for August)

uranusjr · 2025-06-25T05:25:14Z

I added some tests for the endpoint, but couldn’t figure out how to test the looping part. Hopefully this is good enough…

pierrejeambrun

Nice, just a few suggestions / nits.

Indeed an additional test case for running / success state would be great.

pierrejeambrun

LGTM, thanks.

Just a few nits, but nothing blocking

kaxil

Some simple docs would be good. Fine as a follow-up too as long as we have a GH issue to not forget.

@vikramkoka could review that too

uranusjr · 2025-07-09T06:34:05Z

Tracking doc addition #53067

(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1) ## New Features: - Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614)) - Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563)) - Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882)) - Implement pattern search for event log endpoint ([#55114](apache/airflow#55114)) - Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263)) - Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302)) - Add `try_number` as filter to task instances ([#54695](apache/airflow#54695)) - Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049)) - Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004)) - Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652)) - Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408)) - Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177)) - Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437)) - Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738)) - Expose DAG parsing duration in the API ([#54752](apache/airflow#54752)) ## New API Endpoints: - Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776)) - Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920)) - Add TI bulk actions endpoint ([#50443](apache/airflow#50443)) - Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657)) ## Deprecations: - Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150)) ## Bug Fixes: - Fix dag import error modal pagination ([#55719](apache/airflow#55719))

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

boring-cyborg bot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Jun 19, 2025

uranusjr mentioned this pull request Jun 19, 2025

Enable Inference Execution / Synchronous DAG Execution #51711

Open

uranusjr added this to the Airflow 3.1.0 milestone Jun 24, 2025

uranusjr force-pushed the stream-until-run-complete branch 2 times, most recently from 231597b to 007a051 Compare June 25, 2025 04:53

uranusjr marked this pull request as ready for review June 25, 2025 05:25

uranusjr requested review from bbovenzi, bugraoz93, ephraimbuddy, jason810496, jscheffl, pierrejeambrun, rawwar, ryanahamilton and shubhamraj-git as code owners June 25, 2025 05:25

pierrejeambrun reviewed Jun 25, 2025

View reviewed changes

Comment thread airflow-core/src/airflow/api_fastapi/core_api/datamodels/dag_run.py Outdated

Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py Outdated

Comment thread airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py

jason810496 reviewed Jun 27, 2025

View reviewed changes

Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py

uranusjr force-pushed the stream-until-run-complete branch from 007a051 to 152bd6d Compare July 3, 2025 09:31

pierrejeambrun approved these changes Jul 3, 2025

View reviewed changes

Comment thread airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py Outdated

Comment thread airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py Outdated

Comment thread airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml

uranusjr force-pushed the stream-until-run-complete branch from 152bd6d to 34badc1 Compare July 7, 2025 07:04

pierrejeambrun approved these changes Jul 7, 2025

View reviewed changes

uranusjr force-pushed the stream-until-run-complete branch 2 times, most recently from 96812e6 to 422cd94 Compare July 7, 2025 19:27

bugraoz93 approved these changes Jul 7, 2025

View reviewed changes

kaxil reviewed Jul 7, 2025

View reviewed changes

Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py Outdated

kaxil approved these changes Jul 7, 2025

View reviewed changes

Add endpoint to watch dag run until finish

e1e3149

uranusjr added 4 commits July 9, 2025 07:40

Include xcom in result if requested

edf8787

Use 'params' instead of literal params

77ec3ac

Mark endpoint as experimental

f852f6b

Rename param key

58d7fa5

uranusjr force-pushed the stream-until-run-complete branch from 422cd94 to 58d7fa5 Compare July 9, 2025 05:40

uranusjr mentioned this pull request Jul 9, 2025

Add docs for the wait dagrun endpoint #53067

Closed

1 task

uranusjr merged commit 5fbbf64 into apache:main Jul 9, 2025
102 checks passed

uranusjr deleted the stream-until-run-complete branch July 9, 2025 06:34

HsiuChuanHsu pushed a commit to HsiuChuanHsu/airflow that referenced this pull request Jul 10, 2025

Add endpoint to watch dag run until finish (apache#51920)

2a6e549

stephen-bracken pushed a commit to stephen-bracken/airflow that referenced this pull request Jul 15, 2025

Add endpoint to watch dag run until finish (apache#51920)

ad1932e

kaxil mentioned this pull request Oct 22, 2025

Update Python Client to 3.1.0 apache/airflow-client-python#138

Merged

kaxil mentioned this pull request Apr 15, 2026

Fix dispose_orm() not disposing async engine on shutdown #65274

Merged

kaxil mentioned this pull request Apr 15, 2026

Fix async engine missing pool_recycle and pool_pre_ping configuration #65276

Open

vatsrahul1001 mentioned this pull request Apr 15, 2026

[v3-2-test] Fix dispose_orm() not disposing async engine on shutdown (#65274) #65284

Merged

vatsrahul1001 mentioned this pull request Apr 15, 2026

Status of testing of Apache Airflow 3.2.1rc2 #65336

Open

53 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add endpoint to watch dag run until finish#51920

Add endpoint to watch dag run until finish#51920
uranusjr merged 5 commits intoapache:mainfrom
astronomer:stream-until-run-complete

uranusjr commented Jun 19, 2025 •

edited

Loading

Uh oh!

potiuk commented Jun 19, 2025

Uh oh!

uranusjr commented Jun 25, 2025

Uh oh!

pierrejeambrun left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaxil left a comment

Uh oh!

uranusjr commented Jul 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

uranusjr commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Jun 19, 2025

Uh oh!

uranusjr commented Jun 25, 2025

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaxil left a comment

Choose a reason for hiding this comment

Uh oh!

uranusjr commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

uranusjr commented Jun 19, 2025 •

edited

Loading

uranusjr commented Jul 9, 2025 •

edited

Loading