Skip to content

Fix dispose_orm() not disposing async engine on shutdown#65274

Merged
vatsrahul1001 merged 1 commit intoapache:mainfrom
astronomer:fix/dispose-orm-async-engine
Apr 15, 2026
Merged

Fix dispose_orm() not disposing async engine on shutdown#65274
vatsrahul1001 merged 1 commit intoapache:mainfrom
astronomer:fix/dispose-orm-async-engine

Conversation

@kaxil
Copy link
Copy Markdown
Member

@kaxil kaxil commented Apr 15, 2026

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

  • atexit.register(dispose_orm) -- process exit
  • Gunicorn worker_exit hook -- worker restarts
  • reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

@kaxil kaxil added this to the Airflow 3.2.1 milestone Apr 15, 2026
@kaxil kaxil requested review from ashb, dstandish and uranusjr April 15, 2026 01:20
@kaxil kaxil force-pushed the fix/dispose-orm-async-engine branch from 8cd2b62 to 70bc934 Compare April 15, 2026 01:23
When _configure_async_session() was extracted from configure_orm() in
PR apache#51920, the corresponding cleanup in dispose_orm() was not updated.
This left async_engine connections abandoned on process exit, gunicorn
worker restarts, and atexit -- gradually exhausting PostgreSQL's
max_connections.

Dispose async_engine.sync_engine (the synchronous path, matching
the existing clean_in_fork pattern) and clear both async_engine and
AsyncSession references.
@kaxil kaxil force-pushed the fix/dispose-orm-async-engine branch from 70bc934 to f44d845 Compare April 15, 2026 01:25
@vatsrahul1001 vatsrahul1001 merged commit 69a88bf into apache:main Apr 15, 2026
79 checks passed
@vatsrahul1001 vatsrahul1001 deleted the fix/dispose-orm-async-engine branch April 15, 2026 06:00
vatsrahul1001 pushed a commit that referenced this pull request Apr 15, 2026
When _configure_async_session() was extracted from configure_orm() in
PR #51920, the corresponding cleanup in dispose_orm() was not updated.
This left async_engine connections abandoned on process exit, gunicorn
worker restarts, and atexit -- gradually exhausting PostgreSQL's
max_connections.

Dispose async_engine.sync_engine (the synchronous path, matching
the existing clean_in_fork pattern) and clear both async_engine and
AsyncSession references.

(cherry picked from commit 69a88bf)
vatsrahul1001 added a commit that referenced this pull request Apr 15, 2026
…5284)

When _configure_async_session() was extracted from configure_orm() in
PR #51920, the corresponding cleanup in dispose_orm() was not updated.
This left async_engine connections abandoned on process exit, gunicorn
worker restarts, and atexit -- gradually exhausting PostgreSQL's
max_connections.

Dispose async_engine.sync_engine (the synchronous path, matching
the existing clean_in_fork pattern) and clear both async_engine and
AsyncSession references.

(cherry picked from commit 69a88bf)

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
vatsrahul1001 added a commit that referenced this pull request Apr 15, 2026
…5284)

When _configure_async_session() was extracted from configure_orm() in
PR #51920, the corresponding cleanup in dispose_orm() was not updated.
This left async_engine connections abandoned on process exit, gunicorn
worker restarts, and atexit -- gradually exhausting PostgreSQL's
max_connections.

Dispose async_engine.sync_engine (the synchronous path, matching
the existing clean_in_fork pattern) and clear both async_engine and
AsyncSession references.

(cherry picked from commit 69a88bf)

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
vatsrahul1001 added a commit that referenced this pull request Apr 15, 2026
…5284)

When _configure_async_session() was extracted from configure_orm() in
PR #51920, the corresponding cleanup in dispose_orm() was not updated.
This left async_engine connections abandoned on process exit, gunicorn
worker restarts, and atexit -- gradually exhausting PostgreSQL's
max_connections.

Dispose async_engine.sync_engine (the synchronous path, matching
the existing clean_in_fork pattern) and clear both async_engine and
AsyncSession references.

(cherry picked from commit 69a88bf)

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
vatsrahul1001 added a commit that referenced this pull request Apr 15, 2026
…5284)

When _configure_async_session() was extracted from configure_orm() in
PR #51920, the corresponding cleanup in dispose_orm() was not updated.
This left async_engine connections abandoned on process exit, gunicorn
worker restarts, and atexit -- gradually exhausting PostgreSQL's
max_connections.

Dispose async_engine.sync_engine (the synchronous path, matching
the existing clean_in_fork pattern) and clear both async_engine and
AsyncSession references.

(cherry picked from commit 69a88bf)

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
vatsrahul1001 added a commit that referenced this pull request Apr 15, 2026
…5284)

When _configure_async_session() was extracted from configure_orm() in
PR #51920, the corresponding cleanup in dispose_orm() was not updated.
This left async_engine connections abandoned on process exit, gunicorn
worker restarts, and atexit -- gradually exhausting PostgreSQL's
max_connections.

Dispose async_engine.sync_engine (the synchronous path, matching
the existing clean_in_fork pattern) and clear both async_engine and
AsyncSession references.

(cherry picked from commit 69a88bf)

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Apr 16, 2026
When _configure_async_session() was extracted from configure_orm() in
PR apache#51920, the corresponding cleanup in dispose_orm() was not updated.
This left async_engine connections abandoned on process exit, gunicorn
worker restarts, and atexit -- gradually exhausting PostgreSQL's
max_connections.

Dispose async_engine.sync_engine (the synchronous path, matching
the existing clean_in_fork pattern) and clear both async_engine and
AsyncSession references.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants