Integrate JiT compilation into CLI by rafal-hawrylak · Pull Request #2110 · dataform-co/dataform

rafal-hawrylak · 2026-03-09T20:08:25Z

No description provided.

- Enhance error coercion in common/errors/errors.ts - Sync protos/execution.proto and protos/jit.proto with new fields

* feat: add worker process management for JiT compilation - Introduce base worker and JiT-specific child process logic - Implement RPC bridge for database access during JiT - Add VM scripts and loader for isolated execution - Add unit tests for the RPC mechanism

* feat: add action descriptor support to JiT compilation results - Implement action and column descriptor logic in core/jit_compiler.ts - Support overrides for descriptions, labels, and tags in JiT results - Enhance error coercion in common/errors/errors.ts - Add comprehensive JiT metadata tests in core/jit_compiler_test.ts - Sync protos/execution.proto and protos/jit.proto with new result fields

* feat: enable JiT compilation in CLI run, build and compile commands - Connect main CLI execution flow to the JiT worker process - Add comprehensive JiT test suite - Finalize AoT build logic to preserve JiT-specific fields - Update CLI console and entry points for integrated JiT support

kolina

Let's test compatibility with different Dataform core versions.

Also this change seems worth it to bump a minor version of Dataform core

kolina · 2026-03-10T19:55:29Z

cli/api/commands/compile.ts

-      this.childProcess.on("close", exitCode => {
-        if (exitCode !== 0) {
-          reject(new Error(`Compilation child process exited with exit code ${exitCode}.`));
+    const timeoutValue = compileConfig.timeoutMillis || DEFAULT_COMPILATION_TIMEOUT_MILLIS;


It seems that the default timeout is smaller currently?

kolina · 2026-03-10T20:00:51Z

cli/api/commands/compile.ts

-        reject(coerceAsError(messageOrError));
-      });
-
-      this.childProcess.on("close", exitCode => {


Just to clarify: in the base worker you have a hook for "exit", they'll be equivalent?

kolina · 2026-03-10T20:02:58Z

cli/api/commands/compile.ts

-      return await compileInChildProcess;
-    } finally {
-      if (!this.childProcess.killed) {
-        this.childProcess.kill("SIGKILL");


In the base worker you don't send SIGKILL, I'd preserve this logic to avoid compiler ignoring SIGTERM for some reason

kolina · 2026-03-10T20:08:43Z

cli/api/commands/run.ts

    private readonly dbadapter: dbadapters.IDbAdapter,
    private readonly graph: dataform.IExecutionGraph,
-    private readonly executionOptions: IExecutionOptions = {},
+    private readonly projectDir: string,


if it's part of IExecutionOptions, do you need to pass it separately?

kolina · 2026-03-10T20:14:01Z

cli/api/commands/run.ts

-        if (this.stopped) {
-          return actionResult;
+    try {
+      await this.dbadapter.withClientLock(async client => {


This withClientLock function doesn't do anything meaningful today (maybe it was needed before when we supported other warehouses), I'd remove it.

kolina · 2026-03-10T22:06:36Z

cli/jit_run_test.ts

+        type: "table",
+        tableType: "incremental",
+        jitCode: `async (jctx) => {
+          return jctx.incremental() ? "SELECT 'inc' as t" : "SELECT 'full' as t";


add a test for incremental mode?

kolina · 2026-03-10T22:58:05Z

cli/api/commands/compile.ts

-    this.childProcess = childProcess;
+export class CompileChildProcess extends BaseWorker<string, string | Error> {
+  constructor() {
+    super("../../vm/compile_loader");


Do I understand correctly that this will change the order of fork script resolution? I'd prefer to keep the original order

kolina · 2026-03-10T23:31:27Z

packages/@dataform/cli/worker.ts

+      // AoT compile() in cli/vm/compile.ts returns a JSON string of CompiledGraph.
+      // We need to wrap it in a CoreExecutionResponse and Base64 encode it
+      // to match what cli/api/commands/compile.ts expects.


Is it true? From my reading of code:

compile in cli/vm/compile.ts calls main function of Dataform core and returns its result here, in the CLI we always pass base-64 core execution request here

The main function in the Dataform core in this case returns base64-encoded serialized proto response here (comment about older version of the Dataform CLI is a bit confusing)

kolina · 2026-03-10T23:41:28Z

protos/execution.proto

-  reserved 1, 3, 7, 13;
+  bool disabled = 15;
+
+  reserved 1, 3, 7;


why do we have less reserved field numbers here?

kolina · 2026-03-10T23:46:28Z

tests/api/api.spec.ts

        await callback(mockDbAdapterInstance);

-      const runner = new Runner(mockDbAdapterInstance, RUN_TEST_GRAPH);
+      const runner = new Runner(mockDbAdapterInstance, RUN_TEST_GRAPH, ".");


If someone uses the Dataform CLI as a library, will they need to update their code in the same manner?

ikholopov-omni

Blocked on review of https://github.com/dataform-co/dataform/pull/2109/changes

rafal-hawrylak self-assigned this Mar 9, 2026

rafal-hawrylak force-pushed the cli_jit_integration branch from fdccca4 to 371b080 Compare March 9, 2026 20:26

rafal-hawrylak requested review from andrzej-grudzien, apilaskowski, ikholopov-omni and kolina March 9, 2026 20:28

rafal-hawrylak marked this pull request as ready for review March 9, 2026 20:28

rafal-hawrylak requested a review from a team as a code owner March 9, 2026 20:28

rafal-hawrylak enabled auto-merge (squash) March 9, 2026 20:28

rafal-hawrylak added 5 commits March 10, 2026 14:22

JiT: protos update to enable execution in CLI

cc0cdc8

- Enhance error coercion in common/errors/errors.ts - Sync protos/execution.proto and protos/jit.proto with new fields

feat: integrate JiT compilation and finalize test suite

62dc58f

rafal-hawrylak force-pushed the cli_jit_integration branch from 371b080 to 18042dd Compare March 10, 2026 15:49

kolina requested changes Mar 11, 2026

View reviewed changes

ikholopov-omni requested changes Mar 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate JiT compilation into CLI#2110

Integrate JiT compilation into CLI#2110
rafal-hawrylak wants to merge 5 commits intomainfrom
cli_jit_integration

rafal-hawrylak commented Mar 9, 2026

Uh oh!

kolina left a comment

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

kolina Mar 10, 2026

Uh oh!

ikholopov-omni left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

rafal-hawrylak commented Mar 9, 2026

Uh oh!

kolina left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ikholopov-omni left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants