Skip to content

feat: add resource-scoped jobs plugin for Databricks Lakeflow Jobs#223

Open
keugenek wants to merge 5 commits intodatabricks:feat/proto-pluginfrom
keugenek:ekniazev/jobs-plugin-core
Open

feat: add resource-scoped jobs plugin for Databricks Lakeflow Jobs#223
keugenek wants to merge 5 commits intodatabricks:feat/proto-pluginfrom
keugenek:ekniazev/jobs-plugin-core

Conversation

@keugenek
Copy link
Copy Markdown
Contributor

@keugenek keugenek commented Mar 31, 2026

Summary

Resource-scoped jobs plugin following the files plugin pattern. Jobs are configured as named resources discovered from environment variables at startup.

Design

  • Resource-scoped: Only configured jobs are accessible — this is not an open SDK wrapper
  • Env-var discovery: Jobs are discovered from DATABRICKS_JOB_<KEY> env vars (e.g. DATABRICKS_JOB_ETL=123)
  • Single-job shorthand: DATABRICKS_JOB_ID maps to the "default" key
  • Manifest declares job resources with CAN_MANAGE_RUN permission
  • Works with databricks apps init --features jobs

API

// Trigger a configured job
const { run_id } = await appkit.jobs("etl").runNow();

// Trigger and wait for completion
const run = await appkit.jobs("etl").runNowAndWait();

// OBO access
await appkit.jobs("etl").asUser(req).runNow();

// List recent runs
const runs = await appkit.jobs("etl").listRuns({ limit: 10 });

// Single-job shorthand
await appkit.jobs("default").runNow();

Files changed

  • plugins/jobs/manifest.json — declares job resource with CAN_MANAGE_RUN permission
  • plugins/jobs/types.tsJobAPI, JobHandle, JobsExport, IJobsConfig types
  • plugins/jobs/plugin.tsJobsPlugin with discoverJobs(), getResourceRequirements(), resource-scoped createJobAPI()
  • plugins/jobs/index.ts — barrel exports
  • connectors/jobs/client.tslistRuns now respects limit parameter
  • plugins/jobs/tests/plugin.test.ts — 32 tests covering discovery, resource requirements, exports, OBO, multi-job, and auto-fill

Documentation safety checklist

  • Examples use least-privilege permissions
  • Sensitive values are obfuscated
  • No insecure patterns introduced

pkosiec and others added 3 commits March 30, 2026 11:34
databricks#219)

* chore: add SBOM generation and vulnerability audit to release pipeline

- Add @cyclonedx/cdxgen as devDependency for CycloneDX SBOM generation
- Add `pnpm audit --audit-level=high` as before:init hook (blocks release on high/critical vulns)
- Generate per-package CycloneDX SBOM in before:release hook (included in npm tarball)
- Applied to both appkit/appkit-ui and lakebase release configs

Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>

* chore: improve SBOM generation in release pipeline

- Include sbom.cdx.json in published npm packages via files allowlist
- Extract cdxgen into named release:sbom scripts for readability
- Use pnpm exec cdxgen for explicit dependency resolution
- Scan tmp/ (publish artifact) instead of source dirs for accurate SBOMs
- Scope pnpm audit to production deps only (--prod)

Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>

---------

Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
* feat: plugin client config

chore: fixup

chore: fixup

chore: fixup

chore: fixup

chore: fixup

chore: fixup

chore: fixup

* chore: fixup

* chore: fixup

---------

Co-authored-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…nals (databricks#222)

v3.0.1 internally references upload-artifact@v4 by tag, which fails the
org action-pinning policy. v4.0.0 pins its internal dependency by full SHA.

Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
…sage overflow (databricks#196)

* fix: handle in-progress Genie messages on page reload and query overflow

When reloading a page while a Genie message is still loading, the
history endpoint returned the in-progress message which the frontend
silently dropped (no attachments yet). This left the UI broken with
no way to recover.

Add a single-message polling endpoint (GET /:alias/conversations/
:conversationId/messages/:messageId) that SSE-streams status updates
until the message completes. The frontend now detects pending messages
after history load and polls via this endpoint, reusing the existing
processStreamEvent pipeline.

Also fix wide query results overflowing beyond the message bubble by
switching to overflow-x-auto and adding min-w-0 constraints.

Signed-off-by: Jorge Calvar <jorge.calvar@databricks.com>

* fix: prevent Genie message bubbles from overflowing the chat viewport

Message bubbles extended past the visible area for two reasons:

1. The content column used items-start/items-end for alignment, which
   caused Cards to size based on their content width instead of
   stretching to the column. Wide content (tables, long text) pushed
   Cards wider than the 80% column, with overflow clipped visually
   but text wrapping at the wider intrinsic width.

2. Radix ScrollArea inserts a wrapper div with display:table that
   grows to fit content. This made the entire scroll container wider
   than the viewport, so percentage-based widths resolved against
   the wider container.

Fix:
- Remove items-start/items-end from the content column
- Add w-full to all Cards so they always match the column width
- Override Radix's table wrapper to display:block via targeted
  selector on the scroll area viewport
- Add break-words to markdown content and make markdown tables
  scrollable within the bubble

Signed-off-by: Jorge Calvar <jorge.calvar@databricks.com>

* fix: address PR feedback — deduplicate terminal statuses and fix Genie robustness issues

Deduplicates TERMINAL_STATUSES into types.ts, fixes phantom placeholder
on FAILED messages, threads abort signal into streamGetMessage polling,
adds missing .catch() on fire-and-forget SSE promises, and stabilizes
useCallback chain via ref to prevent identity cascade.

Co-authored-by: Isaac
Signed-off-by: Jorge Calvar <jorge.calvar@databricks.com>

---------

Signed-off-by: Jorge Calvar <jorge.calvar@databricks.com>
@keugenek keugenek requested review from calvarjorge and ditadi April 1, 2026 16:02
@keugenek keugenek self-assigned this Apr 1, 2026
Copy link
Copy Markdown
Contributor

@atilafassina atilafassina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found also 2 potential type mismatches with protobuf

  1. @bufbuild/protobuf toJson() defaults to camelCase. The useProtoFieldName: true option is needed for snake_case as documented.

  2. Partial<MessageShape<T>> allows setting properties to undefined explicitly, whereas PartialMessage<T> from protobuf properly handles nested partial messages and enum defaults.

Comment on lines +91 to +95
return this.execute(
(sig) =>
this.connector.runNow(getWorkspaceClient(), request, sig ?? signal),
{ timeout: this.config.timeout ?? 60000 },
) as Promise<jobsTypes.RunNowResponse>;
Copy link
Copy Markdown
Contributor

@atilafassina atilafassina Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the call fails, execute will swallow the error and return undefined. The assertion on line 95 ignores this possibility.

Also, I think that timeout is not being applied. I think the right syntax for that call would be something like:

this.execute(
    (sig) => this.connector.submitRun(getWorkspaceClient(), request, sig ?? signal),
    { default: { timeout: this.config.timeout ?? 60000 } },
  );

Note

this is happening in all this.execute() calls

timeoutMs?: number,
signal?: AbortSignal,
): Promise<jobsTypes.Run> {
return this.connector.waitForRun(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason only this one is not using this.execute ?

Comment on lines +235 to +247
exports() {
return {
submitRun: this.submitRun.bind(this),
runNow: this.runNow.bind(this),
getRun: this.getRun.bind(this),
getRunOutput: this.getRunOutput.bind(this),
cancelRun: this.cancelRun.bind(this),
listRuns: this.listRuns.bind(this),
getJob: this.getJob.bind(this),
createJob: this.createJob.bind(this),
waitForRun: this.waitForRun.bind(this),
};
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

harmless, but these top-level bindings should be automatic on registration call

@keugenek keugenek force-pushed the ekniazev/jobs-plugin-core branch from 9715283 to 11f7d46 Compare April 2, 2026 10:38
pkosiec
pkosiec previously requested changes Apr 2, 2026
Copy link
Copy Markdown
Member

@pkosiec pkosiec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@keugenek Took a quick high-level look and I think we should take a step back and think about the user flow for this plugin. Right now it is a light wrapper over the JS SDK client and effectively re-exports the API. I don't see any significant benefits to use plugin over the SDK :thinking_face:

For all the current plugins, a user can do databricks apps init and select a plugin and interactively all the required resources (or pass them with a flag), and then we put the selection in databricks.yml and .env for seamless local development and zero-config deployment.

I'd assume it should work similarly with the jobs plugin - which means, it would need to have a job-oriented API (multiple jobs can be supported too, but in the interactive selection, we'd support one - similarly as in the model serving API).

So I think we need to agree on the API first, and then we need to have template changes, and test the existing apps init flow.

Let me know if that makes sense! Feel free to dismiss my review if I'd be OOO, but I'd really love to rework the plugin to be more user-oriented. Thank you!

Jobs are configured as named resources (DATABRICKS_JOB_<KEY> env vars)
and discovered at startup, following the files plugin pattern.

API is scoped to configured jobs:
  appkit.jobs('etl').runNow()
  appkit.jobs('etl').runNowAndWait()
  appkit.jobs('etl').lastRun()
  appkit.jobs('etl').listRuns()
  appkit.jobs('etl').asUser(req).runNow()

Single-job shorthand via DATABRICKS_JOB_ID env var.
Supports OBO access via asUser(req).

Co-authored-by: Isaac
Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
@keugenek keugenek force-pushed the ekniazev/jobs-plugin-core branch from 11f7d46 to b0e3be3 Compare April 2, 2026 16:04
@keugenek keugenek changed the title feat: add jobs connector and plugin for Databricks Jobs API feat: add resource-scoped jobs plugin for Databricks Lakeflow Jobs Apr 2, 2026
@pkosiec pkosiec dismissed their stale review April 2, 2026 16:35

Dismissing after the last commit

@pkosiec
Copy link
Copy Markdown
Member

pkosiec commented Apr 2, 2026

Thanks very much @keugenek! I didn't check the PR details but the high-level changes are exactly what I was looking for. Dismissed my review before going offline to avoid blocking merge 👍
image

Thanks again for making those changes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants