Lakeflow: a pipeline that doesn’t belong to one person

Header sketch for the post: Lakeflow: a pipeline that doesn’t belong to one person
In shortSince 5 October 2026 a Lakeflow pipeline can run as an account group, so the team, not a person or a technical account, owns the tables it publishes. I tested it through the REST API and a bundle (CLI v1.20.0), including which grants the group really needs. Watch out for a PUT without run_as and for a manual run-as change that a deploy doesn’t undo.

Problem

By default a pipeline runs as its creator, and when that account is disabled the pipeline stops. The fix so far was a service principal, but then the tables belong to a technical account. In Unity Catalog pipelines the owner of materialized views and streaming tables is the run-as identity, which could only be a user or a service principal, so a group couldn't own the tables.

Since 5 October run-as can be an account group. The team owns the tables, and membership changes break nothing. This is the second of two Lakeflow changes from the same day. The first one, several triggers on one job, is covered in Lakeflow Jobs: multiple triggers on one job.

How it works

A pipeline's run-as can now be an account group, next to a user and a service principal. Updates run with the group's permissions, and in Unity Catalog pipelines the group owns the published tables. The release note doesn't mark the feature as Preview or Beta. A role in RBAC is a regular account group, so a pipeline whose run-as is a group runs as that role.

The docs still call a service principal the best practice for automation. A group makes sense when the pipeline should belong to a team rather than a technical account. Keep in mind, though, that every member of the group gets all the permissions you grant it for the pipeline.

Requirements from the docs:

  • The account group has to be assigned to the workspace. Workspace-local groups don't work, and neither do the built-in admins, users and account users.
  • Run-as is set by a member of the group or someone with Assume on it (when the group is used for exclusive access), who also has CAN_MANAGE on the pipeline. In CI/CD that's the deployment service principal, and group membership also gives it all the group's data permissions.
  • The group needs the Workspace access entitlement (since 14 September 2026 it has to be granted explicitly), Can read on the code and Unity Catalog grants on the target schema and the sources (plus MODIFY on tables it updates and CREATE SCHEMA if it creates schemas). The docs also list Can use on the cluster policy and the right to create compute; on serverless, Workspace access was enough. Your own permissions don't help, because the update runs as the group.
  • Run-as and the pipeline owner are separate settings, so switching run-as to a group doesn't change the pipeline owner.

Jobs can run as a group too (the docs have no Preview label), but in this post I only test pipelines.

In the REST API you set run_as.group_name when you create the pipeline. The variables (catalog, schema, GROUP, PIPE_DIR), the time import, the api client and me come from the notebook's first cell, and the pipeline code is one materialized view in a SQL file:

✓ Tested on Free Edition (2026-10-17): all notebook steps pass (I describe the delay when creating the pipeline in the pitfalls). Requires Unity Catalog and serverless.

pipeline = {
    "name": "dataistheway_etl",
    "catalog": catalog,
    "schema": schema,
    "serverless": True,
    "libraries": [{"file": {"path": PIPE_DIR + "/mv_total_by_country.sql"}}],
    "run_as": {"group_name": GROUP},
}
pid = api.do("POST", "/api/2.0/pipelines", body=pipeline)["pipeline_id"]
print("run_as_user_name:", api.do("GET", f"/api/2.0/pipelines/{pid}")["run_as_user_name"])

def run_update():
    t0 = time.time()
    uid = api.do("POST", f"/api/2.0/pipelines/{pid}/updates", body={})["update_id"]
    while True:
        st = api.do("GET", f"/api/2.0/pipelines/{pid}/updates/{uid}")["update"]["state"]
        if st in ("COMPLETED", "FAILED", "CANCELED"):
            return f"{st} ({time.time() - t0:.0f} s)"
        time.sleep(10)

print("update as the group:", run_update())

When you edit an existing pipeline (PUT /api/2.0/pipelines/{pipeline_id}), the docs say to use the updated_run_as field. In my lab that wasn't enough, so the notebook tries both fields in turn:

✓ Tested on Free Edition (2026-10-17): same result, updated_run_as rejected, run_as accepted.

spec = api.do("GET", f"/api/2.0/pipelines/{pid}")["spec"]
try:
    api.do("PUT", f"/api/2.0/pipelines/{pid}", body={**spec, "updated_run_as": {"user_name": me.user_name}})
    print("updated_run_as accepted")
except Exception as e:
    print("updated_run_as:", str(e).splitlines()[0])

api.do("PUT", f"/api/2.0/pipelines/{pid}", body={**spec, "run_as": {"user_name": me.user_name}})
print("run_as_user_name:", api.do("GET", f"/api/2.0/pipelines/{pid}")["run_as_user_name"])

The API rejected a PUT with the full spec and only updated_run_as (no run_as), as if we wanted to clear run-as:

Once configured, run_as of the pipeline can not be set to null.
Please set run_as to dataistheway-etl-team to preserve the pipeline's run_as

GET returns run_as next to spec, not inside it. PUT replaces the whole definition, so a request body built from spec alone means clearing run-as to the API, even when you only change the name. When I sent both fields with different identities, run_as always won. So every PUT of a pipeline with run-as set has to include run_as, and after the change read run_as_user_name.

Which grants the group really needs

In my lab I took one permission at a time away from the group (schema grants at the schema level) and ran the pipeline from the bundle on the first update (no MV yet) and on a refresh of the existing MV:

Without First update MV refresh
USE CATALOG deploy doesn't create the pipeline update doesn't start
USE SCHEMA FAILED FAILED
CREATE TABLE FAILED FAILED
CREATE MATERIALIZED VIEW FAILED COMPLETED
SELECT FAILED FAILED
MODIFY COMPLETED COMPLETED
Can read on the code folder not tested FAILED

MODIFY wasn't needed for the MV, and CREATE MATERIALIZED VIEW only when creating it. CREATE TABLE was needed for a refresh too, and the pipelines SQL reference for MVs lists it together with CREATE MATERIALIZED VIEW. The errors name the group directly, except a missing Can read on the code, which looks like a missing file: [LIBRARY_FILE_NOT_FOUND] File '.../files/src/mv_total_by_country.sql' was not found or the run_as user does not have permissions to access it. The matrix covers the MV. For streaming tables and updated tables, the list from the docs applies.

Bundle

In Declarative Automation Bundles (formerly Databricks Asset Bundles) you set group_name in the run_as of the pipeline resource itself (resources.pipelines.<key>.run_as). Three things from the lab and the docs:

  • CLI v1.20.0 or newer. On v1.16.1 bundle validate only reports unknown field: group_name, and bundle deploy fails with a 400 and no pipeline is created. The CLI schema still marks the field as Private Preview for pipelines (checked on v1.23.0).
  • The default folder rather than /Workspace/Shared/, even though the Jobs docs suggest /Shared. There the bundle folder inherits CAN_MANAGE for users, so anyone can replace code that runs with the group's permissions; the bundles reference advises against /Shared in production. In the default folder the CLI gave the group CAN_READ.
  • The next deploy doesn't undo a manual run-as change (bundle plan: 0 to change). The group only comes back with a deploy that changes something in the pipeline definition, so check run_as_user_name after every deploy in CI.

✓ Tested on Free Edition (2026-10-19): deploy and run of this bundle with a group pass, and the group owns the MV. I tested the bundle on CLI v1.20.0.

# A pipeline that runs as an account group (run_as.group_name).
# Tested on Databricks CLI v1.20.0 (the field has been in the schema since v1.18.0).
bundle:
  name: run_as_group_demo

variables:
  catalog:
    default: workspace
  schema:
    default: dataistheway_run_as
  group_name:
    default: dataistheway-etl-team

# No workspace.root_path: the code stays in the home folder of whoever deploys
# (in CI: the service principal), so other users can't change it.
# These permissions give the group CAN_VIEW on the pipeline. Can read on the code
# folder is granted by the CLI itself when you keep the default folder.
permissions:
  - level: CAN_VIEW
    group_name: ${var.group_name}

resources:
  pipelines:
    etl:
      name: dataistheway_etl_bundle
      catalog: ${var.catalog}
      schema: ${var.schema}
      serverless: true
      libraries:
        - file:
            path: ./src/mv_total_by_country.sql
      run_as:
        group_name: ${var.group_name}

targets:
  dev:
    default: true

The source_orders table has to exist, and the variables go in BUNDLE_VAR_catalog, BUNDLE_VAR_schema and BUNDLE_VAR_group_name, then databricks bundle deploy and databricks bundle run etl. In my lab the update ended COMPLETED in about 40 s, and the group owns mv_total_by_country.

The lab demo

The pipeline_run_as_group.py notebook does everything through the REST API on serverless, with the pipeline/ folder holding the pipeline code next to it. You run it as a workspace admin, because it creates an account group. The notebook creates the dataistheway-etl-team group, grants it access, creates a pipeline with run_as.group_name, shows the MV owner, tries the users group, switches run-as back to us and tries SET OWNER, then deletes the pipeline, table and group (a deleted group can still show up the next day, so change its name in the widget before a rerun).

In my lab (Azure Databricks Premium, trial) the update as the group ended COMPLETED after 42 s, and dataistheway-etl-team owns mv_total_by_country. After switching run-as back to the user, the MV owner changed immediately, without a new update.

Table owners after the update as the group

Lab result: information_schema.tables, the table_owner column.

Switching run-as from the group to the user

Lab result: the error with updated_run_as alone and the login after the change through run_as.

A pipeline whose run-as is a group: the MV owner, switching run-as and a rejected SET OWNER. Waiting is sped up 8×. Code from the recording: lakeflow-pipeline-run-as-group

Code for this article on GitHub →notebook and a step-by-step README · lakeflow-pipeline-run-as-group

Pitfalls

  • The built-in admins, users and account users can't be a pipeline's run-as. The pipelines endpoint returns (same on Free Edition) The group 'users' cannot be used as a run-as identity for jobs. Broad groups such as 'account users', 'admins', and 'users' are not allowed., although the message talks about jobs.
  • An account group that isn't assigned to the workspace gives the same deploy error as a group that doesn't exist: '<group>' cannot be set as run_as because the group does not exist. Before hunting for a typo, check the group's assignment.
  • Right after a group is created, its membership may not yet be visible to the pipelines API. On Free Edition creating the pipeline right after creating the group failed with Cannot bind the group provided in 'run_as' field ..., and the same POST passed a few minutes later (on the Premium lab right away).
  • The pipeline owner and run-as are two different things: after you change run-as, the group owns the tables, but the pipeline owner stays the same. It's simplest when a bundle from CI/CD on a service principal deploys the pipeline: the SP is then the owner and the group is run-as.
  • The table owner changes together with run-as immediately (after switching run-as in the demo), although the Unity Catalog docs say it happens on the next refresh. A "temporary, just to test" run-as change takes table ownership away from the group, so in production only the CI deployment sets run-as.
  • According to the docs the pipeline refreshes MVs as the run-as identity, so if the group is subject to an ABAC policy on the sources, the MV permanently holds masked data. Adding the group to EXCEPT unmasks every member on all tables the policy covers, also in notebooks and Genie. Readers of the MV are then protected only by a policy on governed tags set by hand on its columns (ALTER MATERIALIZED VIEW ... ALTER COLUMN ... SET TAGS, DBR 16.3+), because tag propagation (Beta) doesn't cover pipeline MVs. I haven't tested an ABAC policy on such an MV. An empty role group isn't fully safe either: after assuming a role listed in EXCEPT, the policy doesn't apply to the session, so give Assume only to the CI service principal.
  • SET OWNER TO won't change the owner of a pipeline table: [CHANGE_OWNER_NOT_ALLOWED_FOR_PIPELINE_TABLE] Changing owner is not supported for a table managed by the pipeline with ID '<id>'. Modify the run-as identity of the pipeline instead. On Free Edition the ending is worded differently, so scripts should catch the error code, not the whole text.
  • According to the docs a run-as change can interrupt an active update: a continuous pipeline restarts with the new identity, and a triggered one applies it on the next run.
  • After bundle destroy the pipeline is gone, but mv_total_by_country stays in the schema, and the first update of a new pipeline fails with Table ... is already managed by pipeline <id>. A DROP MATERIALIZED VIEW before the next deploy helps.

When NOT to use it

  • When the group has people and broad permissions, because every member gets everything you grant it for the pipeline (for example MODIFY on production tables). A service principal or an empty role group with Assume only for the CI service principal is better then. With ABAC such a group has to stay empty, because the EXCEPT exception covers every member.
  • When you deploy the pipeline with a bundle on a CLI older than v1.20.0. On v1.16.1 the deploy fails with a 400 and no pipeline is created, and I wouldn't count on 1.18 or 1.19 (Terraform engine, whose provider doesn't know group_name).

Summary

  • A pipeline running as an account group publishes tables owned by the team, and for an MV the group needed the grants from the matrix, without MODIFY.
  • I deploy a bundle with a group on CLI v1.20.0 or newer and check run_as_user_name after the deploy. The CLI schema marks group_name for pipelines as Private Preview, so I treat the bundle field as Preview.

Want more posts? Follow along via RSS or on LinkedIn.

Comments

Quiet on the trail so far. Be the first to comment.

Leave a comment

Your e-mail will not be published. Comments are moderated and appear once approved. Privacy policy