pipelines.pipelineId worked. Teardown scripts pass cascade explicitly. September also brought maintenance windows for continuous jobs.The problem
Most release-note items are about what we can write in code. The September changes in Lakeflow pipelines (in the docs: Spark Declarative Pipelines on Lakeflow) are about how pipelines live in production: how we delete them and when they restart.
The first one changes a default. Until now, deleting a Unity Catalog pipeline also deleted its materialized views and streaming tables. Since September 25, 2026 the tables stay by default. If a CI/CD or test-environment script cleans up by deleting the pipeline, after that date it leaves tables and data in the catalog that we keep paying storage for, and the next deployment hits names that are already taken.
1. Deleting a pipeline keeps its tables (Sep 25, 2026)
The docs put it this way: when you delete a Unity Catalog pipeline that uses the default publishing mode, its materialized views, streaming tables and views are retained. This applies to the Delete button in the UI and to DELETE in the Pipelines API. The old behaviour comes back with cascade=true in the API call. The flag existed before; on Sep 25 only its default changed. Hive metastore pipelines behave as before. We can check the publishing mode in the effective_publishing_mode field of GET /api/2.0/pipelines/{pipeline_id} (it should be DEFAULT_PUBLISHING_MODE). Pipelines created before February 5, 2025 may still use the legacy mode.
The first notebook creates two identical serverless pipelines, news104_del_a and news104_del_b, each with a streaming table *_orders and a materialized view *_total_by_country. api is WorkspaceClient().api_client, TABLES is a query on information_schema.tables limited to the notebook's tables, and wait waits for a pipeline update to finish. Pipeline A is deleted with a plain DELETE, the same call the UI button makes:
◆ Requires a full (Premium) workspace, Unity Catalog.
api.do("DELETE", f"/api/2.0/pipelines/{pid_a}")
time.sleep(20)
display(spark.sql(TABLES))
display(spark.sql("SELECT country, total FROM del_a_total_by_country ORDER BY country"))
Pipeline A is gone, but del_a_orders and del_a_total_by_country are still on the table list and can still be read. It is easy to conclude that nothing happened, so the next step adds one new order to the source and runs an update of pipeline B, which still exists:
◆ Requires a full (Premium) workspace, Unity Catalog.
spark.sql("INSERT INTO del_source VALUES (6, 'PL', 1000.00)")
print("news104_del_b:", wait(pid_b, api.do("POST", f"/api/2.0/pipelines/{pid_b}/updates", body={})["update_id"]))
display(spark.sql("""SELECT 'del_a_orders' AS table_name, COUNT(*) AS row_count FROM del_a_orders
UNION ALL SELECT 'del_b_orders', COUNT(*) FROM del_b_orders"""))
del_b_orders has 6 rows and del_a_orders still has 5. The table of the deleted pipeline looks alive but stands still, so a dashboard built on it shows old data and raises no error. Pipeline B is deleted with cascade=true:
◆ Requires a full (Premium) workspace, Unity Catalog.
api.do("DELETE", f"/api/2.0/pipelines/{pid_b}", query={"cascade": "true"})
time.sleep(20)
display(spark.sql(TABLES))
With cascade=true the pipeline takes del_b_orders and del_b_total_by_country with it, and only the A tables and the source table are left on the list.
Code for this article on GitHub →notebook and a step-by-step README · lakeflow-september-2026
2. Orphaned tables: how to find them and what to do with them
The docs say the retained tables can be queried but not updated until they are moved into an active pipeline, and point to Move tables between pipelines. That page, however, describes moving a table between two existing pipelines and says nothing about tables left by a deleted one. So the second notebook tests three things that come up in practice with this kind of cleanup. It creates pipeline news104_orph with tables orph_orders and orph_total_by_country, deletes it the default way, then finds the orphaned tables, deploys the same pipeline again and moves the tables.
Every pipeline table remembers the id of its pipeline (pipeline_id in Unity Catalog). An orphaned table is one whose pipeline no longer exists:
ⓘ According to the docs it works on Free Edition, I have not tested it.
alive = {p.pipeline_id for p in w.pipelines.list_pipelines()}
rows = [(t.name, t.table_type.value, t.owner, t.pipeline_id, t.pipeline_id in alive)
for t in w.tables.list(catalog, schema) if t.pipeline_id]
display(spark.createDataFrame(rows, "table_name string, table_type string, owner string, pipeline_id string, pipeline_exists boolean"))
Both tables return pipeline_exists = false and still point at the deleted pipeline's id. The owner stays the account that owned the pipeline. In the lab the owner and run-as were the same account, so I don't know which of them decides when run-as is a group or a service principal. We can run this cell on a schedule against schemas where pipelines come and go automatically. Catalog Explorer shows the difference too: for an orphaned table the "Current pipeline update status" field shows "Logs over 30 days old are archived…" instead of an update state, although the pipeline was deleted a moment ago.
Next we deploy the same code as a new pipeline news104_orph_new, which is exactly what CI/CD does after deleting and recreating an environment:
ⓘ According to the docs it works on Free Edition, I have not tested it.
pid_new, upd = create_pipeline("news104_orph_new", pipeline_code)
state = wait(pid_new, upd)
print("news104_orph_new:", state)
if state == "FAILED":
events = api.do("GET", f"/api/2.0/pipelines/{pid_new}/events", query={"filter": "level='ERROR'", "max_results": 3})
for e in events.get("events", []):
print("-", e["message"][:300])
The update ends FAILED with (shortened): "Table orph_total_by_country is already managed by pipeline <id of the deleted pipeline>. A table can only be owned by one pipeline." The deleted pipeline still holds its tables, so the new pipeline cannot take them over on its own. There are two ways out: DROP the tables and rebuild them from scratch, or move them with the pipelines.pipelineId property:
ⓘ According to the docs it works on Free Edition, I have not tested it.
for stmt in (f'ALTER STREAMING TABLE orph_orders SET TBLPROPERTIES("pipelines.pipelineId"="{pid_new}")',
f'ALTER MATERIALIZED VIEW orph_total_by_country SET TBLPROPERTIES("pipelines.pipelineId"="{pid_new}")'):
try:
spark.sql(stmt)
print(stmt.split(" SET")[0], "-> OK")
except Exception as e:
print(stmt.split(" SET")[0], "->", str(e).splitlines()[0][:300])
spark.sql("INSERT INTO orph_source VALUES (6, 'PL', 1000.00)")
print("news104_orph_new:", wait(pid_new, api.do("POST", f"/api/2.0/pipelines/{pid_new}/updates", body={})["update_id"]))
display(spark.sql("SELECT country, total FROM orph_total_by_country ORDER BY country"))
Both ALTER statements succeeded although the source pipeline no longer existed, and the new pipeline finished its update COMPLETED. The materialized view counted the new order (PL: 1200.50 instead of 200.50), and the streaming table did not reprocess the source, so it took over its checkpoint too. I ran the notebook as the owner and run-as of the new pipeline, with admin rights. The docs require the command to be run by the owner and run-as of both pipelines, so for another user the result may differ.
Deleting a pipeline in CI/CD
In production the new default is safer, because deleting a pipeline by accident no longer deletes data. In ephemeral environments (dev, test, per-branch) it is the other way round: the tables stay, use storage, and the next deployment with the same names fails with the error from the previous section. So we decide on cascade per environment and always pass it explicitly.
- The REST API reference ("By default, this parameter will be true"),
databricks pipelines delete --help(CLI 1.16.1 and the v1.23.0 code) and the Python SDK docstring still describe the old behaviour. In dev and test we pass?cascade=true,databricks pipelines delete <id> --cascadeorw.pipelines.delete(pipeline_id, cascade=True), and in production an explicitfalse. - In Declarative Automation Bundles (formerly Databricks Asset Bundles) the pipeline resource has, per the docs, a
cascade_on_destroyfield (CLI 1.12.0+). Without it,bundle destroyannounces that the tables will be deleted, but per the CLI v1.23.0 code it sendsDELETEwithoutcascade, and so do bundles on CLI 1.19 and older with the terraform engine, so the tables stay. - The Terraform provider (v1.141.0) has no equivalent of
cascade_on_destroyand deletes a pipeline withoutcascade, so afterterraform destroywe clean up the tables with an explicitDROP.
3. Maintenance windows for continuous jobs (Sep 10, 2026, Azure Sep 15, 2026)
A continuous pipeline never finishes, so for a runtime or infrastructure update Databricks restarts it: it starts a new update and stops the old one, and the new one resumes from the checkpoint. Until now such a restart could come at any time. Now we can choose a window: a day of the week, a start hour and a time zone. Maintenance starts within an hour of the window's start. The window is a setting of the continuous job that runs the pipeline; in the UI it is in Jobs (Continuous trigger → Advanced) and, per the docs, also in the pipeline (Schedule → Advanced settings).
The third notebook creates a PAUSED job that runs a pipeline, so nothing starts:
ⓘ According to the docs it works on Free Edition, I have not tested it.
window = {"start_hour": 2, "day_of_week": "SUNDAY", "timezone_id": "Europe/Warsaw"}
job_id = api.do("POST", "/api/2.2/jobs/create", body={
"name": "news104_mw_job",
"tasks": [{"task_key": "pipeline", "pipeline_task": {"pipeline_id": pid}}],
"continuous": {"pause_status": "PAUSED", "maintenance_window": window},
})["job_id"]
print(api.do("GET", "/api/2.2/jobs/get", query={"job_id": job_id})["settings"]["continuous"])
Jobs API 2.2 stored the window unchanged. In the UI it shows up in the job's trigger settings (Advanced → Maintenance window), converted to the viewer's time zone, and the pipeline's Schedule shows the continuous job that runs it. The docs require all three fields and an hour of 0-23, so I checked what happens when we break that:
ⓘ According to the docs it works on Free Edition, I have not tested it.
for label, bad in (("no timezone_id", {"start_hour": 2, "day_of_week": "SUNDAY"}),
("start_hour 24", {**window, "start_hour": 24})):
try:
api.do("POST", "/api/2.2/jobs/update", body={"job_id": job_id, "new_settings": {
"continuous": {"pause_status": "PAUSED", "maintenance_window": bad}}})
print(label, "-> accepted")
except Exception as e:
print(label, "->", str(e).splitlines()[0][:200])
The API rejects both with a clear message: Missing required field: new_settings.continuous.maintenance_window.timezone_id and Invalid value for field jobs.MaintenanceWindow.start_hour: Value must be less than or equal to 23. I did not observe a restart in the window, because the job was paused.
Things to watch:
- Serverless has a maximum cluster lifetime. If the limit comes before the next window, Databricks restarts the pipeline outside the window anyway, and the page on windows does not say how long the limit is.
- The window is computed in the given time zone, so on the day the clocks change the start may shift by an hour.
- The job can send Maintenance start and Maintenance complete notifications (email, Slack, Teams, PagerDuty).
- The window only works for continuous jobs. In bundles it is
resources.jobs.<job>.continuous.maintenance_window(CLI 1.16.0+), and the Terraform provider does not have this field, so a window set by hand in the UI will not end up in code.
Step by step: the lab demo
The code is in three small notebooks (link under the first video), and each one creates what it needs and cleans up after itself. They do everything through the REST API from the notebook, on serverless, and write the pipeline code to workspace files themselves. I ran them as an admin in an Azure Databricks Premium (trial) workspace, so I did not work out the minimum set of privileges. The docs require USE CATALOG on the catalog, USE SCHEMA + CREATE TABLE and USE SCHEMA + CREATE MATERIALIZED VIEW on the schema, and CREATE SCHEMA if the schema has to be created. On top of that the notebooks need permission to write workspace files and to create pipelines and jobs. According to the docs the first notebook will not run on Free Edition, because it runs two pipelines at once and Free Edition allows one active pipeline per type (I have not tested it). Each serverless pipeline update took about a minute, so the notebooks with pipelines take 3-5 minutes.
Pitfalls
- An orphaned table looks alive. It can be read, so the dashboard works, only the data stands still. In the lab, after a new order the live pipeline's table had 6 rows and the orphaned one still 5. The query from section 2, which compares the tables'
pipeline_idwith the list of existing pipelines, helps. - Redeploying after deleting a pipeline without
cascadefails with "already managed by pipeline", because the tables still belong to the deleted pipeline. In CI/CD this shows up only on the next deployment, often for someone else. - Moving with
pipelines.pipelineIdworked in the lab for the tables of a deleted pipeline, but the docs only describe two existing pipelines and require the command to be run by the owner and run-as of both. I tested it as an admin, so pipelines whose run-as is a service principal or a group need a separate check (Lakeflow: a pipeline that doesn't belong to one person). cascade=truedoes not clean up everything at once. In the run of Oct 15, internal__materialization_mat_*tables were still ininformation_schema.tablesmore than 2 minutes after the pipeline was deleted. A script that compares the table list of a schema has to expect that.- The event log disappears with the pipeline. In the Oct 15 lab run the deleted pipeline's event log disappeared, although its MVs and STs stayed. If we need it for audit, we copy it before deleting the pipeline or on a schedule.
When NOT to use it
- The default delete behaviour in ephemeral environments. Use an explicit
cascade=truethere, otherwise the next deployment hits taken names. cascade=truein production. An explicitfalsethere protects data from an accidental pipeline delete.- Maintenance windows when the pipeline does not need to run 24/7. A triggered pipeline on a schedule restarts on every run anyway, and the setting does not exist there.
Summary
- Since Sep 25, 2026, deleting a UC pipeline keeps its tables, which can be read but no longer get new data. In CI/CD we always pass
cascadeexplicitly and per environment, because the REST, CLI and SDK help still describe the old default. - Orphaned tables can be found by a
pipeline_idthat points at a pipeline that no longer exists. Redeploying onto those names fails, and in the lab moving them withpipelines.pipelineIdto the new pipeline worked. - The maintenance window (
continuous.maintenance_window) is set on the job that runs the continuous pipeline. On serverless a restart may still come outside the window.
Want more posts? Follow along via RSS or on LinkedIn.



Comments
Quiet on the trail so far. Be the first to comment.