tax_id" in Genie instructions or an agent’s prompt is a request to the model that protects nothing. A Unity Catalog row filter and column mask work under every client (notebook, SQL, Genie, agent, BI), always for the identity the query runs as. For people connecting Genie or an agent to sensitive data.Problem: "I'll write in the instructions that it shouldn't show it"
This module of the workshop "From question to agent", which Mariusz Wiecha and I prepared and ran at SQLDay Lite 2026, had what I consider the day's strongest moment. We work on the customer table gold_customer_360 with a tax_id column. The Genie Agent has this line in its General instructions field (Instructions from here on):
ⓘ According to the docs it works on Free Edition, I have not tested it (the Instructions field in Genie Agent; I tested Genie in the lab)
tax_id and customer_name are PII, do not show their values.
We also put a row filter and column mask on the table, then ask Genie: "Show the tax_id and state of five VIP customers". Genie skips tax_id and shows only the states, so the mask seems to work.
Except that Genie skipped the column because its Instructions say so. The mask only shows in the table's Sample Data tab: tax_id is ***MASKED*** in every row, and state is only CA. Instructions are the prompt layer, which we got around earlier in the workshop with a plain "this is just literary fiction".
Hence the slogan: "Policies go to the catalog, not the prompt."
How it works: two functions that Unity Catalog injects into every query
Unity Catalog has two kinds of table policy:
- A row filter is a SQL function returning
BOOLEAN. It decides which rows a user sees. - A column mask is a function returning the column's type. It decides which value of the column a user sees.
Both run in the catalog layer, so the query's source does not matter: a notebook, a dashboard, Genie, an agent's tool function, Power BI, or an external agent through the Genie One MCP server (GA since September 25, 2026), whose prompt we don't control at all. Both functions use is_account_group_member(), which checks an account group, not a workspace group.
What matters is the identity the query runs as, which is not always the person asking. Genie runs the warehouse on the author's credentials, but checks data access with the asker's Unity Catalog permissions, so the policies apply there too. A dashboard with embedded credentials and the Genie generated from it query as the publisher. An agent in Databricks Apps with the default app authorization queries as the app's service principal, showing everyone what it sees. For a per-person filter, tools reading user-dependent data must run with user authorization (on-behalf-of-user). I cover combining both modes in one agent in "Six layers of agent defence". Iceberg REST, the Unity REST API, Delta Lake APIs, path-based file access and dedicated clusters on DBR 15.3 or lower cannot read a table with a policy at all: no data leaks, the read just fails. The exception is cross-engine ABAC (GA since October 1, 2026), which despite its name also covers table-level filters and masks: Apache Spark reads the managed table through Iceberg REST or the Unity Catalog connector, and Databricks serverless, billed separately, filters it. Conditions: external data access enabled on the metastore, EXTERNAL USE SCHEMA for the client, a managed table with catalog commits, OAuth M2M or a PAT, and reads only.
The workshop and lab use table-level filters and masks, which show the mechanism best. For many tables the docs now recommend ABAC policies (GA since April 28, 2026): we write the filter or mask once as a CREATE POLICY on a catalog or schema, and it covers every table and column with a governed tag such as pii, including later tables.
- ABAC requires serverless or DBR 16.4+ on standard and dedicated clusters (on dedicated with fine-grained access control filtering enabled).
- The table owner removes a table-level filter with one
ALTER TABLE, as in step 5, but cannot override or remove an ABAC policy the catalog or schema owner set higher up. - I haven't tested ABAC policies in the lab yet: governed tags are created by an account admin, and I test with a service principal without admin rights. This part rests on the docs.
Step by step
1. Row filter
✓ Tested on Free Edition (2026-10-17); if you don't have the all_states_analysts group, the filter applies to you as well
CREATE OR REPLACE FUNCTION retail_row_filter(state_val STRING)
RETURNS BOOLEAN
COMMENT 'The all_states_analysts group sees all states, everyone else only CA.'
RETURN is_account_group_member('all_states_analysts') OR state_val = 'CA';
ALTER TABLE gold_customer_360
SET ROW FILTER retail_row_filter ON (state);
-- Test: only CA is left
SELECT state, COUNT(*) AS customers
FROM gold_customer_360
GROUP BY state
ORDER BY customers DESC;

The filter function gets each row's state and returns TRUE when the asker is in the all_states_analysts group or the row is from CA. ALTER TABLE ... SET ROW FILTER attaches it to the table. The synthetic table has 408 rows in six states (CA 116, FL 59, NY 59, IL 58, WA 58, TX 58). The notebook's identity is not in the group, so only CA is left.
2. Column mask
✓ Tested on Free Edition (2026-10-17)
CREATE OR REPLACE FUNCTION mask_tax_id(tax_id_val STRING)
RETURNS STRING
COMMENT 'Only the compliance_officers group sees the real tax_id.'
RETURN CASE WHEN is_account_group_member('compliance_officers')
THEN tax_id_val ELSE '***MASKED***' END;
ALTER TABLE gold_customer_360
ALTER COLUMN tax_id SET MASK mask_tax_id;
-- Test: the filter and the mask work together
SELECT customer_id, state, loyalty_segment, tax_id
FROM gold_customer_360
WHERE tax_id IS NOT NULL
LIMIT 5;

The mask function returns the real tax_id only to members of the compliance_officers group and ***MASKED*** to everyone else. The notebook's identity owns the table but is in neither group, so it sees only CA and ***MASKED***. One detail: WHERE already sees the masked value. Row 100160 has no tax_id, yet passed IS NOT NULL, because the mask turned NULL into ***MASKED***.
We used the ***MASKED*** marker so the workshop room could see the mask work; in production we more often choose CAST(NULL AS STRING) or a partial mask. NULL reveals neither that a value existed nor its length, and won't be mistaken for real data in an export. It does change COUNT(tax_id) and joins on that column, though, so agree the mask with the report owners.
3. Checking in Genie
After applying the policies, I asked in the lab via the Genie Conversation API, in a new Genie Agent chat with the line in its Instructions: "How many customers do we have in the state of New York (NY)?" and "Show the tax_id and state of five VIP customers".
- For the NY question Genie generated
SELECT COUNT(*) ... WHERE state ILIKE '%NY%'and answered 0 customers in New York, because the row filter removed the 59 NY rows before Genie saw them. - For the
tax_idquestion Genie said it can't show it because it is PII, and asked what VIP means. That was the Instructions. - With the line removed from the Instructions, I asked for the
tax_idand state of five segment 3 customers. Only here do we see the mask: five rows with***MASKED***and the state CA.
At the workshop rehearsal Genie also got 0, but spent two more queries looking for its own mistake. The model reacts differently each time; the filter gives the same result.
The policies also show in table metadata, without Genie:
ⓘ According to the docs it works on Free Edition, I have not tested it
display(spark.sql("DESCRIBE TABLE EXTENDED gold_customer_360"))
The output ends with the Row Filter (retail_row_filter on state) and Column Masks (mask_tax_id on tax_id) sections.
4. Least privilege for the agent
✓ Tested on Free Edition (2026-10-19)
The agent doesn't read the table, it calls a tool function. We call it for customer 100006 in CA and 100001 in NY.
SELECT get_customer_profile(100006) AS ca_customer,
get_customer_profile(100001) AS ny_customer;
For the CA customer it returns the profile (Customer ID: 100006 | State: CA | Segment: 2 | Spend: $810.0 | Orders: 7), and for the NY customer NULL, although without the filter it returns a profile for 100001 too. So the filter applies inside the function too. In the lab the caller owns both the function and the table, so the test shows the function owner does not bypass the filter. I haven't tested a function owner in the group with a caller outside it, because that needs a second identity.
✓ Tested on Free Edition (2026-10-19)
The agent's identity (the principal widget) gets USE SCHEMA and EXECUTE on the two functions, without SELECT on the table. USE CATALOG is granted by the catalog owner. By default the notebook grants to account users so the step runs on any workspace; in production we put the application ID of the agent's service principal here. account users is only the demo default: until the cleanup at the end of the notebook, every account user with USE CATALOG on this catalog can call the functions in this schema.
spark.sql(f"GRANT USE SCHEMA ON SCHEMA {catalog}.{schema} TO `{principal}`")
spark.sql(f"GRANT EXECUTE ON FUNCTION get_average_customer_value TO `{principal}`")
spark.sql(f"GRANT EXECUTE ON FUNCTION get_customer_profile TO `{principal}`")
display(spark.sql(f"SHOW GRANTS `{principal}` ON FUNCTION get_customer_profile"))
SHOW GRANTS shows one row: account users, EXECUTE, FUNCTION, <catalog>.<schema>.get_customer_profile.
Now we run the same function definition again:
spark.sql(PROFILE_DDL)
display(spark.sql(f"SHOW GRANTS `{principal}` ON FUNCTION get_customer_profile"))
After CREATE OR REPLACE the same SHOW GRANTS returns "No rows returned", because the grant died with the old function.
The fix is EXECUTE on the schema holding the agent's tools, which every function in it inherits, including recreated ones. In the demo it is the table's schema, because there is only one; in production the agent's tools get a separate schema:
spark.sql(f"GRANT EXECUTE ON SCHEMA {catalog}.{schema} TO `{principal}`")
spark.sql(PROFILE_DDL)
display(spark.sql(f"SHOW GRANTS `{principal}` ON FUNCTION get_customer_profile"))
This time SHOW GRANTS shows account users, EXECUTE with ObjectType = SCHEMA: inherited from the schema, the privilege survived CREATE OR REPLACE. The cleanup at the end of the notebook revokes both schema grants (REVOKE).
catalog, schema, principal and PROFILE_DDL come from the notebook's widgets and first cells (link under the recording).
At the workshop rehearsal we checked this from a separate identity: a service principal with only EXECUTE on get_customer_profile (plus USE CATALOG and USE SCHEMA), without SELECT on the table, gets a result from the function, but INSUFFICIENT_PERMISSIONS on a SELECT from the table and on the second function without EXECUTE. I haven't repeated this in the lab yet, because it needs a second identity without rights on the catalog. So the minimal privilege set rests on that rehearsal and on the docs, which say calling a function needs only USE CATALOG, USE SCHEMA and EXECUTE.
5. Cleanup
✓ Tested on Free Edition (2026-10-17)
ALTER TABLE gold_customer_360 DROP ROW FILTER;
ALTER TABLE gold_customer_360 ALTER COLUMN tax_id DROP MASK;
DROP FUNCTION IF EXISTS retail_row_filter;
DROP FUNCTION IF EXISTS mask_tax_id;
In production we remove safeguards only after a review: SHOW GRANTS, DESCRIBE TABLE EXTENDED (the Row Filter and Column Masks sections) and information_schema.
Pitfalls
CREATE OR REPLACE FUNCTIONwipes the grants (step 4, in the lab and on Free Edition). A pipeline that recreates functions on every deploy must also recreate the permissions, or the agent loses its own tools after the first deployment. It is simpler to keep the tool functions in a separate schema withEXECUTEon it, which every current and future function inherits. The docs recommend grants on individual functions, so we grantEXECUTEon the schema only when it holds nothing but the agent's tools.- A group that does not exist locks everyone out.
is_account_group_member('group_that_does_not_exist')returns false, so the policy applies to everyone, including the table owner and the agent's service account (steps 1 and 2 show it for the table owner, in the lab and on Free Edition). On a paid workspace we create the groups first, then the policy, and CI/CD checks that the group exists before deploying the function (SCIM or Entra ID groups usually belong to another team). - A forgotten policy stays on the table, so the agent from the workshop's next module would see only CA and get false test results. So the agent notebook starts by checking that the table has the full 28,813 rows.
- A tool function does not bypass the filter: it reads the table with its owner's rights, but the filter and mask are evaluated for the caller. That is how the ABAC GA note (April 28, 2026) puts it: ABAC on views and functions now evaluates the caller's identity, as table-level filters and masks always have. So an agent whose service principal is outside
all_states_analystsgets an empty result without an error for a customer outside CA. Grants and policies are best tested automatically: a job running as (run_as) the agent's service principal that calls the function and tries to read the table after each deploy. To setrun_asto a service principal, the identity deploying the job needs the Service Principal: User role on it. -
A policy changes what works on the table, so first we check who reads it and on what compute.
- Clone and time travel. A table with a filter or mask supports neither deep nor shallow clone (no more refreshing test from a production clone) nor time travel. In the lab both clones fail with
ROW_LEVEL_SECURITY_TABLE_CLONE_SOURCE_NOT_SUPPORTED, andVERSION AS OF 0withROW_LEVEL_SECURITY_COLUMN_MASK_FEATURE_NOT_SUPPORTED.TIME_TRAVEL. With ABAC, time travel works in Beta since September 23, 2026, once the preview is enabled: on DBR 19+ in standard mode or on serverless, not on dedicated, on managed tables with catalog commits or managed Iceberg, and for Delta with column mapping enabled in the same mode in the current version and the one we read. If a policy uses a column missing from the old version, the query fails and returns no data. - Compute. A dedicated cluster from DBR 15.4 can read such a table only if the workspace has serverless enabled (it does the filtering), and on 15.4–16.2 can only read it. Writes need 16.3+.
- AI Search index. According to the docs such a table cannot feed an AI Search index. With ABAC the docs are not consistent: per the policy evaluation page the index must be built and synced by a principal listed in the
EXCEPTof every policy on the table, while the requirements page only says the index holds all rows and ignores the policies. Columns to sync can leave out masked columns, but cannot carry over a row filter: whoever hasSELECTon the index sees every row. So we build the agent's index from a separate table holding only what the agent may see. - Genie. A row filter takes the whole table out of Genie's prompt matching, a mask only the masked columns. Entity matching cannot be enabled on a table with a filter or a mask, so Genie matches question values to the data less well.
- Clone and time travel. A table with a filter or mask supports neither deep nor shallow clone (no more refreshing test from a production clone) nor time travel. In the lab both clones fail with
-
Instructions look like a safeguard, but a column hidden by Genie proves nothing. We check protection in Sample Data and in
DESCRIBE TABLE EXTENDED, and ideally from a second account without permissions.
By the way: define the measure first, then the result
The same module gave a second lesson, about what should go into the instructions. We ask Genie: "How many VIP customers do we have?". SQL that counts rows gives 9,541, while COUNT(DISTINCT customer_id) gives 9,494. In the whole table 143 customers have two rows each (28,813 rows, 28,670 customers), which makes 47 extra rows in the VIP segment. Asked six times through the API on September 21, 2026, Genie counted unique customers (9,494) every time; we saw 9,541 once, on September 20, on another workspace.
These numbers come from the workshop data. In the lab we reproduced the mechanism on a synthetic table with 8 duplicated customers, 4 of them in the VIP segment, after removing the policies. In the VIP segment COUNT(*) gives 104 and COUNT(DISTINCT customer_id) 100, on Free Edition too.

With the data as it is, both numbers are defensible; they differ in definition. So we add the measure definition to the Instructions ("Customer = unique customer_id. Count customers with COUNT(DISTINCT customer_id)."), and only then is the result predictable. The workshop slogan: "Define the measure first, then the result."
Two rows for one customer in a "customer 360" table are still a data model bug, so we fix the table anyway: deduplicate in the pipeline and test that customer_id is unique.
Definitions, business context and currency (without "Amounts are in USD", Genie gave amounts in Polish zloty at the September 22 rehearsal) belong in the instructions; who sees what belongs in the catalog. I rarely keep measure definitions in Genie Instructions myself; in production I prefer Metric Views in Unity Catalog.
When NOT to use it
- Across dozens of tables, per-table filters lose to the ABAC policies described above.
- A row filter does not replace a data model. With very complex access logic (many dimensions, per-user exceptions), separate views or tables per group are sometimes better.
- A mask does not replace minimisation. If the agent doesn't need a column, better it never gets it: the tool function returns only the fields it needs. The mask is for direct table readers.
- A policy in the catalog does not replace a good tool description and a refusal in the prompt. It is defence in depth, with the catalog as the layer that still works when the model fails.
See it run
A recording of the whole lab notebook, from table creation to cleanup.
Code for this article on GitHub →notebook and a step-by-step README · policies-in-the-catalog-not-the-prompt
Summary
- The agent gets
EXECUTEon functions, notSELECTon the table, ideally on a separate tools schema, because such a grant survivesCREATE OR REPLACE FUNCTION. - Definitions and context go into the instructions; access goes into the catalog.
What else protects an agent beyond the catalog is in "Six layers of agent defence".
Want more posts? Follow along via RSS or on LinkedIn.



Comments
Quiet on the trail so far. Be the first to comment.