A follow-up to
[Part 2: Databricks and FSx for ONTAP S3 Access Points]in the "FSx for ONTAP S3 Access Points x Lakehouse Deep Dive" series. Some results changed since Part 2 — if you read that one, see "Why reads fail" below.
I tested whether images and PDFs already sitting on FSx for ONTAP can be referenced from
Databricks without copying them into the lakehouse. The result is: registration works, reads
do not.
If you only need the zero-copy verdict, read "Connecting to an FSx for ONTAP S3 Access Point"
and "Design tips" below. If you want the behaviour of the FILE
type itself, start from the
top.
Databricks shipped FILE
as a beta column type: a reference to an unstructured file held as a
single column in a Delta table. The value is a struct carrying uri
/ offset
/ size
/
content_type
/ checksum
. Because it holds the reference and the metadata as a struct, you
can pass it straight from a table into an AI function such as ai_parse_document
. There are
two variants — FILE EXTERNAL
, which holds only the reference, and FILE MANAGED
, which
copies the bytes into Databricks.
I tested this against files on Amazon FSx for NetApp ONTAP, reached through an FSx for ONTAP S3
Access Point — the feature that exposes an ONTAP volume over an S3-compatible API.
Here is what I found:
FILE EXTERNAL
and
FILE MANAGED
before you ingestGROUP BY
and DISTINCT
are accepted; =
and
ORDER BY
are rejectedBelow: what is established for each of those four, and how to turn it into design decisions.
The templates and scripts are in the repository, so you can re-run the same tests in your own
account.
Repository: Yoshiki0705/fsxn-lakehouse-integrations
| Item | Value |
|---|---|
| Measured | 2026-08-12 |
| Storage | One FSx for ONTAP file system |
| Access Point | One, INTERNET origin, UNIX root identity |
| Workspace | One non-trial workspace, purpose-built, same account and region |
| Compute | Serverless SQL warehouse, and a classic DBR 18.2 cluster |
| Control | A S3 general bucket in the same account, running the same operations alongside |
| Method | SQL Statement Execution API, and boto3 from my workstation |
Running the S3 general bucket control alongside matters. Looking only at the Access Point side, you
cannot separate a defect in your own environment from behaviour specific to the Access Point.
Except where stated otherwise, every conclusion below rests on the same operation succeeding
against S3 general bucket.
Note that the object tag results depend on the ONTAP tag validation implementation, so a
different ONTAP version may behave differently. The FSx console and describe-file-systems
do
not expose the version; use the ONTAP REST API (GET /api/cluster?fields=version
) to check
yours.
First, the behaviour of the type itself. There is prior work on this feature from Databricks
Japan: パス文字列でもバイナリでもない。DatabricksのFILE型を試す
by taka_yayoi. Every operational caveat it lists reproduced in my environment, so I have
tabulated those first. (What follows is my paraphrase and my own measurements, not a translation.)
Some terms first. FILE MANAGED
copies bytes into a Databricks-side location called a
FileSpace. list_files
enumerates files on a Volume and returns FILE values; create_file
constructs a FILE value explicitly. _object_metadata
reads an object's tags and user
metadata.
| Behaviour | Measured | Design implication |
|---|---|---|
| FileSpace on the same volume as the source | ||
CREATE TABLE succeeds; INSERT fails with Cannot get file metadata under managed storage |
||
| Give the FileSpace its own volume | ||
checksum on FILE values from list_files |
||
| Null on every row | Not usable for integrity checks | |
checksum under FILE MANAGED |
||
Populated with ETAG:"…" , matching the etag from _object_metadata on S3 general bucket exactly as a string |
||
| Comes from the object store, so it is usable for reconciliation | ||
Filenames under FILE MANAGED |
||
| Replaced with an opaque UUID; the extension does not survive either | Copy the name to a column before ingesting | |
Format of uri |
||
Prefixed with dbfs: |
||
| Normalise before writing string comparisons | ||
| Serverless notebooks | Unsupported | Use the SQL Statement Execution API against a serverless SQL warehouse |
| Automatic garbage collection in beta | Does not run | The FileSpace grows on every re-run. Plan for manual deletion |
| The Previews page toggle | Used FILE EXTERNAL / FILE MANAGED without touching it (confirmed on 2 workspaces) |
|
| May not be required. Do not assume either way | ||
content_type from create_file |
||
binary/octet-stream for both Japanese and ASCII |
||
| Pass it explicitly if anything downstream branches on it |
What this tells us
The difference between FILE EXTERNAL
and FILE MANAGED
is not only whether bytes are copied.
Whether checksum
is populated and whether the filename survives both change. Reversing the
decision later means re-ingesting, so decide before you ingest.
On content_type
, I got binary/octet-stream
for both Japanese and ASCII input. For the same
files, list_files
reported text/plain
, so the two paths infer differently. If anything
downstream branches on content_type
, pass it explicitly.
Both the documentation and the prior article state that FILE columns cannot be used in grouping
expressions. Testing it, support splits more finely than that.
| Operation | Result |
|---|---|
GROUP BY file |
|
| accepted | |
SELECT DISTINCT file |
|
| accepted | |
GROUP BY file.uri |
|
| accepted (the documented approach) | |
d.file = r.attachment |
|
rejected — The = does not support ordering on type "FILE" |
|
ORDER BY file |
|
rejected — The sortorder does not support ordering on type "FILE EXTERNAL" |
|
What this tells us
GROUP BY
and DISTINCT
both need equality semantics to decide group membership. Both are
accepted while the equality operator itself is explicitly rejected. Since the error names the
type's operator surface, this does not look like row-count-dependent behaviour.
That said, this was confirmed on two rows. I cannot rule out that grouping degrades to identity
comparison rather than value equality. In practice, use GROUP BY file.uri
and do not depend on
GROUP BY file
being accepted. This is beta, so the behaviour may change.
This is the main question. The answer is "registration works, reads do not", and the cause is
not a lack of Access Point support.
| Item | Required form | Symptom when wrong |
|---|---|---|
| External ID in the trust policy | The Databricks account UUID | |
403 Forbidden from the storage provider. The metastore ID and the workspace ID do not work |
||
| Self-assume in the trust policy | Account root as the principal, with an aws:PrincipalArn condition naming the role ARN |
|
Naming the role as its own principal fails at creation with Invalid principal , because IAM validates that the principal exists. Unity Catalog requires the role to be able to assume itself |
||
| Resource in the IAM permission policy | The access point ARN (arn:aws:s3:<region>:<account>:accesspoint/<name> and .../object/* ) |
|
AccessDeniedException . The alias-as-bucket-name form alone does not work |
||
| External location URL | The alias form (s3://<alias>/ ) |
|
An ARN-style URL is rejected immediately with url does not specify a valid bucket name |
||
What this tells us
There is an asymmetry worth committing to memory: the same Access Point must be written as an
ARN in the IAM policy and as an alias in the external location URL. The AWS CLI works with the
alias form of the ARN, so verifying connectivity with the CLI first and then writing the IAM
policy will trip you up.
The aws:PrincipalArn
approach works because that key resolves to the role ARN rather than the
assumed-role session ARN. Rather than trust my memory of the condition key, I deployed the
template, assumed the role, and assumed it again with those credentials to confirm.
With all four satisfied, CREATE EXTERNAL LOCATION
succeeds with skip_validation=False
—
meaning Unity Catalog's own validation passed. The external volume on top can be created too.
After registration, every read path returns 403 or an authorisation error. The same on
serverless SQL and on a classic DBR 18.2 cluster.
This is the test that isolated the cause. Unity Catalog will vend the temporary credentials it
uses if you ask for them. I requested them for both paths and used them from my workstation,
with no Databricks compute or network involved.
| Vended for | ListObjectsV2 |
HeadObject |
|---|---|---|
| S3 general bucket control path | success | success |
| FSx for ONTAP S3 AP path | AccessDenied | 403 |
Same role, same session, same network; the only variable is which path the credentials were
scoped to. The error states the cause explicitly:
is not authorized to perform: s3:ListBucket on resource:
"arn:aws:s3:<region>:<account>:accesspoint/<name>"
because no session policy allows the s3:ListBucket action
What this tells us
because no session policy allows
is the decisive part. What is named is the session policy,
not the role's permissions. By session policy I mean the down-scoped policy passed to
AssumeRole
to constrain the role's permissions on a per-request basis.
The policy body is generated by Unity Catalog and is not readable by the user. But since adding
an access-point-ARN allowance to the role changes nothing, the session policy appears not to mention the access point ARN resource form at all. Only the overlap between the session policy
There is no workaround on the user side. Unity Catalog generates the policy, so the fix has
to happen there. Emitting the access point ARN form in the session policy should resolve it,
though I do not know what implementation constraints apply.
Note also that validation at registration time (skip_validation=False
) passes while reads
return 403. What that validation checks is not published, so I cannot account for the
difference.
Part 2 of this series (measured 2026-05) reported that listing top-level files and reading an
explicit file did work through a UC external location. In this run (2026-08-12), read_files
,
list_files
, to_file
and dbutils.fs.ls
all returned 403; no read path partially succeeded.
That environment is gone, so I cannot account for the difference conclusively. The likely
explanation is that the earlier partial successes were observed on an instance-profile path
rather than through UC-vended credentials — Part 2 lists instance-profile direct access as a
separate approach that works but sits outside UC governance, so the paths may not have been
cleanly separated.
This run had a S3 general bucket control alongside and includes the vended-credential test outside
Databricks entirely, so take these results as the current ones. I will annotate the Part 2
table accordingly.
This test has prerequisites. Enable External Data Access on the metastore and grant
EXTERNAL USE SCHEMA
and EXTERNAL USE LOCATION
. Both are disabled by default. They are the
controls that allow credentials to be used externally, so they are worth understanding
independently of this issue.
Arranged so that anyone hitting the same 403 can check top to bottom. None of these is the cause.
| What to suspect | Finding |
|---|---|
| Unity Catalog volume privileges | The failing volume had READ VOLUME ; the working control volume had no privileges at all |
| IAM role permissions | Assumed the same role from my workstation and listed successfully. HEAD on a real object returned 200 |
HEAD on a missing key returning 403 (killing the _delta_log probe) |
|
| The Access Point returned 404 for every shape I tried — same as S3 general bucket | |
x-amz-expected-bucket-owner being rejected |
|
| Accepted by both the Access Point and the control bucket | |
| A different endpoint | Both Databricks and the AWS CLI use <alias>.s3.<region>.amazonaws.com , which CNAMEs to s3-r-w.<region>.amazonaws.com . Same host |
| Insufficient compute role permissions | |
sts:GetCallerIdentity raised NoCredentialsError on the driver. The cluster has no default AWS credentials; the compute role is not in the path |
|
| The S3 gateway endpoint | No change after disassociating the route. The failure reproduces outside the VPC with vended credentials, so this path is not the problem |
One note on the last row. A Databricks-managed VPC creates an S3 gateway endpoint at workspace
creation, and the private route tables point the S3 managed prefix list at it. That was not the
cause here, but for designs that reach an S3 AP from inside a VPC, check first whether the
alias resolves into the prefix list's range. I did not reconcile the resolved CIDRs against the
prefix list contents in this environment.
If you want metadata to live with the file, object tags are one option: tag at write time, read
from the table side later. That path has constraints.
The following was measured without Databricks in the picture — boto3 from my workstation
straight against the FSx for ONTAP S3 Access Point, with the same operations running against
the S3 general bucket control bucket.
Start with what works. PutObjectTagging
, GetObjectTagging
and DeleteObjectTagging
all
function, as do x-amz-meta-*
headers and x-amz-tagging
on the same PutObject
as the data.
The limits (10 tags per object, 128 characters for a key, 256 for a value) are the same values
S3 general bucket documents; I did not measure the boundaries.
Reading through a second access point on the same volume returned tags written through the
first. Tag retention is not scoped per access point. For existing NAS assets shared across
several paths, that property works in your favour. But I did not observe where the tags are
physically stored, so visibility from NFS or SMB, and retention across SnapMirror, FlexClone
and Snapshot restore, remain unverified.
Two constraints change how you design.
An object overwrite clears tags and user metadata. Nothing errors. A pipeline that rewrites
a file has to re-apply them in the same PutObject
. My understanding is that S3 general bucket behaves
the same way, since PutObject
replaces the object, so treat this as a general S3 design point
rather than something specific to FSx for ONTAP. I did not run a side-by-side control for this
particular case.
Tag values are effectively ASCII. Every printable Latin-1 character I tried
(U+00A1–U+00FF) was accepted. Above U+0100, most are rejected with InvalidTag
— but not all.
A minority of CJK strings are accepted, deterministically, 6 runs out of 6.
To find the unit of validation, I tested single characters:
| Input | Result |
|---|---|
| U+5206 | InvalidTag |
| U+985E | InvalidTag |
| U+5206 U+985E | accepted |
| U+6771 | InvalidTag |
| U+4EAC | InvalidTag |
| U+6771 U+4EAC | InvalidTag |
What this tells us
Two two-character strings, every constituent character rejected on its own, and the two strings
disagree with each other. Validity is determined by the complete byte sequence, not as a function of the characters in it — so this is not a per-character allowlist.
For the two-character pairs I tested, results were identical forwards and backwards, and
identical as a tag key and as a tag value. I did not test longer strings or other scripts.
I raised this with AWS Support. After checking it against the S3 and FSx for ONTAP
documentation, they gave the view that the pattern does not match any intentional validation
they could identify, and escalated it to the service team as a potential defect in the tag
validation layer.
Their reply also quoted the documented character set: letters, whitespace, and
+ - = . _ : / @
. That does not predict the split under either reading. If "letters" means
Unicode letters, every string I tested qualifies yet half are rejected. If it means ASCII
letters, all of them should be rejected yet half are accepted.
The design guidance is simple: keep tags ASCII and keep localised text in a column. Today
the situation is "some strings pass and some do not", which is more awkward to handle than a
uniform restriction. I will update the repository and this article when a determination
arrives.
The same material, ordered the way you actually make the decisions.
FILE type
FILE EXTERNAL
and FILE MANAGED
INSERT
fails — and because the error surfaces at ingest rather than at configuration time, diagnosis is slowerFILE MANAGED
. It is replaced with a UUID and the extension does not survivecontent_type
inference; pass it explicitlyfile.uri
. Do not depend on GROUP BY file
being acceptedUnity Catalog and FSx for ONTAP S3 Access Points
aws:PrincipalArn
condition naming the role ARN.../object/*
. The alias-form ARN alone does not workintegrations/athena
, integrations/glue
, integrations/emr-spark
, integrations/redshift-spectrum
and integrations/lake-formation
)Object tags
PutObject
when you rewrite a file. An overwrite clears themHow to test
All of the above is in the repository as templates and scripts: one template, one script, one
runbook.
| Artefact | Role |
|---|---|
integrations/databricks/uc-storage-credential-role.yaml |
|
| The IAM role the Unity Catalog storage credential assumes, plus a S3 general bucket control bucket | |
shared/scripts/probe_uc_external_location.py |
|
Seeds identical object tags on both sides, registers both external locations with validation on, reads _object_metadata through each, prints a verdict |
|
docs/en/databricks-verification-runbook.md |
|
| Prerequisites, where each parameter comes from, the verdict branches, dependency-ordered teardown, measured cost |
python3 shared/scripts/audit_databricks_workspace_footprint.py \
--region <region> --save /tmp/baseline.json
aws cloudformation deploy --region <region> \
--stack-name fsxn-databricks-uc-credential \
--template-file integrations/databricks/uc-storage-credential-role.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides file://cfn-params/databricks-uc-storage-credential.json
.venv/bin/python shared/scripts/probe_uc_external_location.py \
--profile <your-profile> --role-arn <from stack output> \
--ap-alias <alias>-ext-s3alias --ap-name <name> \
--control-bucket <from stack output> --region <region> --vend-check
python3 shared/scripts/audit_databricks_workspace_footprint.py \
--region <region> --compare /tmp/baseline.json
The script prints one of four verdicts. The first is the important one.
One note on the parameter file: a _comment
key in a CloudFormation parameter file makes the
CLI fail with Unknown parameter in Parameters[n]
. The description of each parameter lives in
cfn-params/README.md
instead, so fill in values with that open alongside.
The specific bug is Databricks'. The shape is not, and the shape is the part worth taking away
if you are looking at a different product.
Managed lakehouse platforms generally reach into your cloud account by assuming a role you
create, then constrain that role at runtime with a down-scoped session policy so that
credentials handed to a query cannot roam. That is good design.
But the policy has to name your storage. If the storage is addressed through an access point
while the policy names it in bucket-form ARNs, the two do not match and authorisation fails —
however correct your own role policy is.
So the question to put to a platform you are evaluating, before you commit to an architecture,
can be narrow and answerable:
When you vend temporary credentials for an external location, which resource ARN forms does
the session policy contain? Specifically, does it include the S3 access point ARN form in
addition to the bucket form?
And if the platform exposes credential vending, you can verify it without asking anyone:
request the credentials, use them from your own machine, and compare a native bucket path
against an access point path. That is what --vend-check
above does, and it was the only test
in this exercise that produced a conclusion with no room for interpretation.
I created a test workspace and destroyed it the same day. The workspace mode that uses your own
AWS account creates a VPC with a NAT Gateway. A NAT Gateway runs roughly 45 USD a month per
gateway (hourly charge only, excluding data processing, at the August 2026 Tokyo region rate).
These resources are created directly rather than through a CloudFormation stack, so deleting the workspace does not remove them. This is the easiest place to make a foolish mistake during
I recorded a baseline before starting and reconciled afterwards. Available NAT Gateways back to
zero; VPC, both IAM roles and the bucket gone, with no unassociated Elastic IPs.
I turned that check into a script (shared/scripts/audit_databricks_workspace_footprint.py
). In
a shared account a plain listing is not a verdict, because most of what it shows belongs to
someone else. Run it once before creating anything and once after teardown; it exits non-zero if
anything exists that was not in the baseline. Running it confirmed this teardown was clean and
also surfaced leftover roles from an earlier workspace. The cost breakdown is in the runbook with
measured figures.
Established
CREATE STORAGE CREDENTIAL
/ CREATE EXTERNAL LOCATION
/ CREATE EXTERNAL VOLUME
succeed against an FSx for ONTAP S3 AP alias, with Unity Catalog validation left on_object_metadata
on S3 general bucket returns object tags and user metadata correctlyNot established
_object_metadata
would read tags through an Access Point if the session policy were fixedai_parse_document
over files served from an Access PointGROUP BY
on a FILE column is correct at scale, or merely acceptedReporting status
The object tag behaviour has been escalated through AWS Support to the FSx for ONTAP service
team and is awaiting a determination. What I asked for is not a timeline but the distinction:
a defect that will be fixed, or an undocumented restriction that gets written down. Either
answer lets me state the constraint accurately.
The session policy issue and the operator inconsistency go to Databricks when this article
publishes.
This repository carried an entry from May 2026 saying Unity Catalog external locations do not
support S3 Access Points. Registration does work; the entry has been corrected. If you read the
earlier version, please note the change.
The FILE type is a well-built mechanism for bringing unstructured data under table governance.
For referencing files on an FSx for ONTAP S3 Access Point with zero copies, though, it does not
get there today. Registration works, so you can try it, but reads fail, and because the cause
sits in the session policy on the vending side, users cannot fix it.
If zero-copy is the requirement, AWS-native engines that authorise in the caller's own IAM
context are the more straightforward path for now — in exchange for assembling table-level
governance yourself with Lake Formation. If Unity Catalog governance is the requirement, then
for the time being data placement has to be part of the design.
The object tag character set is awaiting a determination. When it moves, I will update both the
article and the repository.
I hope this is useful to someone.
See you next time.
All test environments have been deleted. The behaviours described are measurements from
2026-08-12 in one specific configuration and may change with platform updates. Beta features are
involved, so please check against current documentation before making production decisions.