How to spot adversaries who sign in with stolen credentials, using Databricks verbose audit logs and Genie
by Solomon Boyd, Alexis Merritt, Arun Pamulapati, Sam Pezzino and David Veuve
At Databricks, we know your data is your most valuable asset. Databricks has worked with thousands of customers to securely deploy the Databricks Data + AI Platform with the appropriate features to meet their security, privacy, and regulatory requirements.
The modern cyberattack does not always begin with a shattered perimeter; it begins with a valid login. Once an adversary acquires a leaked session cookie, personal access token (PAT), or service-principal credential, the initial breach is already complete. From that moment on, their entire goal is to blend in and operate under the guise of your compromised user or service.
But having a key doesn't mean you know the building. Imagine an intruder swiping a badge to enter a secure office. They may have cleared the revolving doors, but they still have to navigate the security guard, guess which floors they have access to, and figure out where the vault is. Because an attacker lacks the deep, localized context of your actual employees, their behavior will eventually betray them. Spotting those irregularities is how you catch them.
In this post, we walk through how a credential-driven attack unfolds on the Databricks platform, how to detect the shift in an identity’s behavior using Databricks verbose audit logs and Genie, how to respond once you confirm a compromise, and which controls limit what a stolen credential can do in the first place.
When establishing access via legitimate credentials, adversaries, like security practitioners, also understand their access may get revoked at any point when they deviate too greatly from the identity’s expected behavior. Adversaries rely on non-linear attack sequencing, such as validating one set of credentials to map your environment, then retrieving another set of credentials to return through a new identity or workspace that doesn’t have reconnaissance activity associated with it. Additionally, adversaries may gather information without establishing persistence or establish their way back into your Databricks platform before wrapping up their objectives.
Consider a data engineer whose personal access token is copied from a misconfigured continuous integration and delivery (CI/CD) log. Within minutes, the token is used from a new IP to list secret scopes, catalogs, and repos across the workspace, something this engineer has never done. A secret holding a service-principal credential is read. An hour later, that service principal, not the engineer, creates a function, adds itself to an admin group, and exports a handful of init scripts. Each of these events could pass as routine platform use, but taken together they show one intrusion moving through the attack chain.
At Databricks, we expect non-linear event sequencing in relation to credential-driven attacks and use this model to connect individual events into one investigation rather than treating them as unrelated events.
When adversaries leverage compromised credentials in a Platform-as-a-Service (PaaS) environment, such as Databricks, their activity can be grouped into six stages, although they rarely move through them in a straight line. Each stage below describes what the adversary is trying to do and what to look for in Databricks audit logging tables. Treat this guidance as a starting point and tune it to your environment's normal automation.
Note: Events are written as service_name.action_name, for example secrets.getSecret.
The login itself rarely looks unusual. Because the credential is real, authentication succeeds as it would for the legitimate owner. However, what the identity does in the minutes after signing in often looks very different from its normal activity, and that change is what responders can detect. In the data engineer example, the stolen token authenticates successfully, and the only difference at login is a source IP the engineer has never used. The activity that follows is what stands out.
Adversary goal: Access the workspace with valid credentials, whether a stolen browser session, a replayed PAT, or a service-principal credential, and blend into activity the controls are designed to allow.
What to look for:
user_identity, source_ip_address, user_agent, sessionId, request_id, and workspace_id let you follow what the credential does next.accounts.tokenLogin events that arrive in a short burst from an unfamiliar client and are followed by use of the same token from a different source IP. This pattern can indicate that someone is testing a stolen token, and requestParams.tokenId lets you tie those events to the same token.
Next, the adversary needs to learn what the credential can reach before the access is revoked. As a result, a compromised identity can shift from its ordinary activity to listing data, secrets, code, and network controls within seconds. This reconnaissance usually spans several services at once and moves faster than a person navigating the UI. In our scenario, this stage is the burst of calls listing secret scopes, catalogs, and repos within minutes of the token being used, which is activity the engineer has never performed before.
Adversary goal: Build a map of accessible data, credentials, repositories, and guardrails to plan a path deeper into the environment before access is revoked.
secrets.listScopes / listSecrets, unityCatalog.listMetastores / listCatalogs / listSchemas / listExternalLocations / listStorageCredentials, repos.listRepos, and account-level accountsManager.listWorkspaceConfigurations / listNetworkConfigurations / listStorageConfigurations. For example, five or more distinct discovery APIs called within one minute is unlikely to be a person navigating the platform.SHOW TABLES against access-control and configuration schemas, is more meaningful than a high volume of calls. When verbose audit logging is enabled, commandSubmit preserves the query in commandText. IpAccessDenied, especially when it follows accounts.oidcTokenAuthorization or repeats across workspaces. In that case, authorization succeeded but the request came from a blocked network, which is often the earliest record of a credential being replayed from an untrusted location.
Harvesting credentials usually takes several steps. The adversary lists secret scopes, lists the secrets in them, and then reads the ones that look useful. The value they retrieve allows them to return as a different, often more privileged, identity. For this reason, every secret read by a suspect identity is worth following up on. In the data engineer example, the adversary reads a secret that holds a service-principal credential. From that point on, the investigation has to follow two identities: the engineer and the service principal.
Adversary goal: Retrieve a token, API key, or service-principal secret that upgrades the initial foothold into environment-wide access under a new identity.
listScopes followed by getSecret, repeated across multiple scopes in quick succession, indicates harvesting. submitCommand and Databricks SQL read_files() against paths that contain credentials. Keep in mind that the submitted text shows what ran, but it does not confirm that a value was returned.generateTemporaryTableCredential / generateTemporaryVolumeCredential calls. During response, rotate every credential the identity could have reached, and check where each one was used in addition to where it was stored.
Once access is established, adversaries look for a way to run their own code. On Databricks, this typically happens through user-defined functions (UDFs) and jobs. A function can wrap a payload in ordinary-looking code that can be called repeatedly, and a job can run that code on a schedule across different compute. As such, the key question for detection is whether an identity has started performing actions it has not performed before. Returning to the data engineer, an hour after the secret is read, the service principal creates a function. Because the service principal, rather than the engineer, performs this action, a search scoped only to the engineer's activity would miss it.
Adversary goal: Establish a reliable, reusable way to execute payloads, with a fallback when one compute route is blocked.
function_info in system.access.audit is masked except for account admins and databricks_pii_access, so pair audit data with function inventories. importNotebook, then changeJobAcl, Jobs create, runNow, and getRunOutput. These events rarely preserve the notebook body, so retain notebook versions and job configurations alongside them. Repeated import-and-run activity from one source is a stronger signal than either action alone.submitRun followed shortly by startWarehouse, or by a new job on different compute, may indicate an adversary working around a blocked path. The response codes and the short gap between events are what separate this sequence from routine errors. Serverless compute is a common fallback.
With a way to run code, adversaries next try to make sure they can return after the original credential is revoked. Common persistence methods include access grants, newly created tokens, group and workspace membership changes, and control-plane edits. The most useful signal is a sensitive change made by an unfamiliar identity or execution context. For example, a permission change made from inside a job deserves more scrutiny than the same change made by an administrator in the console. Next in the scenario, the service principal adds itself to an admin group. Even if the engineer's token is revoked, that group membership keeps the adversary's access in place.
Adversary goal: Create durable, independent access paths that survive revocation of the compromised credential.
putAcl on a secret scope, updatePermissionAssignment, addPrincipalToGroup, or setAdmin. Pay particular attention when a service principal grants another identity access, or when multiple assignments occur in a short period.setSetting / deleteSetting, IP access-list edits, and createPrivateAccessSettings. Review the requested change, the response, and the source before concluding whether a control was actually bypassed. Also review SCIM group changes from anything other than the expected IdP.generateDbToken calls from an unfamiliar source or during a job run. Each resulting token is a separate credential that will need to be revoked.run_by, run_as, and the expected run window. If a job keeps modifying groups, permissions, or settings after the adversary's interactive activity stops, it may be maintaining access by itself. Preserve the job definition and artifacts before terminating it.
Data collection can be as small as a few targeted exports, and cleanup can involve more than deleting files. Watch for targeted exports, temporary-credential requests, removal of short-lived tooling, and obfuscated commands that make the activity harder to reconstruct. Any one of these events can look routine, so the sequence around them matters more than the individual export or delete. Finally, the service principal from our scenario exports a handful of init scripts. On its own, that is a small export. However, because it follows a secret read and a new function from the same service principal, it should be investigated as part of the same intrusion.
Adversary goal: Take the data that matters and erase the most obvious tooling while leaving quieter access in place.
workspaceExport records the path, the format, and whether it was a direct download. Prioritize exports by unfamiliar identities, to unusual paths or in unusual formats, from previously unseen networks, or that follow secret access or new execution. A small, targeted export of init scripts can leak more than a bulk SOURCE dump.createFunction followed by deleteFunction for the same function in one session, or a repeated create, delete, and recreate cycle, may indicate an attempt to remove evidence. Preserve the definitions and timing before reverting anything. Objects that survive are often given more plausible names than the ones that were deleted.
Using the credential-driven attack chain as a guide, you can direct Genie to work through the attack stages without writing every query from scratch. Every detection in this post runs on telemetry you already have. Databricks writes control-plane activity to the system.access.audit system table.
To get started, install the Detection App, which adds a Genie agent for Databricks’ audit logging. You can work it from Genie One in two ways:
Keep two limits in mind. First, an empty result doesn't mean nothing happened. The event may be outside your time window, or verbose audit logging may be disabled. Second, audit logs record that something changed and who changed it, but often not the old or new values.
Once a hunt surfaces an identity that doesn't fit its normal pattern, the focus shifts to what access the adversary still has. Because credential-driven attacks are non-linear, the first access you find is rarely the only access. In the data engineer example, revoking the engineer's token would leave the service principal's credential, its admin group membership, and the function it created in place. A complete response removes the original credential as well as every path the adversary created afterward, including grants, tokens, functions, and jobs that keep running after the user is disabled.
For a suspected workspace compromise, responders should: To return to the office building: context-based ingress decides who gets through the revolving doors, and least privilege decides which floors their badge can open. Early detection buys response time, but preventive controls limit what a stolen credential can do. The controls below are grouped by what they take away from the adversary, in roughly the order they break their attack chain.
The strongest preventive control against credential reuse is context-based ingress control. It evaluates who is making a request, where it comes from, and what it is for, instead of relying on a single network perimeter. This is especially useful as more organizations move away from VPNs. Even without a corporate network to anchor on, you can limit a stolen token to the contexts its owner actually uses. For example, you can restrict a privileged service principal to API requests from trusted networks while applying a different policy to human users.
system.access.inbound_network, protect privileged service principals first, and enforce after you've accounted for legitimate automation and partner integrations.
Remember, a stolen credential’s blast radius is only as damaging as the privileges assigned to it. Reduce that exposure and be able to hunt for anomalous activity by taking these steps:
Subscribe to our blog and get the latest posts delivered to your inbox.