ADH-8486: fix Spark Ranger stale policy cache bypass

[iamlapa]

Summary

This PR fixes the Spark3 + Ranger stale policy cache bypass we reproduced around policy download authorization.

Spark3 runs under the end user's account, unlike Hive/Impala service processes. Because of that, Spark historically needed * in policy.download.auth.users and tag.download.auth.users to let user-launched Spark apps download Ranger policies. That works functionally, but it also means any authorized Spark user can call Ranger Admin APIs and download the full Hive policy bundle.

When * is removed, Ranger Admin correctly returns 401/403, but the Spark plugin used to keep authorizing with its last local policy cache. So after a user lost access, an already running spark3-shell could still use stale cached policies.

This PR addresses the issue in three layers:

1. Add group-based policy/tag/role download authorization.
2. Make Spark fail closed on explicit Ranger Admin 401/403.
3. Isolate local policy cache and audit spool directories by user or group.

What changed

Ranger Admin download authorization

Added new service config parameters:

• policy.download.auth.groups
• tag.download.auth.groups

Policy and role download now allow access if the caller matches either:

• policy.download.auth.users
• policy.download.auth.groups

Tag download now allows access if the caller matches either:

• tag.download.auth.users
• tag.download.auth.groups

The existing * behavior in *.auth.users is preserved for compatibility, but Spark managed configs can now use a real group instead, for example:

policy.download.auth.groups=spark_users
tag.download.auth.groups=spark_users

Group membership is resolved through Ranger's user/group store on each request. Kerberos-style login IDs are normalized into candidate short names, so principals like:

hive/host@REALM
user@REALM

can still match Ranger users/groups stored as short names.

Spark fail-closed behavior

RangerAdminRESTClient now treats 401 and 403 as a dedicated access-denied condition instead of a generic refresh failure.

A new exception was added:

RangerAdminClientAccessDeniedException

This is thrown for policy, role, and tag download when Ranger Admin explicitly denies access.

Spark/plugin refresh code now handles that differently from transient failures:

• 401/403: mark authz state invalid, do not fall back to stale cache.
• timeout/network/5xx/other transient errors: keep existing last-known behavior.
• successful refresh: clear the authz-denied state.

A new plugin config controls the fail-closed behavior:

ranger.plugin.<service>.policy.refresh.authz.denied.mode=failclosed

When enabled, any new access check is denied while policy/role/tag refresh is in authz-denied state.

This covers:

• regular access checks
• batch access checks
• row filter policy evaluation
• data mask policy evaluation

Policy cache directory isolation

Added policy cache subdir mode:

ranger.plugin.spark.policy.cache.subdir.mode=disabled|peruser|pergroup

Modes:

disabled -> existing shared directory behavior
peruser  -> $base/$USER, dir 0700, files 0600
pergroup -> $base/$GROUP, dir 0770, files 0660

This is used by:

• policy cache
• role cache
• tag cache

The helper validates user/group path elements and rejects unsafe values like empty names, path separators, and ...

It also rejects unsafe existing directories and symlinks.

Important detail: for peruser and pergroup, the shared base directory must already exist and have expected permissions. The plugin only creates the final $USER or $GROUP leaf directory. This avoids the first Spark user accidentally creating /srv/ranger/... as a private 0700 parent and blocking everyone else.

Audit spool directory isolation

Added audit spool subdir mode:

xasecure.audit.destination.solr.batch.filespool.subdir.mode=disabled|peruser|pergroup

Modes are the same:

disabled -> existing shared spool behavior
peruser  -> $base/$USER, dir 0700, files 0600
pergroup -> $base/$GROUP, dir 0770, files 0660

This was wired into the file audit spool implementations so Spark users no longer need to share the same writable audit spool directory.

Why these changes are needed

The original bug is not just that a user could not refresh policies after losing download permission. The real issue is that Spark treated that condition as a normal refresh failure and kept using old cached policies.

That creates this unsafe flow:

user has access
Spark downloads policies
admin revokes access
Ranger Admin returns 403
Spark keeps authorizing with stale cache

After this PR, the flow becomes:

user has access
Spark downloads policies
admin revokes access or removes user from spark_users
Ranger Admin returns 403
Spark marks refresh authz as denied
new access checks fail closed

This means revocation takes effect after the next refresh instead of being bypassed by local cache.

Recommended Spark config

Example target config for Spark3:

policy.download.auth.users=
tag.download.auth.users=
policy.download.auth.groups=spark_users
tag.download.auth.groups=spark_users

ranger.plugin.spark.policy.refresh.authz.denied.mode=failclosed
ranger.plugin.spark.policy.cache.subdir.mode=peruser
xasecure.audit.destination.solr.batch.filespool.subdir.mode=peruser

pergroup is also supported for clusters where Spark users are intentionally administered through trusted primary groups.