Apache Knox Tutorial: Security Gateway for Hadoop (2026)
Apache Knox provides perimeter security for the Hadoop ecosystem by consolidating authentication and authorization at a single entry point. After deploying Knox for organizations with strict security requirements, I appreciate how it simplifies secure access to Hadoop services without exposing them directly to the network.
This tutorial covers Knox's architecture, authentication providers, service routing, SSO integration, and best practices for securing Hadoop clusters behind a unified security gateway.
Introduction to Apache Knox
Apache Knox is a reverse proxy that provides a single point of access for Hadoop cluster services. It eliminates the need to expose individual service endpoints (HDFS, YARN, Hive, etc.) directly to the network. Knox handles authentication, SSL termination, and can enforce authorization policies through integrated providers.
Knox acts as a gateway in front of Hadoop services, routing requests to the appropriate backend service while handling authentication tokens, session management, and audit logging. It supports multiple authentication methods including LDAP, Kerberos, SAML, and custom providers.
# Download and install Knox
wget https://downloads.apache.org/knox/apache-knox-1.8.0/knox-1.8.0.tar.gz
tar -xzf knox-1.8.0.tar.gz
cd knox-1.8.0
# Initialize Knox master key
bin/knoxcli.sh create-master --masterKeyPassword masterPass
# Configure gateway-site.xml
# gateway.port=8443
# gateway.path=/gateway
# knox.master.secret=masterPass
# Start Knox
sbin/knox.sh start
# Access the web UI
open https://knox-host:8443/gateway/webui
# Verify Knox is running
curl -k https://knox-host:8443/gateway/admin/api/v1/webssocookie
# Check cluster health
curl -k -u admin:admin-123 https://knox-host:8443/gateway/admin/api/v1/clustersService Topology and Routing
Knox uses topology definitions to define which Hadoop services are accessible through the gateway. Topologies are XML files that configure service URIs, authentication providers, and authorization policies. Each topology defines a logical cluster view with its associated services and their backend endpoints.
Service routing maps gateway URLs to backend Hadoop service endpoints. For example, /gateway/cluster/hdfs/* routes to the HDFS NameNode, while /gateway/cluster/hive/* routes to HiveServer2. This abstraction allows Knox to manage access patterns independently of the underlying service configuration.
cluster
authentication
AuthenticationProvider
true
authorization
AclsAuthorizationProvider
true
audit
AuditLogger
true
namenode
http://namenode:9870
resource-manager
http://resourcemanager:8088
hive
http://hiveserver:10000/cliservice
webhdfs
http://namenode:9870/webhdfs
Authentication Providers
Knox supports multiple authentication providers: SSOProvider (web SSO via browser cookie), HadoopAuthFilter (Kerberos/SPNEGO for Hadoop services), LdapAuthProvider (LDAP/AD authentication), PAMAuthProvider (Linux PAM), and custom providers. Each topology can specify different authentication providers.
For enterprise deployments, SAML-based SSO with corporate identity providers (Okta, Azure AD) is common. Knox acts as a SAML service provider, authenticating users against the corporate IdP and issuing gateway tokens for backend service access.
# LDAP Authentication Provider
authentication
LdapAuthProvider
true
ldap.url
ldap://ldap.corp.com:389
ldap.baseDn
dc=corp,dc=com
ldap.userDnPattern
uid={0},ou=people
ldap.groupSearchBase
ou=groups
ldap.groupSearchFilter
member={0}
# SSO Authentication
authentication
SSOAuthenticationProvider
true
sso.authentication.provider.url
https://login.corp.com/sso
Knox and Hadoop Service Access
Knox proxies requests to Hadoop services, rewriting URLs and adding authentication headers. Clients interact with Knox URLs instead of direct service endpoints. Knox translates gateway requests to backend service calls and returns responses to clients transparently.
For WebHDFS access, Knox adds authentication headers and routes to the NameNode. For Hive, Knox authenticates and proxies to HiveServer2's thrift interface. For YARN, Knox routes ResourceManager API calls. All requests are SSL-encrypted between client and Knox, and optionally between Knox and backend services.
# Access HDFS via Knox
curl -k -u user:password \
'https://knox-host:8443/gateway/cluster/hdfs/v1/user/data/file.txt'
# Access Hive via Knox
curl -k -u user:password \
-X POST \
'https://knox-host:8443/gateway/cluster/hive' \
-d '{"operation": "execute", "statement": "SELECT count(*) FROM logs"}'
# Access YARN via Knox
curl -k -u user:password \
'https://knox-host:8443/gateway/cluster/resourcemanager/v1/cluster/apps'
# Knox proxy configuration
gateway.service.config.acls.enabled=true
gateway.proxyuser.knox.users=*
gateway.proxyuser.knox.groups=*
gateway.proxyuser.knox.hosts=*Audit and Security Policies
Knox logs every request through the gateway: user, service, operation, timestamp, and result. Audit logs are written to files and can be forwarded to SIEM systems. Knox audit captures authentication attempts, authorization decisions, and proxy activity.
Authorization policies control which users can access which services. ACLs in topology definitions map users/groups to permitted services. This provides defense-in-depth: even if a user has Kerberos credentials, Knox can deny access based on gateway-level policies.
# Knox audit configuration
# gateway-audit.xml
audit.log.enabled
true
audit.log.dir
/var/log/knox/audit
audit.log.max.file.size
10485760
audit.log.max.backup.count
5
# Authorization ACLs
authorization
AclsAuthorizationProvider
true
knox.acl.enabled
true
knox.acl.mode
AND
# Audit log entry format
# timestamp | user | service | operation | status | client_ip | server_ipDeployment and High Availability
Deploy Knox behind a load balancer for high availability. Knox nodes share topology configuration via NFS or ZooKeeper. Session affinity is required for web SSO but not for API access with explicit credentials. Knox's stateless design makes horizontal scaling straightforward.
For production, deploy 2+ Knox nodes behind an HA load balancer, use TLS termination at the load balancer, and configure LDAP/Kerberos authentication against your corporate directory.
# Knox HA deployment
# Load balancer (HAProxy/NGINX) in front of Knox nodes
# Backend: knox-node1:8443, knox-node2:8443
# Knox node configuration
# gateway-site.xml
gateway.port
8443
gateway.admin.port
8443
# Topology sync via shared storage
# Place topology XML files in /etc/knox/topologies/
# Mount via NFS across all Knox nodes
# Start Knox on each node
$ sbin/knox.sh start
# Verify via load balancer
curl -k https://knox-lb.corp.com/gateway/admin/api/v1/clusters
# Session management for web SSO
# Cookie-based: use sticky sessions at load balancer
# Token-based: no sticky sessions neededFrequently Asked Questions
What is the difference between Knox and Apache Ranger?
Knox provides perimeter security (authentication, SSL termination, proxy). Ranger provides data-level authorization (who can access which tables/columns). They complement each other: Knox authenticates users, Ranger authorizes their data access.
Can Knox replace individual service authentication?
Yes, Knox centralizes authentication so services do not need individual auth configuration. Users authenticate to Knox, which adds authentication headers for backend services. This simplifies client configuration.
How does Knox handle Kerberos?
Knox can act as a Kerberos service, authenticating users via SPNEGO and impersonating them when proxying to backend Hadoop services. Knox holds a service keytab and performs Kerberos authentication on behalf of clients.
Is Knox suitable for multi-cluster environments?
Yes, Knox topologies can define multiple clusters. Clients authenticate to Knox and access different clusters through topology-specific URLs. This simplifies multi-cluster security administration.
Originally published on Ayodhyyya. Last updated June 1, 2026.