kafka启用Kerberos认证
1.环境准备
1.1.建用户
创建用户组ywjk groupadd ywjk
新增用户useradd -g ywjk ywjk
设置密码passwd ywjk
1.2.上传kafk压缩包
从官网下载kafka压缩包，这里用的是kafka_2.11-0.10.2.2.tgz
上传kafka压缩包，利用git客户端从windows客户端上传至每台linux主机，传至/home/ywjk目录下面
scp -r ./kafka_2.11-0.10.2.2.tgz ywjk@192.168.1.96:/home/ywjkscp -r ./kafka_2.11-0.10.2.2.tgz ywjk@192.168.1.97:/home/ywjkscp -r ./kafka_2.11-0.10.2.2.tgz ywjk@192.168.1.98:/home/ywjk
Administrator@DESKTOP-QD6V450 MINGW64 /e/workspace/soft$ lltotal 37160-rw-r--r-- 1 Administrator 197121 38048170 12月 28 10:25 kafka_2.11-0.10.2.2.tgzAdministrator@DESKTOP-QD6V450 MINGW64 /e/workspace/soft$ scp -r ./kafka_2.11-0.10.2.2.tgz ywjk@192.168.1.96:/home/ywjkywjk@192.168.1.96's password:kafka_2.11-0.10.2.2.tgz 100% 36MB 4.4MB/s 00:08Administrator@DESKTOP-QD6V450 MINGW64 /e/workspace/soft$ scp -r ./kafka_2.11-0.10.2.2.tgz ywjk@192.168.1.97:/home/ywjkThe authenticity of host '192.168.1.97 (192.168.1.97)' can't be established.ECDSA key fingerprint is SHA256:4jriaM+47zOyv+In1m2ndnYAZt5sXfYHE2Wo9S7jEqE.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesWarning: Permanently added '192.168.1.97' (ECDSA) to the list of known hosts.ywjk@192.168.1.97's password:kafka_2.11-0.10.2.2.tgz 100% 36MB 5.0MB/s 00:07Administrator@DESKTOP-QD6V450 MINGW64 /e/workspace/soft$ scp -r ./kafka_2.11-0.10.2.2.tgz ywjk@192.168.1.98:/home/ywjkThe authenticity of host '192.168.1.98 (192.168.1.98)' can't be established.ECDSA key fingerprint is SHA256:sLwxbrt8Gq/43E2nW9q0iEwDJKuALl9cwFQrv3yCqxE.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesWarning: Permanently added '192.168.1.98' (ECDSA) to the list of known hosts.ywjk@192.168.1.98's password:kafka_2.11-0.10.2.2.tgz
在每台机器上面解压
tar -zvxf ./kafka_2.11-0.10.2.2.tgz
2.Kerbersoe安装
三台机器信息如下：
192.168.1.96 KDC Server
192.168.1.97 Client
192.168.1.98 Client
2.1.服务器安装
在KDC Server安装服务端
可以在这机器中192.168.1.96建立yum源，/etc/yum.repos.d/cdrom.repo 仓库配置如下:
[cdrom]name=cdrombaseurl=/centosenabled=1gpgcheck=0
yum -y install krb5-server krb5-libs krb5-workstation
会生成 kerberos配置文件
ll /var/kerberos/krb5kdc/
[ywjk@hrxjb1 root]$ ll /var/kerberos/krb5kdc/total 24-rw------- 1 root root 24 Dec 28 13:41 kadm5.acl-rw------- 1 root root 451 Sep 30 21:21 kdc.conf-rw------- 1 root root 8192 Dec 28 14:25 principal-rw------- 1 root root 8192 Dec 28 13:42 principal.kadm5-rw------- 1 root root 0 Dec 28 13:42 principal.kadm5.lock-rw------- 1 root root 0 Dec 28 14:25 principal.ok
2.2.修改配置文件
2.2.1. krb5.conf
/etc/krb5.conf
把krb5.conf复制至其它客户端机器(192.168.1.97、192.168.1.98)
# Configuration snippets may be placed in this directory as wellincludedir /etc/krb5.conf.d/[logging]default = FILE:/var/log/krb5libs.logkdc = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.log[libdefaults]dns_lookup_realm = falseticket_lifetime = 24hrenew_lifetime = 7dforwardable = truerdns = falsepkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crtdefault_realm = default_ccache_name = KEYRING:persistent:%{uid}[realms] = {kdc = admin_server = [domain_realm]. = =
2.2.2. kadm5.acl
cat /var/kerberos/krb5kdc/kadm5.acl
[root@hrxjb1 ~]# cat /var/kerberos/krb5kdc/kadm5.acl*/admin@*
配置说明
Kadm5.acl文件域名要跟 /etc/krb5.conf.d/中的 realms 配置节一致
2.2.3.初始化KDC数据库
kdb5_util create -r -s
2.3.启动服务
2.3.1.启动kdc服务
systemctl status krb5kdc查看状态状态为 inactive(dead)
[root@hrxjb1 ~]# systemctl status krb5kdc● krb5kdc.service - Kerberos 5 KDCLoaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled)Active: inactive (dead)
systemctl start krb5kdc
[root@hrxjb1 ~]# systemctl start krb5kdc[root@hrxjb1 ~]# systemctl status krb5kdc● krb5kdc.service - Kerberos 5 KDCLoaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled)Active: active (running) since Mon -12-28 18:00:31 CST; 1s agoProcess: 26940 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=0/SUCCESS)Main PID: 26941 (krb5kdc)CGroup: /system.slice/krb5kdc.service└─26941 /usr/sbin/krb5kdc -P /var/run/krb5kdc.pidDec 28 18:00:31 systemd[1]: Starting Kerberos 5 KDC...Dec 28 18:00:31 systemd[1]: Started Kerberos 5 KDC.
2.3.2.启动kadmin服务
systemctl status kadmin
[root@hrxjb1 ~]# systemctl status kadmin● kadmin.service - Kerberos 5 Password-changing and AdministrationLoaded: loaded (/usr/lib/systemd/system/kadmin.service; disabled; vendor preset: disabled)Active: inactive (dead)
systemctl start kadmin
[root@hrxjb1 ~]# systemctl start kadmin[root@hrxjb1 ~]# systemctl status kadmin● kadmin.service - Kerberos 5 Password-changing and AdministrationLoaded: loaded (/usr/lib/systemd/system/kadmin.service; disabled; vendor preset: disabled)Active: active (running) since Mon -12-28 18:01:50 CST; 2s agoProcess: 27080 ExecStart=/usr/sbin/_kadmind -P /var/run/kadmind.pid $KADMIND_ARGS (code=exited, status=0/SUCCESS)Main PID: 27081 (kadmind)CGroup: /system.slice/kadmin.service└─27081 /usr/sbin/kadmind -P /var/run/kadmind.pidDec 28 18:01:50 systemd[1]: Starting Kerberos 5 Password-changing and Administration...Dec 28 18:01:50 systemd[1]: Started Kerberos 5 Password-changing and Administration.
2.4.client安装
在每台节点机上执行(192.168.1.97和192.168.1.98)
yum -y install krb5-workstation
2.5.生成密钥文件
kafka的生成随机密码
addprinc -randkey kafka/@addprinc -randkey kafka/@addprinc -randkey kafka/@
加入至密钥文件
ktadd -k /etc/security/keytabs/kafka1.keytab kafka/@ktadd -k /etc/security/keytabs/kafka2.keytab kafka/@ktadd -k /etc/security/keytabs/kafka3.keytab kafka/@
zookeeper的生成随机密码
addprinc -randkey zookeeper/@addprinc -randkey zookeeper/@addprinc -randkey zookeeper/@
加入至密钥文件
ktadd -k /etc/security/keytabs/kafka1.keytab zookeeper/@ktadd -k /etc/security/keytabs/kafka2.keytab zookeeper/@ktadd -k /etc/security/keytabs/kafka3.keytab zookeeper/@
利用klist查看添加的用户
klist -ket /etc/security/keytabs/kafka1.keytab klist -ket /etc/security/keytabs/kafka1.keytabklist -ket /etc/security/keytabs/kafka1.keytab
把 /etc/security/keytabs/kafka_server.keytab 拷贝至客户端对应的目录
cp -r /etc/security/keytabs/kafka1.keytab /etc/security/keytabs/kafka.keytabscp -r /etc/security/keytabs/kafka2.keytab root@192.168.1.97:/etc/security/keytabs/kafka.keytabscp -r /etc/security/keytabs/kafka3.keytab root@192.168.1.98:/etc/security/keytabs/kafka.keytab
3.安装kafka+zookeeper
3.1.配置zookeeper
3.1.1.创建zookeeper 使用的 zookeeper.jaas
三台机器的zookeeper.jaas，要注意principal与/etc/security/keytabs/kafka.keytab里面的用户相对应
以192.168.1.96为例
cat /home/ywjk/kafka_2.11-0.10.2.2/config/zookeeper.jaas
Server{com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=truestoreKey=truekeyTab="/etc/security/keytabs/kafka.keytab"#这里要注意每台机器的/etc/security/keytabs/kafka.keytab是否包含如下用户名principal="zookeeper/@"userTicketCache=false;};
3.1.2. zookeeper配置文件
加入如下配置：
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProviderrequireClientAuthScheme=sasljaasLoginRenew=3600000
3.1.3.修改zookeeper启动脚本
在zookeeper启动脚本中加入
/home/ywjk/kafka_2.11-0.10.2.2/bin/zookeeper-server-start.shexport KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/ywjk/kafka_2.11-0.10.2.2/config/zookeeper.jaas"
3.2.配置kafka
3.2.1.创建kafka 的kafka.jaas
cat /home/ywjk/kafka_2.11-0.10.2.2/config/kafka.jaas
KafkaServer{com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=truestoreKey=trueserviceName="kafka"keyTab="/etc/security/keytabs/kafka.keytab"principal="kafka/@";};KafkaClient{com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=truestoreKey=trueserviceName="kafka"keyTab="/etc/security/keytabs/kafka.keytab"principal="kafka/@"userTicketCache=true;};Client{com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=truestoreKey=trueserviceName="kafka"keyTab="/etc/security/keytabs/kafka.keytab"principal="kafka/@"userTicketCache=true;};
3.2.2.配置kafka的 server.properties
cat /home/ywjk/kafka_2.11-0.10.2.2/config/server.properties
在server.properties文件中加入
zookeeper.connect=:2182,:2182,:2182listeners=SASL_PLAINTEXT://192.168.1.96:9092authorizer.class.name=kafka.security.auth.SimpleAclAuthorizersecurity.inter.broker.protocol=SASL_PLAINTEXTsasl.mechanism.inter.broker.protocol=GSSAPIsasl.enabled.mechanisms=GSSAPIsasl.kerberos.service.name=kafka
3.2.3.修改kafa启动脚本
kafka启动脚本中加入
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/ywjk/kafka_2.11-0.10.2.2/config/kafka.jaas"
3.3. 启动zookeeper
/home/ywjk/kafka_2.11-0.10.2.2/bin/zookeeper-server-start.sh -daemon /home/ywjk/kafka_2.11-0.10.2.2/config/zookeeper.properties
3.4. 启动kafka
/home/ywjk/kafka_2.11-0.10.2.2/bin/kafka-server-start.sh -daemon /home/ywjk/kafka_2.11-0.10.2.2/config/server.properties
3.5.测试kafka
3.5.1.创建topic
在/home/ywjk/kafka_2.11-0.10.2.2/bin/kafka-topics.sh
/home/ywjk/kafka_2.11-0.10.2.2/bin/kafka-console-consumer.sh
/home/ywjk/kafka_2.11-0.10.2.2/bin/kafka-console-producer.sh
加入
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/ywjk/kafka_2.11-0.10.2.2/config/kafka.jaas"
/home/ywjk/kafka_2.11-0.10.2.2/bin/kafka-topics.sh --create --zookeeper :2182,:2182,:2182 --replication-factor 1 --partitions 3 --topic hnhrtlrealdata
3.5.2.启动生产者
在 /home/ywjk/kafka_2.11-0.10.2.2/config/producer.properties 加入如下配置
security.protocol=SASL_PLAINTEXTsasl.mechanism=GSSAPIsasl.kerberos.service.name=kafka
命令启动生产者发送消息
/home/ywjk/kafka_2.11-0.10.2.2/bin/kafka-console-producer.sh --broker-list :9092,:9092,:9092 --topic hnhrtlrealdata --producer.config /home/ywjk/kafka_2.11-0.10.2.2/config/producer.properties
3.5.3.启动消费者
在/home/ywjk/kafka_2.11-0.10.2.2/config/consumer.properties 加入如下配置
security.protocol=SASL_PLAINTEXTsasl.mechanism=GSSAPIsasl.kerberos.service.name=kafka
/home/ywjk/kafka_2.11-0.10.2.2/bin//kafka-console-consumer.sh --bootstrap-server :9092,:9092,:9092 --topic hnhrtlrealdata --from-beginning --consumer.config /home/ywjk/kafka_2.11-0.10.2.2/config/consumer.properties
3.6.flink加入kerberos认证
flink安装在/usr/local/flink-1.11.2
cat /usr/local/flink-1.11.2/conf/flink-conf.yaml
在flink-conf.yaml中加入
security.kerberos.login.use-ticket-cache: truesecurity.kerberos.login.keytab: /etc/security/keytabs/kafka.keytabsecurity.kerberos.login.principal: kafka/@# The configuration below defines which JAAS login contextssecurity.kerberos.login.contexts: Client,KafkaClient
要把kafka的kafka.jaas 和kerberos认证生成的 kafka.keytab拷贝至flink每台机器
如果觉得《kafka启用Kerberos认证》对你有帮助，请点赞、收藏，并留下你的观点哦！
