MSK (+Serverless) Setup

This guide has been written for MSK Serverless but is applicable for MSK in general as well.

Authentication options for Kafbat-UI:

KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL
KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM=AWS_MSK_IAM
KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule required;'
KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS='software.amazon.msk.auth.iam.IAMClientCallbackHandler'

Example ECS service configuration:

environment: [
    {
      name: "KAFKA_CLUSTERS_0_NAME",
      value: config.mskClusterName
    },
    {
      name: "KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS",
      value: config.mskBootstrapServers
    },
    {
      name: "KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION",
      value: "true"
    },
    {
      name: "KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL",
      value: "SASL_SSL"
    },
    {
      name: "KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM",
      value: "AWS_MSK_IAM"
    },
    {
      name: "KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS",
      value: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    },
    {
      name: "KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG",
      value: "software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;"
    },
]

Creating an instance

1. Go to the MSK page

2. Click "create cluster"

3. Choose "Custom create"

4. Choose "Serverless"

5. Choose VPC and subnets

6. Choose the default security group or use the existing one

Creating a policy

1. Go to IAM policies

2. Click "create policy"

3. Click "JSON"

4. Paste the following policy example in the editor, and replace "MSK ARN" with the ARN of your MSK cluster

{
    "Version": "2012-10-17",
    "Statement": [
        
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:AlterClusterDynamicConfiguration",
                "kafka-cluster:DescribeClusterDynamicConfiguration",
            ],
            "Resource": "arn:aws:kafka:eu-central-1:297478128798:cluster/test-wizard/7b39802a-21ac-48fe-b6e8-a7baf2ae2533-s2"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:ReadData",
                "kafka-cluster:WriteData",
                "kafka-cluster:DescribeTopicDynamicConfiguration",
                "kafka-cluster:AlterTopicDynamicConfiguration",
                "kafka-cluster:AlterTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:DeleteTopic"
            ],
            "Resource": "arn:aws:kafka:eu-central-1:297478128798:topic/test-wizard/7b39802a-21ac-48fe-b6e8-a7baf2ae2533-s2/*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DeleteGroup",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:AlterGroup"
            ],
            "Resource": "arn:aws:kafka:eu-central-1:297478128798:group/test-wizard/7b39802a-21ac-48fe-b6e8-a7baf2ae2533-s2/*"
        }
    ]
}

Attaching the policy to a user

Creating a role for EC2

1. Go to IAM

2. Click "Create role"

3. Choose AWS Services and EC2

4. On the next page find the policy which has been created in the previous step

Attaching the role to the EC2 instance

1. Go to EC2

2. Choose your EC2 with Kafbat-UI

3. Go to Actions -> Security -> Modify IAM role

4. Choose the IAM role from previous step

5. Click Update IAM role