Kafka Monitoring

Monitor the performance of your entire Apache Kafka infrastructure with Site24x7's ready-to-install Kafka monitoring plugin integration. Our Kafka monitoring tool tracks critical metrics to help you stay on top of your Kafka stack and prevent potential data loss.

Add your first server monitoring plugin for free and subsequent plugins for just $1 each per month.

Start 30-day free trial Try now, sign up in 30 seconds
Kafka Monitoring

What is Apache Kafka?

Apache Kafka is an open-source, distributed stream-processing platform used to collect, store and process real-time data streams. It was originally developed by LinkedIn and was open sourced in 2011. Kafka later became a part of the Apache Software Foundation. Written in Java and Scala, its core capabilities include high throughput, scalability, and low latency. Apache Kafka is often used to build stream data pipelines and real-time, event-driven applications.

What is Apache Kafka monitoring?

Apache Kafka monitoring involves collecting, analyzing, and visualizing metrics from the Kafka cluster, including producer, broker, and consumer data. It ensures Kafka's performance, stability, and resource utilization while facilitating fault detection and capacity planning for efficient data streaming pipelines and issue resolution.

Kafka performance monitoring with Site24x7

Track the health of your Kafka infrastructure and identify bottlenecks affecting its performance. With our Kafka monitoring tool, keep track of unexpected trends Kafka broker instances through our metric graphs, so that you can troubleshoot quickly.

Get insights on critical Kafka performance metrics

Leverage key Kafka availability, usage, and performance metrics, including producer's request rate, network request rate by broker, and consumer's fetch rate, and stay ahead of issues affecting them.

Stay updated with instant alerts

Get instant notifications every time values cross pre-defined threshold limits and swiftly take preventive or recovery measures.

Remediate incidents with event-driven IT automation

Set up automated workflows to take corrective measures in Kafka when pre-defined threshold breaches occur.

Create custom dashboards

Assemble data from different parts of your infrastructure with drag-and-drop widgets and build personalized dashboards to get a holistic view of Kafka stack's performance.

Add the first plugin monitor for free and subsequent monitors for only $1 each per month.

Sign up for a 30-day free trial. No credit card required.

Get started with Site24x7's Kafka monitoring tool

Get Started

  • Supported Platforms:
  • Python Version:
    2.7 & above
  • Linux Agent Version:
    15.0.0 & above

Performance Metrics

The following metrics are captured by the Kafka Producer monitoring plugin:

Name Description
Bytes In Per Sec Rate of bytes received by the broker per second.
Bytes Out Per Sec Rate of bytes sent by the broker per second.
Bytes Rejected Per Sec Rate of bytes rejected by the broker per second.
Failed Fetch Requests Per Sec Rate of fetch requests that failed per second.
Failed Produce Requests Per Sec Rate of produce requests that failed per second.
Fetch Message Conversions Per Sec Rate of fetch message conversions per second.
Invalid Magic Number Records Per Sec Rate of records with invalid magic numbers per second.
Invalid Message Crc Records Per Sec Rate of records with invalid message CRC per second.
Invalid Offset Or Sequence Records Per Sec Rate of records with invalid offsets or sequences per second.
Messages In Per Sec Rate of messages received by the broker per second.
No Key Compacted Topic Records Per Sec Rate of records in compacted topics with no key per second.
Produce Message Conversions Per Sec Rate of produce message conversions per second.
Reassignment Bytes In Per Sec Rate of bytes reassigned to the broker per second.
Reassignment Bytes Out Per Sec Rate of bytes reassigned from the broker per second.
Replication Bytes In Per Sec Rate of bytes replicated to the broker per second.
Total Fetch Requests Per Sec Total rate of fetch requests per second.
Total Produce Requests Per Sec Total rate of produce requests per second.
At Min Isr Partition Count Number of partitions with replicas currently at or below the minimum in-sync replicas.
Failed Isr Updates Per Sec Rate of failed in-sync replica updates per second.
Isr Shrinks Per Sec Rate of in-sync replica shrinkage per second.
Leader Count Number of leaders managed by the replication manager.
Partition Count Total number of partitions managed by the replication manager.
Partitions With Late Transactions Count Number of partitions with late transactions.
Producer Id Count Number of active producer IDs.
Reassigning Partitions Number of partitions currently being reassigned.
Under Min Isr Partition Count Number of partitions with replicas currently under the minimum in-sync replicas.
Under Replicated Partitions Number of under-replicated partitions.
Active Controller Count Number of active controllers in the Kafka cluster.
Offline Partitions Count Number of partitions that are offline.
Leader Election Rate Rate of leader elections in the cluster.

Setup

Quick installation

If you're using Linux servers, use the Kafka plugin installer that checks the prerequisites and installs the plugin with a bash script. You don't need to manually set up the plugin if you're using the installer.

Execute the command below in the terminal to run the installer and follow the instructions displayed on-screen:
wget https://raw.githubusercontent.com/site24x7/plugins/master/kafka/installer/Site24x7KafkaPluginInstaller.sh && sudo bash Site24x7KafkaPluginInstaller.sh

Standard installation

If you're not using Linux servers or want to install the Kafka plugin manually, follow the steps below.

Prerequisites

Ensure you meet the following requirements before installing the plugin:

  • Download and install the Site24x7 server monitoring agent (Linux | Windows) on the server where you intend to run the plugin.
  • Ensure you have Python 3 or a higher version installed in your server.
  • Enable the Kafka JMX ports.
    • To enable it, find the following code block in the kafka-server-start.sh script:
      if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
      export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
      fi

      Below the aforementioned code block, paste the following lines:

      export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -
      Dcom.sun.management.jmxremote.authenticate=false -
      Dcom.sun.management.jmxremote.ssl=false -
      Dcom.sun.management.jmxremote.port=9999"
      export JMX_PORT=9999

      Restart the Kafka Broker service after making the above changes.

  • Install the jmxquery module for Python3.
    pip install jmxquery

Installation

  • Create a folder named kafka.
  • Download the kafka.py and the kafka.cfg files from our Github repository and place them in the kafka folder.
    wget https://raw.githubusercontent.com/site24x7/plugins/master/kafka/kafka.py
    wget https://raw.githubusercontent.com/site24x7/plugins/master/kafka/kafka.cfg
  • To check if the plugin is working, execute the command below with appropriate arguments, and check for a valid JSON output with applicable metrics and their corresponding values. file:
    python3 kafka.py --kafka_host "KAFKA_BROKER_HOST_IP" --kafka_jmx_port "KAFKA_BROKER_PORT_NO" --kafka_server_port "KAFKA_SERVER_PORT_NO" --kafka_home "KAFKA_HOME" --kafka_group_name "KAFKA_GROUP_NAME" --kafka_topic_name "KAFKA_TOPIC_NAME"
  • Add the following configuration in kafka.cfg file:
    [kafka_instance]
    kafka_host="localhost"
    kafka_jmx_port=9999
    kafka_server_port=9092
    kafka_topic_name="quickstart-events
    kafka_home="/home/users/kafka"
    kafka_group_name="console-consumer"
  • Follow the steps in this article to learn how to run the python script on a Windows server. You don't need to do this for Linux.
  • Move the kafka_producer_monitoring folder with the individual files within to the Site24x7 server monitoring plugin directory.

    For Linux: /opt/site24x7/monagent/plugins/

    For Windows: C:\Program Files (x86)\Site24x7\WinAgent\monitoring\plugins\

The agent will automatically execute the plugin within five minutes and send performance data to the Site24x7 data center.

To view performance charts and set thresholds for the various performance metrics:

  • Log in to Site24x7.
  • Navigate to Plugins > click the required monitor.

Plugin Contribution

Feel free to contribute to our existing plugin and come up with suggestions or feedback on our Community.

  • Supported Platforms:
  • Python Version: 2.7 & above
  • Linux Agent Version:15.0.0 & above

Over 13,000 actively paying customers trust Site24x7

Brand logos of our various customers

Check out our 100+ plugin integrations, or build your own

Choose from our list of more than 100 ready-to-use plugins and monitor your full stack of applications across web servers, databases, load balancers, and more.

MySQL Monitoring

MySQL
HAProxy Monitoring

HAProxy
Oracle Monitoring

Oracle
MongoDB Monitoring

MongoDB
Wordpress Monitoring

Wordpress
Tomcat Monitoring

Apache Tomcat
Elasticsearch Monitoring

Elasticsearch
Apache Monitoring

Apache
Dell iDRAC Monitoring

Dell iDRAC
NGINX Monitoring

NGINX and NGINX Plus
See all plugin integrations
Start 30-day free trial Try now, sign up in 30 seconds