Apache Kafka Development

Description:

This comprehensive 4-day training equips participants with the skills needed to develop, deploy, and manage real-time data streaming applications using Apache Kafka. Starting with an overview of Big Data architectures and the role of Kafka in modern data pipelines, participants will explore Kafka’s publish-subscribe model, distributed architecture, and core components such as producers, consumers, topics, partitions, and offsets.

The training dives deep into advanced development topics, including the Kafka Streams API for real-time data processing, and Kafka Connect for integrating Kafka with relational databases, cloud storage systems, and other external platforms. Learners will gain hands-on experience building Java-based Kafka applications, managing offsets and delivery guarantees, and constructing end-to-end streaming pipelines.

Beyond application development, the course also addresses Kafka’s operational aspects, such as cluster management, topic configuration, monitoring, and performance tuning. It concludes with an in-depth look at Kafka security—covering SSL, SASL, ACLs, and encryption strategies—along with Zookeeper administration and troubleshooting techniques for production environments. Real-world examples, best practices, and guided labs ensure learners are ready to build and support scalable, secure, and reliable Kafka-based systems in enterprise environments.

Duration: 4 Days

Course Code: BDT 507

Learning Objectives:

By the end of this course, participants will be able to:

1. Understand Big Data architecture and Kafka’s role in real-time processing.
2. Deploy Kafka clusters and work with Producers, Consumers, and Streams.
3. Integrate Kafka with databases and storage using Kafka Connect.
4. Secure Kafka clusters with SSL/SASL and implement access control.
5. Monitor, troubleshoot, and scale Kafka deployments for production workloads.

This course is ideal for:

1. Data Engineers and Software Developers
2. DevOps and Platform Engineers managing Kafka clusters
3. Streaming and Real-Time Analytics Professionals
4. Architects designing event-driven systems

1. Familiarity with Java (or any another programming language)
2. Basic understanding of distributed systems
3. Exposure to Linux command line

Course Outline:

Module 1: Big Data and Kafka Introduction

1. 3Vs of Big Data and evolution
2. Real-time vs batch processing
3. Industry use cases

Module 2: Overview of Hadoop & Streaming Frameworks

1. Apache Hadoop: HDFS and MapReduce overview
2. Spark and Storm for real-time data analysis

Module 3: Introduction to Zookeeper

1. Zookeeper basics and recipes (leader election, locking)
2. Installing/configuring Zookeeper
3. Hands-on: Ubuntu setup, SSH, and connecting with PuTTY

Module 4: Introduction to Apache Kafka

1. Kafka design principles
2. Kafka as a distributed commit log
3. Partitions, topics, replication, and message queues
4. Publish-subscribe pattern explained

Module 5: Kafka Installation and Configuration

1. Installing Kafka and configuring brokers
2. Cluster setup using Zookeeper
3. Hands-on: Installing and configuring Kafka + Zookeeper

Module 6: Kafka Producers and Consumers

1. Producer API and serializers (Avro, JSON, Protobuf)
2. Consumer API, groups, offsets, and delivery semantics
3. At-most-once, at-least-once, exactly-once guarantees

Module 7: Kafka Data Model and Java Integration

1. Topics, partitions, and offsets in practice
2. Compiling and running Kafka Java clients
3. Hands-on: Writing Kafka Producer and Consumer apps

Module 8: Kafka Streams API

1. Difference between batch and stream processing
2. Kafka Streams API: KStreams, KTables, stateful operations
3. Event-time processing and windowing
4. Hands-on: Real-time word count and stream joins

Module 9: Kafka Connect and Data Integration

1. Kafka Connect architecture
2. Source/sink connectors: JDBC, S3, HDFS, Elasticsearch
3. Building ETL-style pipelines using Kafka Connect
4. Hands-on: Configuring Kafka Connect with databases

Module 10: Real-World Kafka Development Lab

1. Build a full pipeline from producer → Kafka → stream processor → sink
2. Hands-on: End-to-end pipeline using Kafka Streams + Connect

Module 11: Kafka Cluster Operations

1. Broker lifecycle: adding/removing brokers
2. Topic configuration, replication, and leader election
3. Monitoring clusters using metrics and logs

Module 12: Kafka Security

1. Authentication with SSL and SASL
2. Access control with ACLs
3. Encryption at rest and in transit
4. Hands-on: Securing Kafka brokers and clients

Module 13: Kafka Performance Tuning

1. Tuning producers and consumers for throughput
2. Compression, batching, and partitioning strategies
3. Broker-level tuning

Module 14: Zookeeper for Kafka Operations

1. Zookeeper’s role in Kafka cluster metadata
2. Best practices and operational tips
3. Securing Zookeeper nodes

Module 15: Monitoring and Troubleshooting

1. Tools: Kafka Manager, Grafana, Prometheus
2. Common issues: unbalanced partitions, lag, GC, leader election issues
3. Hands-on: Benchmarking performance, tuning parameters, viewing metrics

Training Material Provided:

Course slides and reference guides