Senior Data Engineer - Streaming - The Coca-Cola Company

Overview

A high number of candidates may make applications for this position, so make sure to send your CV and application through as soon as possible.

In this role you will design and codify enterprise streaming patterns to shift Coca-Cola’s data estate toward a continuous, event-driven model. You’ll build reusable frameworks and templates that enable teams to deploy real-time data streams on Microsoft Fabric, with a focus on Kappa patterns and end-to-end reliability. You will act as the internal authority on streaming design, driving scalable ingestion and robust backfill, with dashboards to monitor health metrics. This is a high-impact opportunity to shape how real-time data supports business SLAs and decision-making.

Compensation / Benefits
• medical benefits
• financial benefits
• annual incentive/support
• growth culture
• inclusive and agile environment
• remote/hybrid options (implied by large company practices)

Responsibilities
• Build a reusable streaming architecture playbook emphasizing Kappa patterns and near-real-time analytics on Fabric
• Architect, document, and templatize replay/backfill mechanisms (offset reset, checkpoint/state management, dedupe strategies)
• Develop architectural blueprints and guardrails for sub-second/near-real-time latency aligned with business SLAs
• Package advanced streaming concepts (stateful processing, watermarking, late-arriving data remediation, complex windowing) into reusable templates
• Codify end-to-end idempotency with deterministic replay and effective-once delivery
• Create metadata-driven frameworks that tolerate upstream schema drift without disrupting downstream streaming queries
• Serve as internal authority on streaming design across Kafka/Event Hubs for scalable Fabric ingestion
• Develop standardized dashboards to monitor end-to-end data lag, throughput, backpressure, and DLQ routing

Key requirements
• 8+ years in data platform architecture
• 3+ years architecting/operating production-grade Streaming and Kappa architectures
• 4+ years Spark Streaming and Apache Kafka xklufbp pipelines (topic partitioning, consumer groups, schema registries, high-throughput tuning)
• Deep familiarity with Spark Structured Streaming, Kafka Streams, or Apache Flink
• Strong Scala and/or PySpark experience with stateful Structured Streaming
• Advanced knowledge of Delta Lake append-only sinks and checkpointing
• Framework mindset with developer-facing frameworks or configurable streaming engines
• Ability to articulate complex streaming concepts to developers and executives
• strong communication
• cross-functional collaboration
• problem-solving
• Apache Kafka
• Spark Streaming
• Kafka Streams