Senior Kafka Engineer

We are seeking a highly skilled and experienced Senior Integration Engineer to join our dynamic team. This role will be pivotal in designing, developing, and maintaining robust integration solutions, with a strong focus on leveraging Apache Kafka for real-time data streaming and AWS integration services, particularly AWS Glue for ETL and data cataloging. The ideal candidate will have a deep understanding of enterprise integration patterns, event-driven architectures, and cloud-native solutions. You will collaborate closely with various engineering, product, and business teams to ensure seamless and efficient data flow across our diverse system landscape.

 

Key Responsibilities:

• Design, develop, implement, and maintain scalable and resilient integration solutions using Apache Kafka and its ecosystem (Kafka Connect, Kafka Streams, Schema Registry).
• Monitor, troubleshoot, and optimize Kafka cluster performance, including throughput, latency, and resource utilization, to ensure stability and efficiency.
• Architect and build ETL/ELT pipelines using AWS Glue for data extraction, transformation, and loading from various sources into data lakes, data warehouses, and other target systems.
• Develop and manage AWS Glue Data Catalog and ensure data quality, consistency, and governance across integrated systems.
• Integrate a variety of cloud-based applications, databases, and third-party services using appropriate AWS services (e.g., Lambda, S3, API Gateway, SNS, SQS, EventBridge) and other integration technologies.
• Champion and implement best practices for event-driven architecture, microservices integration, and API development.
• Collaborate with architects, data engineers, software developers, and business stakeholders to understand integration requirements and translate them into technical designs and solutions.
• Optimize and troubleshoot existing integration solutions to improve performance, reliability, and scalability.
• Develop and maintain comprehensive documentation for integration designs, processes, and data flows.
• Implement robust monitoring, alerting, and logging for integration solutions to ensure operational excellence.
• Mentor junior engineers and provide technical leadership within the integration domain.
• Stay current with emerging technologies and industry trends in data integration, Kafka, and AWS services.
• Ensure security best practices are implemented and maintained for all integration solutions.

Required Qualifications:

• Bachelor's degree in computer science, Engineering, Information Technology, or a related field (or equivalent practical experience).
• 5+ years of hands-on experience in designing and developing enterprise-level integration solutions.
• Proven, in-depth experience with Apache Kafka, including designing and implementing Kafka producers, consumers, topics, connectors, and performance tuning.
• Strong experience with AWS cloud services, particularly AWS Glue for ETL, data cataloging, and job orchestration.
• Proficiency with other AWS integration and data services such as S3, Lambda, API Gateway, Kinesis, Redshift, RDS, DynamoDB.
• Solid understanding of data modeling, data warehousing concepts, and ETL/ELT processes.
• Experience with various data formats (e.g., JSON, Avro, Parquet, XML) and API protocols (e.g., REST, SOAP, gRPC).
• Proficiency in one or more programming languages commonly used in integration development (e.g., Python, Java, Scala).
• Experience with SQL and NoSQL databases.
• Strong understanding of integration patterns, event-driven architecture, and microservices principles.
• Experience with version control systems (e.g., Git) and CI/CD pipelines.
• Excellent analytical, problem-solving, and troubleshooting skills.
• Strong communication and interpersonal skills, with the ability to collaborate effectively with technical and non-technical teams.

Preferred Qualifications:

• Confluent Certified Developer for Apache Kafka (CCDAK) or similar Kafka certification.
• AWS Certifications (e.g., AWS Certified Data Analytics - Specialty, AWS Certified Developer - Associate/Professional, AWS Certified Solutions Architect - Associate/Professional).
• Experience with other ETL tools (e.g., Apache Airflow, Talend, Informatica).
• Familiarity with containerization technologies (e.g., Docker, Kubernetes).
• Experience with Infrastructure as Code (IaC) tools (e.g., Terraform, AWS CloudFormation).
• Knowledge of data governance and data security best practices.

 

We thank you for your interest. Only those selected for an interview will be contacted.

GFL is committed to equal opportunity for all, without regard to race, religion, color, national origin, citizenship, sex, sexual orientation, gender identity, age, veteran status, disability, genetic information, or any other protected characteristic. If you are interested in applying for employment and need special assistance or an accommodation to apply for a posted position, please contact myworkdayrecruitment@gflenv.com