Required Qualifications:5+ years of experience in data engineering using Python with a focus on AWS S3, EMR, Glue, Step Functions, Apache NiFi and Spark.Proven track record of building scalable data pipelines in cloud environments.Proficiency in flow design, processors, and data provenance in Apache NiFi.Strong expertise in Spark, Hadoop, and distributed computing on AWS EMR.In-depth knowledge of AWS services (S3, Glue, Redshift, RDS, Lambda, Step Functions).Experience with data formats (JSON, CSV, Parquet, Avro) and transformation techniques.Strong problem-solving skills and ability to troubleshoot complex data processing issues.Excellent communication skills with the ability to document and explain technical details clearly.Preferred Qualifications:AWS Certified Solutions Architect or Data Analytics Specialty.Experience with data governance frameworks and compliance requirements.Familiarity with CI/CD pipelines and version control (GitLab, Jenkins).Key Responsibilities:Design & Develop Data Pipelines:Architect and implement end-to-end data pipelines using AWS S3, EMR, Glue, Step Functions, Apache NiFi, Spark.Manage data ingestion processes from AWS S3, ensuring secure and efficient data transfer.Implement initial data routing, validation, and transformations using Apache NiFi processors and Spark Data Engines.Data Processing & Transformation:Integrate using AWS EMR, Apache NiFi, Spark to perform complex data transformations and analytics.Optimize Spark jobs for processing large-scale datasets with a focus on performance and resource utilization.Handle both historical and incremental data loads, ensuring data consistency and integrity.Data Storage & Management:Define and implement data storage strategies across S3, RDS, and Redshift, adhering to business requirements.Manage data catalog creation and schema management using AWS Glue.Automation & Orchestration:Develop and manage workflows using Apache Airflow, AWS Step Functions to automate data processing tasks.Implement monitoring, error handling, and retries within the orchestration framework.Security & Compliance:Ensure data security with encryption (AES-256, TLS) and IAM role-based access controls.Implement data governance policies using AWS Glue Data Catalog to ensure compliance with regulatory requirements.Performance Monitoring & Optimization:Utilize AWS CloudWatch to monitor the performance of EMR clusters, NiFi flows and data storage.Continuously optimize Spark job configurations and NiFi data flows for maximum throughput and minimal latency.#J-18808-Ljbffr
