.EY-GDS- DnA Consulting – Data Engineer Azure-Snowflake (Batch)As part of our EY- GDS-DnA Consulting team, you will help clients use various methods to transform raw data into useful data systems.
You will align data systems with business goals.
You must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products.
The ideal candidate should have 3-5 years of experience in Data Engineering, KAFKA, Advanced SQL, Advanced Python programming, Spark, ETL and ELT Process knowledge, Data structures knowledge, MPP Databases (Teradata, Snowflake), Blob Storage, Azure Data Lake Storage, and AZURE cloud.
The Data Engineer will interact with different members of the team such as database architects, data analysts, and data scientists on data initiatives and ensure optimal data delivery architecture is consistent with business strategy.
Our clients usually have their headquarters in the USA, however they span across multiple industries, regions, and countries.The opportunityWe are looking for a Senior Data Engineer with expertise in developing and managing Kafka-based data pipelines, implementing complex stored procedures, and best practices with data warehouse and ETL concepts, and deploying fully operational data warehouse solutions into production on Snowflake to join our GDS Mexico team.
This is a fantastic opportunity to be part of a leading firm whilst being instrumental in the growth of a new service offering.Your key responsibilitiesDesign and develop batch and streaming data processing.Build data systems and pipelines.Evaluate business needs and objectives.Prepare data for prescriptive and predictive modeling.Build algorithms and prototypes.Combine raw information from different sources.Explore ways to enhance data quality and reliability.Identify opportunities for data acquisition.Monitor and optimize data storage and data processing.Design and implement data security.Skills and attributes for success3-5 years of experience on Apache Kafka, Kafka stream, AZURE Event Hub.3-5 years of experience in processing real-time data.Advanced knowledge in processing large amounts of data (billions of records) in minutes.Deep knowledge of producers (ACK/NACK), consumers, and brokers.Deep knowledge of topics, partitions, and segments.3-5 years of advanced knowledge in SQL, Python programming, ETL, and ELT Process.Advanced knowledge of data structures, MPP Databases, and Blob Storage.3-5 years of experience with AZURE data lake Storage (Teradata, Snowflake), AZURE cloud.Previous experience as a data engineer or in a similar role.Data engineering certification (AZURE Certified Data Engineer) is a plus.Flexible and adaptable; able to work in ambiguous situations.Able to work effectively at all levels in an organization.Must be a team player and able to work collaboratively with and through others.Must enjoy learning.To qualify for the role, you must haveBachelor's in technology or engineering or similar