Career Level - IC4
**Responsibilities**:
Data Engineering:
1. Design, develop, and maintain scalable data pipelines and ETL processes to collect, process, and store large volumes of data from diverse sources.
2. Implement and optimize distributed computing frameworks such as Apache Hadoop, Apache Spark, or Apache Flink to handle big data processing tasks efficiently.
3. Collaborate with cross-functional teams to understand data requirements and deliver innovative solutions for data ingestion, transformation, and storage.
4. Monitor and troubleshoot data pipeline performance issues, implementing solutions to improve reliability and efficiency.
Data Warehousing:
1. Design dimensional data models and database schemas optimized for analytical queries and reporting needs.
2. Develop and maintain ETL workflows to extract, transform, and load data from source systems into the data warehouse, ensuring data quality and consistency.
3. Implement and optimize data warehouse architectures and indexing strategies for efficient data storage and retrieval.
4. Collaborate with business analysts and stakeholders to understand reporting and analytics requirements and translate them into technical solutions.
Qualifications:
Technical Skills:
1. Proficiency in programming languages such as Python with experience in SQL databases.
2. Hands-on experience with big data technologies such as Apache Spark, or Apache Flink, and familiarity with related ecosystem tools.
3. Experience designing and implementing dimensional data models for data warehousing solutions.
4. Knowledge of ETL tools and techniques for data integration and transformation.
Analytical Skills:
1. Strong analytical and problem-solving skills, with the ability to understand complex data requirements and translate them into technical solutions.
2. Experience working with stakeholders to gather and analyze business requirements and deliver solutions that meet their needs.
Communication and Collaboration:
1. Excellent communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.
2. Ability to communicate technical concepts to non-technical stakeholders and collaborate with colleagues from diverse backgrounds.
Preferred Qualifications:
1. Experience with cloud platforms such as OCI, AWS, Azure, or Google Cloud Platform, and proficiency in cloud-based data services.
2. Familiarity with any one of the data visualization tools such as OBIEE, Tableau, Power BI.
3. Certification or training in relevant data engineering and data warehousing technologies.