Project overview: The Advanced Analytics management team is a transformational team whose objective is to generate measurable value that lasts over time across all areas of the company. To achieve this, custom data products are created based on analytical models, mostly from ML but being able to use a broader spectrum of techniques; either by delivering new analytical capabilities to the areas we work with or by directly interacting with their systems.
- The development of the products and the associated change management is carried out using Agile methodologies (development) that incorporate the scientific method and LEAN tools (change management). To ensure the correct execution of our products, we work with an MLOps environment.
Position overview: The role of the Data Engineer is essential for our client's Advanced Analytics team. It is them who allow us to feed our Data Products with data at different stages of the life cycle of their construction. It can be from an ungoverned source, so there is no certainty will be sufficient to generate the desired analytics value, or fully governed sources that already belong to the company's Data Lake. The intake to the Data Products can be in batches or streamed in. This is a role with great technical responsibility! You should feel comfortable working in a terminal shell in order to implement a scalable, usable, functional and decoupled solution. You will need to work closely with Data Scientists and ML Engineers to automate the various tasks in robust and resilient data pipelines. Data Engineers are essentially creators, their natural habitat is data, and that is where they perform to their limit, always working as a team, with a great collaborative spirit.
- Responsibilities: Collection of large volumes and varied data sets
- Work directly with the team of Data Scientists and ML Engineers creating data pipelines robust and resilient that allow feeding the Data Products
- Collection of interactions with real ML models for their subsequent retraining
- Collaborate in the design, implementation, and production of the necessary parts for the ingestion of data into our Data Lake
- Promote a technical culture promoting data products with MLOps best practices
- Work directly with the team that supports the operation of the Data Products, to generate innovative ideas and solutions that improve their production processes
- Define data models that unite disparate data across the organization
- Design, implement, and maintain ETL pipelines
- Design, develop, and continuously test solutions based on the data
- Requirements: At least 2 years of experience in work environments as a Software Engineer or Data Engineer
- Solid experience with Python
- Experience working in UNIX environments
- Excellent analytical skills related to working with unstructured data sets, and advanced knowledge of SQL, including query optimization
- Strong understanding of data structures
- Experience using CI/CD pipelines
- Experience working with Docker
- Passion in data processing problems
- Experience in distributed processing using cloud services
- Understanding of the complete lifecycle of Machine Learning models
- Advanced oral and written English
- Experience in the construction and optimization of data pipelines, message queues, and big architectures and highly scalable data
- Experience testing data pipelines
- Excellent communication skills and collaborative work experience
- Nice to have: Terraform experience
- Experience in building scalable APIs
- Experience configuring and optimizing CI/CD pipelines
- Stream processing experience
- Experience with GCP, especially the machine and data processing suite of services learning
- Ability to visualize possible improvements, problems, and solutions for architectures