Gcp Data Engineer

Detalles de la oferta

.Responsibilities:Build, maintain, and optimize end-to-end ETL pipelines using Google Cloud Dataflow, Apache Beam, Cloud Composer, and Cloud Functions.Automate data ingestion, transformation, and integration processes from multiple sources (structured and unstructured data) into cloud-based data platforms.Design and implement scalable cloud-based data architectures using services like Google BigQuery, Google Cloud Storage (GCS), Cloud Pub/Sub, and Cloud Dataproc.Architect data solutions that are reliable, performant, and cost-effective for large-scale data processing and storage needs.Build and manage data lakes using Google Cloud Storage and ensure data is properly ingested, stored, and processed in an organized and secure manner.Implement Google BigQuery data warehouses, optimize data models for performance, and ensure they are scalable to handle large datasets.Leverage Google Cloud Dataflow or Apache Beam to transform data into formats suitable for reporting and machine learning models.Integrate diverse data sources such as relational databases, NoSQL databases, APIs, and flat files into the data pipeline architecture.Automate the deployment of data pipelines, processes, and infrastructure using Google Cloud Deployment Manager, Terraform, or Cloud Composer.Design and manage CI/CD pipelines for efficient deployment of data engineering solutions.Implement best practices for data governance, ensuring data privacy and compliance with security standards (e.G., encryption, data masking, and access control).Manage permissions and security with Google IAM, and ensure that sensitive data is securely handled and stored.Continuously monitor the performance of data pipelines and storage solutions, identifying opportunities for optimization and cost reduction.Troubleshoot and resolve data integration and pipeline issues to ensure smooth and efficient data processing.Collaborate with cross-functional teams (data scientists, analysts, business stakeholders) to understand data requirements and deliver high-quality data solutions.Provide support for data-related questions, troubleshooting, and assist with data visualization and reporting tasks.Document data engineering processes, including pipeline architectures, configurations, and best practices.Prepare and deliver regular updates on data engineering projects and their progress.What you'll bring:Experience in building ETL pipelines using Cloud Dataflow, Apache Beam, Google Cloud.Ability to integrate structured and unstructured data sources into cloud-based systems.Strong experience working with Google BigQuery for building and managing data warehouses and running optimized analytical queries.Expertise in data ingestion, transformation, and integration using GCP tools, including Google Cloud Pub/Sub, Cloud Dataflow, and BigQuery.Experience with Apache Spark and PySpark is a plus for handling large-scale data processing


Salario Nominal: A convenir

Fuente: Jobtome_Ppc

Requisitos

Becario De Sistemas

GIA EVOLUTION S.A DE C.V SOLICITABECARIO DE SISTEMASEmpresa dedicada a manufactura de suplementos alimenticios solicita becario de apoyo en el área de Sistem...


Gia Evolution S.A. De C.V. - Aguascalientes

Publicado 5 days ago

Coordinador/A

Nos encontramos en la búsqueda de un/a Coordinador/a Administrativo que se encargue de organizar y supervisar las tareas administrativas de nuestra empresa.¡...


Servicios Operativos Del Atlantico - Aguascalientes

Publicado 5 days ago

Ie Maintenance Technician

.Your tasksMantener en condiciones iniciales los equipos, mecanismos, cargadores y micrologística a cargo, mediante la ejecución de planes de mantenimiento p...


Continental - Aguascalientes

Publicado 5 days ago

Especialista De Servicios (Temporal 6 Meses)

Especialista de Servicios al personal para atención de plantas vehículos y motor. Manejo de proveedores, acceso de proveedores de limpieza, jardinería, insum...


Nissan - Aguascalientes

Publicado 5 days ago

Built at: 2024-12-23T03:03:18.738Z