Senior Data Engineer (Databricks & AI Platforms) (4809)

Descrição da vaga

Come work for a large global financial and insurance products company! This is your chance !!

Start a successful career in a renowned company in the international market! Great opportunity!

Global insurance and asset management company seeks a responsible, organized, dynamic and team-oriented person.

Responsabilidades e atribuições

The Senior Data Engineer is responsible for designing, building, and optimizing scalable data platforms and pipelines that support analytics, business intelligence, machine learning (ML), and AI-driven solutions. 

This role partners with data scientists, AI engineers, architects, and business stakeholders to deliver trusted, high-quality data products using Databricks, cloud technologies, and modern data engineering practices. The position also serves as a technical leader in enabling enterprise AI and Generative AI initiatives through robust, secure, and governed data platforms.

Key Responsibilities:

Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL;

Build and optimize batch, streaming, and real-time data integration solutions from enterprise and third-party data sources;

Implement Databricks Lakehouse architectures utilizing Delta Lake and Medallion design patterns;

Develop and maintain data products that support analytics, predictive modeling, machine learning, and Generative AI applications;

Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI and ML use cases;

Design and implement feature engineering pipelines and support ML lifecycle processes;

Develop data solutions that support Retrieval Augmented Generation (RAG), vector search, semantic search, and LLM-based applications;

Optimize Spark jobs, SQL workloads, and data processing frameworks for performance, scalability, and cost efficiency;

Implement data quality, observability, lineage, governance, and monitoring capabilities;

Ensure compliance with data privacy, security, and responsible AI standards;

Contribute to CI/CD, Infrastructure-as-Code, and DataOps practices across the data platform;

Mentor junior engineers and promote engineering best practices across the organization.

Requisitos e qualificações

Required Qualifications:

Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field;

7+ years of experience in data engineering, ETL development, or large-scale data platform engineering;

3+ years of hands-on experience with Databricks and Apache Spark;

Strong proficiency in Python, SQL, and distributed data processing frameworks;

Experience building cloud-based data lakes, data warehouses, and Lakehouse architectures;

Experience supporting AI, machine learning, or advanced analytics initiatives;

Strong understanding of data modeling, data governance, and enterprise data management practices;

Experience developing and optimizing large-scale data pipelines in AWS, Azure, or Google Cloud.

Preferred Qualifications:

Experience with Databricks Delta Lake, Unity Catalog, Delta Live Tables, MLflow, Mosaic AI, and Databricks Workflows;

Hands-on experience supporting Generative AI, Large Language Models (LLMs), RAG architectures, vector databases, or AI-powered applications;

Familiarity with AI frameworks such as LangChain, Semantic Kernel, OpenAI APIs, Hugging Face, or similar technologies;

Experience with feature stores, model training pipelines, and machine learning operationalization (MLOps);

Experience with Kafka, Event Hub, or other streaming technologies;

Databricks Certified Data Engineer Professional or equivalent cloud certification.

Key Competencies:

Databricks Platform Engineering;

Apache Spark Development;

AI & Machine Learning Data Engineering;

Generative AI Data Solutions;

Lakehouse Architecture;

Data Modeling & Data Warehousing;

Data Governance & Security;

Data Observability & Reliability Engineering;

Cloud Data Platforms (AWS/Azure);

DataOps, CI/CD & Automation;

Technical Leadership & Mentoring.

Success Measures:

Delivery of scalable, reliable, and secure data platforms supporting analytics, AI, and business operations;

Successful enablement of AI and Generative AI use cases through high-quality, governed data products;

Improved data quality, pipeline reliability, and platform performance;

Increased automation, operational efficiency, and reuse of engineering frameworks;

Adoption of data engineering standards and best practices across development teams;

Measurable improvements in AI/ML solution delivery speed and business value realization;

Fluent english.

Informações adicionais

Modelo de contratação:

PJ.

Forma de atuação:

Híbrido (3x por semana presencial no escritório de Pinheiros/SP).



Somos uma empresa de consultoria em TI com mais de 10 anos no mercado e contamos com um time de especialistas em recrutamento tech. Nosso processo é 100% focado na experiência de quem tanto importa, o candidato.

Optamos por fazer a diferença e temos orgulho em dizer que todos que passam pela Keep Simple se sentem especiais. Possuímos um ambiente descontraído, colaborativo, e adotamos o ágil de verdade.

Faça parte da nossa história, #vemprakeep