Вакансия Databricks Senior Data Engineer

Skills

• DataBricks
• Snowflake
• Apache Spark
• PySpark
• Delta Lake
• MLflow
• AWS
• Azure
• Python
• SQL
• Scala
• Github
• Azure DevOps
• GitHub Actions
• Apache Airflow
• Jenkins
• Tableau
• Alteryx
• Azure Synapse
• Netezza

Описание вакансии

We are seeking an accomplished Senior Data Engineer to join our Dublin-based team. This role provides an exciting opportunity to influence our data architecture, working with innovative cloud technologies to drive impactful, data-centric projects. The ideal candidate will have in-depth experience with Databricks, Snowflake, AWS, and MLOps to support and enhance the deployment and scalability of machine learning models. You’ll play a pivotal role in ensuring data accessibility, optimising data sourcing pipelines, and enhancing the performance of large-scale data solutions.

Key responsibilities & duties include:

• Design and Implement Cloud-Native Data Solutions: Develop scalable, resilient data platforms using cloud-native technologies, data mesh frameworks, and integration across diverse data sources
• Build and Maintain MLOps Pipelines: Use tools like MLflow to create reliable, efficient pipelines for deploying machine learning models in production environments
• Establish Data Governance and Quality Standards: Develop and uphold data governance practices, ensuring robust data quality and compliance using tools like Databricks Unity Catalog
• Oversee Data Integration and Migration: Lead migration projects from legacy data systems to modern cloud platforms, focusing on optimising operational costs and efficiencies
• Performance Optimisation and Tuning: Use tools such as Snowflake and Delta Lake to enhance data accessibility, reliability, and performance, delivering robust, high-quality data products

Key Projects/Deliverables:

• Data Mesh Architecture: Design and deploy data mesh frameworks to support seamless data integration and scalability across business functions
• Operationalise MLOps Pipelines: Develop and manage MLOps pipelines to streamline machine learning model deployment and operational efficiency
• Data Migration and Cost Optimisation: Lead large-scale data migrations to Azure and AWS, focusing on business-critical data sources and substantial cost reductions
• Data Governance Applications: Create applications to reinforce data governance, quality, and enterprise standards, ensuring a secure and compliant production environment

Required Experience:

• Data Engineering Expertise: Proven experience in architecting and delivering large-scale, cloud-native data solutions
• Advanced Knowledge in Databricks and Snowflake: Hands-on experience in Databricks, Spark, Pyspark, and Delta Lake, with strong skills in data warehousing and lakehouse solutions
• MLOps Skills: Practical experience in MLOps, ideally with MLflow for model management and deployment
• Cloud Proficiency: Strong knowledge of AWS, with additional experience in Azure advantageous for multi-cloud setups
• Programming Proficiency: Advanced coding abilities in Python, SQL and Scala
• Tooling Competence: Familiarity with version control (GitHub), CI/CD tools (Azure DevOps, GitHub Actions), orchestration tools (Airflow, Jenkins), and dashboarding tools (Tableau, Alteryx)

Desirable Skills:

• Experience with Synapse Analytics, Netezza, and legacy data systems
• Knowledge of data governance best practices and tools
• Excellent problem-solving skills, with the ability to work both autonomously and collaboratively in cross-functional teams