Вакансія Senior Data Engineer (Poland)

Skills

• Python
• Pandas
• Spark SQL
• PySpark
• Azure Data Factory
• Azure Databricks
• Microsoft Azure
• Azure Data Lake
• Azure Data Lake Storage
• Machine learning
• TensorFlow
• scikit-learn

Опис вакансії

Impressit is looking for a highly skilled Senior Data Engineer with expertise in Python and Big Data technologies for our British customer - one of the world's largest oil and gas companies. The ideal candidate will have a strong track record in data warehousing, extensive experience with Azure cloud services, and proficiency in working with large datasets. The role will primarily focus on designing, developing, and maintaining data pipelines to support our analytical and operational needs.

What you will do:

• Design, develop, and maintain scalable data pipelines using SparkSQL, PySpark, Azure Data Factory, and Azure Databricks.
• Utilize Spark for Delta Live Tables to enable real-time data processing and analysis.
• Collaborate with cross-functional teams to understand data requirements and design optimal solutions.
• Implement efficient data ingestion, transformation, and storage processes to ensure data quality and integrity.
• Optimize performance and reliability of data pipelines to meet SLAs and business objectives.
• Develop and maintain API frameworks for Python to facilitate data access and integration.
• Stay abreast of emerging technologies and best practices in data engineering and contribute to continuous improvement initiatives.

What we expect:

• 5+ years of hands-on experience as a Data Engineer with a focus on data warehousing and cloud technologies.
• Strong proficiency in Python programming and experience with PANDAs for data manipulation.
• Extensive hands-on experience with SparkSQL, PySpark, Azure Data Factory, and Azure Databricks.
• Solid understanding of Big Data concepts and architectures, including distributed computing and parallel processing.
• Experience working with large datasets and implementing scalable solutions.
• Familiarity with Azure cloud services such as Azure Data Lake and Azure Data Lake Storage.
• Excellent problem-solving skills and ability to troubleshoot complex data issues.
• Strong communication and collaboration skills with the ability to work effectively in a team environment.
• Upper-intermediate or a higher level of English.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

As a plus:

• Certification in Azure cloud services or Big Data technologies.
• Knowledge of machine learning concepts and frameworks.