Вакансії Data Engineer / Big Data Architect

We’re opening the position of a Lead Data Engineer to join our team aimed at developing a data warehouse solution for a big US food company. You will play a key role in shaping our data architecture, integrating data from multiple sources, and powering business-critical dashboards used across the company.
As a Lead Data Engineer, you will also be expected to guide the Development Team, making technical decisions and ensuring best practices are followed across data architecture, pipeline design, and performance optimization.

Responsibilities:

• Build a data mart design and documentation of sophisticated data architecture solutions, including data models, data integration patterns, and data storage mechanisms
• Collaborate with the Product Owner and the team to implement data solutions aligned with the organization’s strategic goals
• Work alongside Data Engineers to ensure seamless integration of data pipelines and data products with the overall data architecture
• Optimize data storage and retrieval processes to fully leverage Snowflake capabilities, focusing on performance, scalability, and cost efficiency
• Provide technical leadership, mentorship, and support to the Data Engineering Team
• Perform code reviews and ensure adherence to coding standards and best practices

Requirements:

• 5+ years in data engineering, data modelling, database design, data warehousing, and data governance practices
• Production experience with building, maintaining, and performance optimization of Snowflake DWH
• Proven experience in leading data projects, architecting and designing new solutions
• Experience in building and maintaining data pipelines
• Proficiency in SQL
• Ability to conduct code reviews and ensure adherence to coding standards
• Excellent problem-solving and analytical skills
• Strong communication and collaboration abilities
• English level – Upper-Intermediate or higher

Will be a plus:

• Experience with Dataiku for pipeline orchestration
• SnowPro® Certifications
• Experience with Azure data factory

The customer is a leading company specializing in digital solutions and services for the maritime industry. It aims to optimize and enhance shipping processes by offering a range of products that improve operational efficiency and transparency. Key areas include digital solutions, analytics, and consulting. It is committed to innovation and sustainable development, leveraging advanced technologies and best practices. The company prides itself on its international team of professionals dedicated to delivering high-quality and reliable solutions to clients.
The project is a comprehensive digital solution for managing maritime documentation, automating processes from agent appointments to final payments. It improves operational efficiency, regulatory compliance, and cost management, while also supporting communication, information sharing, vessel support, and payment services.

Responsibilities

• Developing, deploying, and optimizing LLM-based applications using Python and Azure OpenAI.
• Implementing Retrieval-Augmented Generation (RAG) pipelines for efficient data retrieval.
• Setting up and configuring inference parameters (like top-k, top-p, temperature, seed, penalty, etc.) to enhance model performance.
• Leveraging Azure Foundry for AI model integration and deployment.
• Working with cloud-based architectures to ensure scalable and efficient AI solutions.
• Collaborating with data scientists, ML engineers, and software developers to build robust AI-driven applications.
• Monitoring, troubleshooting, explaining, and enhancing AI models for improved accuracy and efficiency.

Requirements

• Experience of Python development for 3+ years, particularly in AI/ML projects.
• Experience with Azure OpenAI and Azure Foundry.
• Hands-on expertise with LLM deployments and inference parameters setup.
• Knowledge of Retrieval-Augmented Generation (RAG) techniques.
• Familiarity with vector databases (e.g., Milvus, Qdrant, Chroma, Pinecone) and embedding for RAG applications.
• Strong problem-solving skills and ability to work in a fast-paced environment.
• Level of English – from Upper-Intermediate and above.

Desired skills

• Experience with MLOps and CI/CD pipelines for AI models (e.g. MLFlow, Kubeflow, Azure ML).
• Understanding of model evaluation metrics and performance tuning.
• Familiarity with APIs and microservices architecture.
• Knowledge of REST API Python frameworks (e.g. FastAPI, Flask, DRF).

Big Data & Analytics is the Center of Excellence's data consulting and data engineering branch. Hundreds of data engineers and architects nowadays build data & analytics end-to-end solutions from strategy through technical design and proof of concepts to full-scale implementation. We have customers in the healthcare, finance, manufacturing, retail, and energy domains.
We hold top-level partnership statuses with all the major cloud providers and collaborate with many technology partners like AWS, GCP, Microsoft, Databricks, Snowflake, Confluent, and others.

If you are

• Experienced Big Data Engineer with at least 4 years focused on designing and building scalable data pipelines
• Well-versed in batch or streaming data processing
• Proficient in SQL and Python
• Experienced in data engineering on Google Cloud Platform (GCP)
• Skilled in tools like Apache Spark (GCP Dataproc), Cloud Dataflow, or Apache Beam
• Familiar with orchestration tools such as Apache Airflow, Cloud Composer, or Amazon MWAA
• Experienced with Google BigQuery and other GCP-native data services like Memorystore for Redis or CloudSQL
• Familiar with transformation tools like dbt or Dataform
• Aware of AWS services, including S3, Redshift, RDS, EMR, or Athena (as part of migration context)
• Confident in using cloud-agnostic technologies like PostgreSQL, MySQL, SQL Server, Kafka, and PySpark
• Knowledgeable in NoSQL technologies such as Cassandra and Aerospike
• Capable of working with Databricks@GCP and Snowflake@GCP workloads in cloud-to-cloud migration scenarios

And you want to

• Be part of a data-focused engineering team migrating large-scale systems from AWS to GCP
• Work with a modern technology stack, including BigQuery, Dataflow, Dataproc (Serverless), Cloud Run, and Composer
• Collaborate with experts on re-architecting and optimizing cloud-native data platforms
• Participate in the entire project lifecycle: from discovery and PoCs to MVPs and full production rollout
• Engage with customers ranging from global enterprises to innovative startups
• Continuously learn, share knowledge, and explore new cloud services
• Contribute to building a data platform that integrates batch, streaming, and real-time components
• Work in an environment that values technical ownership, code quality, and clean design

Наш клієнт, всесвітньо відома багатонаціональна технологічна компанія, розташована в Каліфорнії та входить у Велику п’ятірку технологічних компаній, розширює партнерство з EPAM.
Спеціалізація клієнта:

• побутова електроніка
• програмне забезпечення
• онлайн-послуги

Ви працюватимете з експертами та технологіями світового рівня з самого серця Кремнієвої долини, займаючись розробкою продуктів та послуг, якими щодня користуються мільярди людей.
Ми шукаємо досвідченого Lead Software Data Engineer для поповнення нашої команди

Чим ви будете займатися у цій ролі

• Розробка дата пайплайнів, задач для створення нових вихідних датасетів в рамках розробки нового функціоналу
• Адаптація існуючих дата пайплайнів для інтеграції з різноманітними платформами, інструментами та сервісами
• Правка помилок в коді та даних, які були згенеровані через неправильну логіку або реалізацію
• Швидкий аналіз та дослідження даних для допомоги продакт менеджмент команді
• Здійснення моніторингу і усунення виробничих проблем з наявними дата пайплайнами
• Розробка та впровадження якісних перевірок даних (контроль якості)
• Планування обсягу нових розробок: оцінювання необхідних ресурсів, планування часових рамок виконання завдань
• Підтримка чіткості процесів з обробки тікетів
• Покращувати завдання, додатки та систему, якою ми керуємо, у всіх аспектах: якість коду, складність, обслуговування, документація
• Комунікувати з іншими інженерами з обробки даних у команді, колегами та керівниками проєктів щодо статусу, проблем, оцінок, часових рамок тощо

Навички

• 5+ років практичного досвіду роботи із Big Data: Hadoop (HDFS, YARN або Mesos), Spark
• Знання та практичний досвід застосування SQL у контексті роботи із Big Data: Spark SQL, Hive QL
• Знання Spark
• Оптимізація планів виконання Spark за допомогою Spark UI
• Хороше знання Scala, включаючи фунціональні бібліотеки Scala (Cats)
• Знання основ пакетної обробки та принципів ETL у сховищах даних
• Досвід з оркестрацією даних та знання умов повноти даних
• Досвід з дата клінзінгом (обробка пошкоджених, некоректних та дубльованих даних)
• Знання міграції схем та еволюції наборів даних
• Володіння усною та письмовою англійською мовою не нижче рівня B2
• Уміння швидко навчатись та вивчати новий набір інструментів і технологій, що використовуються на проєкті: сервісні платформи, постачальники телеметрії, Spark-as-a-Service, система збору та багато іншого

Буде перевагою

• Розуміння ідей і принципів функціонального програмування
• Досвід написання та використання веб-сервісів
• Досвід роботи з Teradata, Vertica, Oracle або Tableau
• Досвід роботи зі Spark Streaming і Kafka
• Досвід роботи або знання наступних технологій: Apache Iceberg, Trino (Presto), Druid, Cassandra, Blob-сховища, наприклад AWS
• Досвід роботи зі Splunk
• Знання або досвід роботи з Azkaban або Airflow

We are hiring a Senior Full-Stack Software Developer. Our client team consists of frontend and backend developers, data engineers, data scientists, QA engineers, cloud engineers, and project managers.

Responsibilities

• Participate in requirements clarification and sprint planning sessions.
• Design technical solutions and implement them, inc ETL Pipelines
• Build robust data pipelines in PySpark to extract, transform, using PySpark
• Optimize ETL Processes
• Enhance and tune existing ETL processes for better performance, scalability, and reliability
• Writing unit and integration tests.
• Support QA teammates in the acceptance process.
• Resolving PROD incidents as a 3rd line engineer.

Must have skills

• Min 7 Years of experience in IT/Data
• Bachelor in IT or related field.
• Exceptional logical reasoning and problem-solving skills
• Programming: Proficiency in PySpark for distributed computing and Python for ETL development.
• SQL: Strong expertise in writing and optimizing complex SQL queries, preferably with experience in databases such as PostgreSQL, MySQL, Oracle, or Snowflake.
• Data Warehousing: Experience working with data warehousing concepts and platforms, ideally DataBricks
• ETL Tools: Familiarity with ETL tools & processes
• Data Modelling: Experience with dimensional modelling, normalization/denormalization, and schema design.
• Version Control: Proficiency with version control tools like Git to manage codebases and collaborate on development.
• Data Pipeline Monitoring: Familiarity with monitoring tools (e.g., Prometheus, Grafana, or custom monitoring scripts) to track pipeline performance.
• Data Quality Tools: Experience implementing data validation, cleansing, and quality framework

Nice to have

• Understanding of Investment Data domain.

We are seeking an experienced developer to create a high-performance, scalable, and flexible behavioral analytics engine platform.
You will be a key member of our team, responsible for the architecture, development, and optimization of core components for processing and analyzing large volumes of data [terrabytes].

Required professional experience:

• 7+ years of experience in developing analytics platforms or big data processing systems.
• Deep knowledge of programming languages such as Java, Scala, Python, Go, C++, or Rust.
• Experience with distributed systems and big data technologies [Kafka, Flink, Spark, Apache BEAM].
• Understanding of scalable system design principles and architectures for real-time data processing.
• Experience with NoSQL databases [Cassandra, MongoDB].
• Experience with OLAP databases [ClickHouse, StarRocks, Doris].
• Knowledge of SQL.
• Understanding of statistical methods and principles of data analysis.
• Experience with Kubernetes [Helm, ArgoCD].

Desired Skills:

• Experience with open table format [Apache Iceberg/Delta Lake/Hudi].
• Experience with cloud platforms [Google Cloud, AWS, Azure].
• Knowledge of data security methods and compliance with regulatory requirements [GDPR, CCPA].

Key Responsibilities:

• Design and develop the architecture of an behavioral analytics platform for real-time big data processing.
• Implement key engine systems [data collection, event processing, aggregation, prepare data for visualization].
• Optimize the platform performance and scalability for handling large data volumes.
• Develop tools for user behavior analysis and product metrics.
• Collaborate with data analysts and product managers to integrate the engine into analytics projects.
• Research and implement new technologies and methods in data analysis.

Boosty Labs is one of the most prominent outsourcing companies in the blockchain domain. Among our clients are such well-known companies as Ledger, Consensys, Storj, Animoca brands, Walletconnect, Coinspaid, Paraswap, and others.
About project: Advanced blockchain analytics and on-the-ground intelligence to empower financial institutions, governments & regulators in the fight against cryptocurrency crime

Requirements:

• 3+ years of experience in data engineering or a similar role
• Strong programming skills in Python
• Solid hands-on experience with Apache Kafka for real-time data streaming
• Experience working with ClickHouse or other columnar databases
• Understanding of Data Lake architecture and cloud data storage solutions
• Familiarity with Argo Workflows or similar workflow orchestration tools (e.g., Airflow, Prefect)
• Experience with CI/CD processes and containerization (Docker, Kubernetes) is a plus
• Strong problem-solving skills and the ability to work independently

Responsibilities:

• Design and implement scalable, efficient, and reliable data pipelines
• Work with real-time and batch data processing using Kafka and ClickHouse
• Develop and maintain ETL/ELT processes using Python
• Manage and optimize data storage in cloud-based Data Lake environments
• Use Argo Workflows to orchestrate complex data workflows
• Collaborate with data scientists, analysts, and engineering teams to support their data needs
• Ensure data quality, consistency, and governance throughout the pipeline

We’re looking for a Lead Big Data Engineer to join our client, a prominent American ebook and audiobook subscription service with a vast library of over 200 million titles.
As a Lead Big Data Engineer, you will play a key role in designing, developing, and maintaining large-scale data processing systems, while providing technical leadership and mentoring to other teams.
This remote position, ideally suited for candidates located in Europe or the US time zone, is perfect for someone with a deep technical background in big data technologies, looking to drive innovation in data engineering.

About the project

Client is an American e-book and audiobook subscription service that includes one million titles. The platform hosts 60 million documents on its open publishing platform.
Core Platform provides robust and foundational software, increasing operational excellence to scale apps and data. We are focused on building, testing, deploying apps and infrastructure which will help other teams rapidly scale, inter-operate, integrate with real-time data, and incorporate machine learning into their products. Working with our customers in the Data Science and Content Engineering, and our peers in Internal Tools and Infrastructure teams we bring systems-level visibility and focus to our projects.
Client’s goal is not total architectural or design perfection, but rather choosing the right trade-offs to strike a balance between speed, quality and cost.

• Tech Stack: Scala and/or Python, Apache Spark/PySpark/Spark SQL DSL, AWS (EC2, IAM, S3, Glue, EMR), AWS data platform services, Datalake/Lakehouse, SQL and NoSQL, Apache Airflow, Terraform, Jenkins
• Team Composition: A distributed team consisting of 9 KITRUM Big Data Engineers (Middle/Senior)

Must-have for the position

• 5+ years Scala and/or Python;
• 3+ years Spark/PySpark/Spark SQL;
• Experience with AWS (EC2, IAM, S3, Glue, EMR);
• Fluency in at least one dialect of SQL and NoSQL;
• Strong technical and team leadership experience;
• English Level: Upper-Intermediate English or higher.

Will be a strong plus

• Experience with Apache Airflow, including writing plugins and custom operators.
• Experience with Terraform;
• Strong grasp of AWS data platform services and their strengths/weaknesses;
• Strong experience using Jira, Slack, JetBrains IDEs, Git, GitLab, GitHub, Docker, Jenkins;

Responsibilities

• Lead the design, development, and maintenance of scalable and efficient data pipelines;
• Conduct code reviews and create design documents for new features;
• Provide technical support and mentorship to other teams within the organization;
• Participate in management meetings to discuss priorities, scopes, deadlines, and cross-team dependencies;
• Develop, optimize, and tune Apache Spark jobs (Scala, PySpark, Spark SQL).
• Implement CI/CD processes using Jenkins and GitHub Actions;
• Design and implement data processing pipelines using AWS and GCP services;
• Work with Apache Airflow to manage and automate workflows, including custom plugin development;
• Utilize Databricks for job scheduling, SQL warehousing, and data visualization;
• Integrate with 3rd party services to download and upload data as required.

Software Engineering and Development Office is looking for a Middle/Senior Data Engineer in Ukraine.

About the project

Our customer is a Swiss company in commodity trading. We are helping out the customer with various applications that support their business (chain management, risk management solutions).

Requirements

• 3+ years of experience in Data Engineering
• Advanced proficiency in SQL for data manipulation and querying
• Proven experience with Power BI, including modeling, visualization, and workspace management
• Basic knowledge of Power Automate for workflow automation
• Strong analytical and problem-solving skills
• Excellent communication and collaboration abilities
• Ability to work independently and as part of a team
• At least an Upper-Intermediate level of English

Nice to have

• Experience in data analysis and business intelligence
• Familiarity with other data visualization tools and technologies
• Certification in Power BI or related fields

Responsibilities

• Power BI Modeling: Develop and maintain complex data models using Power BI to support business reporting and analytics needs
• Power BI Visualization: Create and manage interactive and visually appealing dashboards and reports to present data insights effectively
• Power BI Workspace Management: Oversee the organization and management of Power BI workspaces, ensuring efficient collaboration and data governance
• Power BI Subscriptions: Set up and manage Power BI subscriptions to automate report distribution and ensure timely delivery of insights
• SQL Advanced: Write and optimize advanced SQL queries to extract, transform, and load data from various sources into Power BI
• Power Automate Basics: Utilize Power Automate to streamline and automate workflows related to data processing and reporting