Вакансии Data Engineer / Big Data Architect

ELEKS Software Engineering and Development Office is looking for a Middle/Senior Data Engineer in Poland, Croatia and Ukraine.

About project

The customer is a British company producing electricity with zero carbon emissions.

Requirements:

• Experience in Data Engineering, SQL, ETL(data validation + data mapping + exception handling) 3+ years
• Hands-on experience with Databricks 2+ years
• Experience with Python
• Experience with AWS (e.g. S3, Redshift, Athena, Glue, Lambda, etc.)
• Knowledge of the Energy industry (e.g. energy trading, utilities, power systems etc.) would be a plus
• Experience with Geospatial data would be a plus
• At least an Upper-Intermediate level of English

Responsibilities:

• Building Databases and Pipelines: Developing databases, data lakes, and data ingestion pipelines to deliver datasets for various projects
• End-to-End Solutions: Designing, developing, and deploying comprehensive solutions for data and data science models, ensuring usability for both data scientists and non-technical users. This includes following best engineering and data science practices
• Scalable Solutions: Developing and maintaining scalable data and machine learning solutions throughout the data lifecycle, supporting the code and infrastructure for databases, data pipelines, metadata, and code management
• Stakeholder Engagement: Collaborating with stakeholders across various departments, including data platforms, architecture, development, and operational teams, as well as addressing data security, privacy, and third-party coordination

As a Senior Search Engineer, you will be a key player in our dynamic product development team to help our customers achieve this goal. This role calls for a deep commitment to crafting high-quality, user-focused solutions at terabyte scales.
You will be working primarily with Python, Go, Elasticsearch, and AWS Services: Aurora PostgreSQL, S3, and SageMaker for vectorization.

Key Responsibilities:

• Engage in back-end development to meet our product's evolving needs, ensuring robust, scalable, and high-performance solutions.
• Contribute to all stages of the product life cycle, from conception to deployment, refinement, and scaling.
• Design, implement, and maintain real-time search environments, aligning with the product’s performance and scalability requirements.
• Work collaboratively with product managers, architects, and other team members to deeply understand and address user needs and business objectives.
• Uphold high standards in software development with a focus on clean, testable, and maintainable code.
• Participate actively in code reviews, architecture discussions, and continuous refinement of our development processes.
• Foster a culture of ownership where team members feel empowered and accountable for their contributions.

Technical Skills:

• In-depth knowledge of building efficient data models for Elasticsearch or related full-text search and vector databases at scale.
• Proven experience in Golang or Python development, and a strong willingness to learn and adapt.
• Strong proficiency in SQL, including experience with complex queries, database design, optimization techniques, and working with large datasets.
• Solid foundation in software development for production systems, including OOP principles, test-driven development, CI/CD, and Agile methodologies.

Desired Qualifications:

• 2+ years of production-level Elasticsearch or related experience.
• 4+ years of software development experience for production systems.
• 4+ years of relational database (RDBMS) experience.
• Excellent English communication skills.

CommerceCore is looking for an experienced Data Engineer. This position will play a pivotal role in fostering collaboration and defining processes essential to the company's operations.

What you will do:

• Design, develop, and maintain scalable data pipelines and ETL processes to handle large volumes of structured and unstructured data;
• Collaborate with other teams to understand data requirements and ensure the availability and reliability of data;
• Optimize and tune the data infrastructure for performance, cost, and security;
• Implement data quality checks, validation, and monitoring processes to ensure the accuracy and integrity of the data;
• Develop and maintain documentation related to data architecture, processes, and workflows.
• Work with cloud platforms to manage and scale data storage and processing;
• Participate in the design and development of data warehouses and data lakes to support analytics and reporting;
• Stay current with industry trends and emerging technologies to ensure that our data infrastructure remains cutting-edge.

What we expect:

• Proven experience as a Data Engineer or in a similar role;
• Strong proficiency in SQL and experience with relational databases such as PostgreSQL, MySQL, or SQL Server;
• Proficiency in programming languages such as Python, R;
• Experience with big data technologies such as Hadoop, Spark, or Kafka;
• Familiarity with data modeling, data warehousing concepts, and ETL processes;
• Experience with cloud platforms such as AWS, Azure, or Google Cloud;
• Knowledge of data governance, security best practices, and compliance;
• Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment;
• Strong communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.

Кожен заслуговує на шанс реалізувати себе – навіть початківець із мінімальним досвідом. Ми в NIX доводимо це на практиці, тож зараз шукаємо Junior Data Engineer Scala. У нас цей спеціаліст отримає всі можливості для потужного старту!

Чим ти будеш займатися:

• Співпрацювати з командою над впровадженням нових функцій для підтримки зростаючих потреб у даних.
• Створювати та підтримувати конвеєри для вилучення, перетворення та завантаження даних з широкого спектру джерел даних.
• Відстежувати та передбачати тенденції в інженерії даних та пропонувати зміни відповідно до цілей та потреб організації.
• Ділитися знаннями з іншими командами на різні теми з інженерії даних або пов’язані з проєктами.
• Співпрацювати з командою, для вирішення, які інструменти та стратегії використовувати в рамках конкретних сценаріїв інтеграції даних.

Що тобі знадобиться:

• Мінімальний рівень англійської мови B2.
• 1+ років підтвердженого досвіду розробки програмного забезпечення з використанням Scala.
• Хороші знання SQL.
• Знання однієї або декількох з наступних баз даних – PostgreSQL, MySQL, Oracle, MSSQL.
• Знання основних концепцій та технологій обробки потокових даних (в реальному часі, близькому до реального часу).
• Розуміння принципів зберігання даних та концепцій моделювання.
• Розуміння систем контролю версій (таких як Git).
• Командний гравець з відмінними навичками співпраці.

Буде перевагою:

• Знання базових концепцій будь-якої хмарної платформи (GCP, AWS, Azure).
• Знання базових концепцій підходів, патернів та технологій розподілених обчислень.
• Знання базових концепцій нереляційних баз даних (MongoDB).
• Практичний досвід з будь-якими інструментами оркестрування (Kafka, Flink тощо).

Svitla Systems Inc. is looking for a Data Architect for a full-time position (40 hours per week) in Ukraine. Our client is a leading global provider of Environmental, Social, and Governance (ESG) performance and risk management software, data, and consulting services focusing on Environment, Health, Safety & Sustainability (EHS&S), Operational Risk Management, and Product Stewardship. That means gathering information and reporting to improve the environmental and social impacts on the customer’s business. They are a leading middle-market private equity firm focused on investments in targeted segments of the software, industrial technology, financial services, and healthcare industries. For over 30 years, they have served over 7,000 customers and millions of users in 80 countries to optimize workflows and navigate the complex and dynamic global regulatory structure.
We are preparing for a new project and need a seasoned Data Engineer/Architect.

Requirements:

• 5+ years of experience with data, data lake, and data warehouse architecture
• 5+ years with data-ingestion and compute technologies (ADF and Databricks).
• ​Experience in architecting modern data lake platforms.
• 5+ years above average understanding and practical use of programming in a modern programming language (Python, SQL).
• Understanding using version control (Azure DevOps, Git, etc.) for collaborative code development.
• Understanding of container architecture (Docker, Kubernetes).
• Knowledge of cloud architectures (Azure service).
• Knowledge of Azure Analytics Services (preferred).
• Knowledge of RDBMS systems for OLTP workloads, such as Postgres or MySQL and Synapse.
• Bachelor’s degree in Computer Science or a related field.
• Strong passion for data and architectural patterns.
• Excellent communication skills, including a knack for clear documentation and the ability to work using Agile methodologies.
• Ability to work quickly and collaboratively in a fast-paced, entrepreneurial environment.
• Self-motivated and self-managing skills.
• Strong desire to finish and win.
• Team player.
• High integrity and ethical behavior.

Responsibilities:

Your responsibilities may include, but are not limited to, the following:

• Perform as a part of the Data Services Team under the high-profile platform organization.
• Architect and be the resident expert regarding the highly scalable framework for data pipeline, transforming and enhancing data at scale.
• Help design, architect, implement, and bring data lake to reality.
• Manage design, architect, and implement data services and processes.
• Build data models and create data warehouse solutions for the client’s BI applications.

Digital transformation project for aftermarket support for Hitachi Global Air Power

Requirements:

• Bachelor’s degree in Computer Science, Engineering or related field
• 5+ years of overall work experience working on Data first systems
• 2+ years of experience on Data Lake/Data Platform projects on Azure
• Strong knowledge of SQL for handling relational databases and familiarity with NoSQL databases to manage structured and unstructured data effectively.
• Understanding of data warehousing concepts, including data storage, data marts, and ETL processes.
• Skilled in using Microsoft Azure services such as Azure Data Lake, Azure Data Factory, Azure Synapse Analytics, and Azure Databricks. These tools are essential for data ingestion, storage, processing, and analytics.
• Knowledge of cloud storage solutions provided by Azure, such as Blob Storage and Table Storage, which are integral for data lakes.
• Familiarity with ETL tools and frameworks for data extraction, transformation, and loading. Skills in Azure Data Factory or similar tools are particularly valuable.
• Ability to perform data cleaning, transformation, and enrichment to ensure data quality and usability.
• Proficient in programming languages such as Python or Scala, which are widely used in data engineering for scripting and automation.
• Skills in scripting to automate routine data operations and processes, improving efficiency and reducing manual errors.
• Understanding of how to develop and maintain APIs, particularly JDBC/ODBC APIs for data querying. Knowledge of RESTful API principles is also beneficial.
• Awareness of data security best practices, including data encryption, secure data transfer, and access control within Azure.
• Understanding of compliance requirements relevant to data security and privacy, such as GDPR.
• Experience with data testing frameworks to ensure the integrity and accuracy of data through unit tests and integration tests.
• Proficiency with version control tools like Git to manage changes in data scripts and data models.
• Familiarity with DevOps practices related to data operations, including continuous integration and continuous deployment (CI/CD).
• Basic skills in data analysis to derive insights and identify data trends, which can help in troubleshooting and improving data processes.

Responsibilities:

• Data Ingestion & Integration:
  • Assist in the development and maintenance of data ingestion pipelines that collect data from various business systems, ensuring data is integrated smoothly and efficiently into the data lake.
  • Implement transformations and cleansing processes under the guidance of senior engineers to prepare data for storage, ensuring it meets quality standards.
• Data Management:
  • Help manage the organization of data within the data lake, applying techniques for efficient data storage and retrieval.
  • Assist in managing and configuring data schemas based on requirements gathered by senior team members, ensuring consistency and accessibility of data.
  • Support the maintenance of the data catalog, ensuring metadata is accurate and up-to-date, which facilitates easy data discovery and governance.
• API Support & Data Access:
  • Support the development and maintenance of JDBC/ODBC-based SQL query APIs, ensuring they function correctly to allow end-users to access and query the data lake effectively.
  • Assist in optimizing and supporting simple analytical queries, ensuring they run efficiently and meet user and business requirements.
• Quality Assurance & Testing:
  • Conduct routine data quality checks as part of the data ingestion and transformation processes to ensure the integrity and accuracy of data in the data lake.
  • Participate in testing of the data ingestion and API interfaces, identifying bugs and issues for resolution to ensure robustness of the data platform.
• Collaboration and Teamwork:
  • Work closely with the Principal Data Architect and Principal Data Engineer, assisting in various tasks and learning advanced skills and techniques in data management and engineering.
  • Occasionally interact with other business stakeholders to understand data needs and requirements, facilitating better support and modifications in data processes.
• Platform Monitoring & Maintenance:
  • Help monitor the performance of data processes and the data lake infrastructure, assisting in troubleshooting and resolving issues that may arise.
• Learning & Development:
  • Continuously learn and upgrade skills in data engineering tools and practices, especially those related to Azure cloud services and big data technologies.
  • Contribute ideas for process improvements and innovations based on day-to-day work experiences and challenges encountered.

We are looking for a Data Engineer for our client.
Leading programmatic media company, specializing in ingesting large volumes of data, modeling insights, and offering a range of products and services across Media, Analytics, and Technology.

Hybrid work (2-3 days a week from the office in Warsaw).

Responsibilities:

• Follow and promote best practices and design principles for Big Data ETL jobs.
• Help in technological decision-making for the business’s future data management and analysis needs by conducting POCs.
• Monitor and troubleshoot performance issues on data warehouse/lakehouse systems.
• Provide day-to-day support of data warehouse management.
• Assist in improving data organization and accuracy.
• Collaborate with data analysts, scientists, and engineers to ensure best practices in terms of technology, coding, data processing, and storage technologies.
• Ensure that all deliverables adhere to our world-class standards.

Skills:

• Over 3+ years of experience in Data Warehouse development and database design;
• In-depth knowledge of distributed computing principles;
• Experience with AWS cloud services and big data platforms such as EMR, Databricks, EC2, S3, and Redshift;
• Proficiency with Scala, Spark, Hive, Yarn/Mesos, etc.;
• Strong skills in SQL and NoSQL databases, including data modeling and schema design;
• Proficient in programming languages such as Java, Scala, or Python for data processing algorithms and workflows;
• Familiarity with Presto and Kafka is advantageous;
• Experience with DevOps practices and tools for automating deployment, monitoring, and managing big data applications is a plus;
• Excellent communication, analytical, and problem-solving abilities.

N-iX is seeking a Lead Data Engineer to join our team in delivering a world-class data platform for one of the largest transportation companies in the U.S. This role is primarily focused on data modeling within a vast data environment, including a data lake and data warehouse, integrating over 30 data sources such as Salesforce, Workday, and other advanced applications.
The initial project is expected to last 2 years, after which the successful candidate will continue working within the CTO office on similar high-impact projects. You'll have the opportunity to collaborate with and learn from a world-class engineering manager and architect, who will provide mentorship and guidance.

Key Responsibilities:

• Lead the design and development of robust data models for a large-scale data platform, encompassing data lakes and data warehouses.
• Integrate and model data from 30+ diverse data sources, including Salesforce, Workday, and other enterprise applications.
• Apply Kimball principles to ensure high-quality data modeling that supports scalable and efficient data solutions.
• Collaborate closely with engineering teams, product managers, and stakeholders to define data requirements and solutions.
• Participate in the ongoing optimization and evolution of the data platform.
• Contribute to the mentorship and development of junior engineers on the team.

Must-Have Skills:

• 5+ years of experience in data modeling, particularly within large-scale data platforms.
• Expert knowledge of Kimball data modeling principles.
• Strong proficiency in English, with excellent communication skills.
• Proven experience in integrating and modeling data from multiple sources, including enterprise systems like Salesforce, Workday, JD Edwards and a bunch of home-made systems.

Nice-to-Have Skills:

• Experience with AWS data services, including Redshift, S3, and Glue.
• Experience working with Snowflake or other cloud-based data warehouses.
• Background in working with large enterprise companies, particularly in building extensive data platforms.
• Familiarity with Scaled Agile Framework (SAFe) principles.

Our client is a software company providing client management applications to organizations in the healthcare industry. For the past 15 years, DataArt's specialists have been helping them create online intake services for hospitals, medical centres, and private practices in the US.
An application is designed to help people schedule appointments, pay medical bills, find test results, and receive notifications about appointments with doctors.

Position overview

As a Software Engineer – (Data Engineer) on our team, you will contribute to the development of the fastest-growing direct-to-consumer health information platform in the US. Collaborating with a team of seasoned data engineers and architects, you will work with diverse healthcare data from both the US and international sources. This role will provide you with an in-depth understanding of various aspects of the healthcare industry, particularly healthcare data and technology. You will be responsible for building and maintaining complex data pipelines that ingest, process (using our algorithms), and output petabytes of data. Additionally, you will work closely with architects, product managers, and SMEs to develop and maintain algorithms that generate unique insights, aiding patients in finding better care more quickly.

Responsibilities

• Work closely with data engineers, the product team, and other stakeholders to gather data requirements, and design and build efficient data pipelines
• Create and maintain algorithms and data processing code in Java/Groovy
• Implement processes for data validation, cleansing, and transformation to ensure data accuracy and consistency
• Develop Python scripts to automate data extraction from both new and existing sources
• Monitor and troubleshoot the performance of data pipelines in Airflow, proactively addressing any issues or bottlenecks
• Write SQL queries to extract data from BigQuery and develop reports using Google’s Looker Studio
• Participate in daily stand-ups, sprint planning, and retrospective meetings
• Engage in peer code reviews, knowledge sharing, and assist other engineers with their work
• Introduce new technologies and best practices as needed to keep the product up to date
• Assist in troubleshooting and resolving production escalations and issues

Requirements

• Bachelor's degree or equivalent programming experience
• 4-5 years of overall experience as a backend software developer, with at least 2 years as a Data Engineer using Spark with Java/Groovy and/or Python
• Strong coding skills, and knowledge of data structures, OOP principles, databases, and API design
• Highly proficient in developing programs and data pipelines in Java/Groovy or Python
• 2+ years of professional experience with Apache Spark/Hadoop

Nice to have

• Work experience with AWS (EMR, S3, lambda, EC2, glue, RDS)
• Work experience with SQL (MYSQL is a Plus) and NoSQL Databases
• Experience with Elasticsearch
• Experience with Python
• Experience with Scala (Zeppelin)
• Experience with Airflow or other ETL
• Certification or verified training in one or more of the following technologies/products: AWS, ElasticSearch