Вакансії Data Engineer / Big Data Architect

Sigma Software is seeking an experienced Data Engineer to join our growing team.
If you thrive on solving complex problems with advanced cloud-based data collection, transformation, analysis, and monitoring tools, this opportunity is for you!
Teamwork is one of our core values, so we would like to welcome specialists who excel in collaboration, contribute to a positive and friendly atmosphere, and are passionate about their work.

Requirements

• At least 5 years of experience in data engineering and with cloud computing services solutions in the area of data and analytics, Azure is preferred one.
• Conceptual knowledge of data analytic fundamentals, e.g., dimensional modeling, ETL, reporting tools, data governance, data warehousing, and structured and unstructured data
• Knowledge of SQL and experience with at least one programming language (Python)
• Excellent communication skills and business fluency in English
• Understanding Big Data DB like Snowflake, BigQuery, etc
• Experience in database development and data modeling, ideally with Databricks/Spark
• Experience in the pre sales activities
• Excellent interpersonal skills
• Strong problem-solving and decision-making abilities, with a focus on driving results and meeting deadlines
• Strong passion for learning/practicing new techniques

Would be a plus:

• A subject matter expert in a specific domain: Logistics, Aviation, or Sustainability
• Experience with Infra as code (Terraform or similar)
• Good communication skills in German

Responsibilities

• Work closely with clients and other stakeholders to clarify the tech requirements and expectations
• Implement architectures based on Azure Cloud platforms (Data Factory, Databricks, etc.)
• Design, develop, optimize, and maintain squad-specific data architecture and pipelines that adhere to defined ETL and Data Lake principles
• Discover, understand, and organize disparate data sources and structure them into clean data models with clear and understandable schemas
• Contribute to the evaluation of new tools for analytical data engineering
• Participate in evaluating and preparing proposals for new opportunities
• Suggest and contribute to training and improvement plans regarding analytical data engineering skills, standards, and processes

Must haves

• BSc. in Computer Sciences from a top university, or equivalent;
• 5+ years in data engineering, and data pipeline development in high-volume production environments;
• 2+ years experience with Java;
• 2+ years experience with monitoring systems (Prometheus, Grafana, Zabbix, Datadog);
• Ability to develop, design, and maintain end-to-end ETL workflows, including data ingestion and transformation logic, involving different data sources;
• Experience with data-engineering cloud technologies as: Apache Airflow, K8S, Clickhouse, Snowflake, Redis and cache technologies;
• Experience with relational and non-relational DBs. Proficient in SQL and queries optimizations;
• Experience with designing infrastructure to maintain high availability SLAs;
• Experience with monitoring and managing production environments;
• Upper-intermediate English level.

Svitla Systems Inc. is looking for a Senior Data Engineer for a full-time position (40 hours per week) in Ukraine. Our client is a renewable energy company that offers solutions for organizations transitioning to sustainable energy sources. Based in Portland, Oregon, the company provides a comprehensive platform that helps businesses reduce their carbon footprints by sourcing renewable energy across their supply chains. Their services include financial, logistical, and technical support for integrating renewable energy solutions, such as Power Purchase Agreements (PPAs), energy credits, and carbon offsets. It unlocks cost savings and the ability to meet carbon commitments in time. They identify energy needs within a company’s operations using data-driven insights and remote sensing technology and create tailored solutions that optimize costs and environmental impact.

Requirements:

• A Bachelor’s or Master’s degree in Computer Science, Software Engineering, Data Engineering, or a related field.
• 7+ years of experience in data engineering, particularly in designing and implementing scalable, high-performance, cloud-based software solutions using TypeScript, Node.js, Python, and related frameworks.
• 7+ years of experience in building and scaling complex software systems in fast-paced environments, focusing on performance optimization, data throughput, and fault-tolerant architectures.
• Extensive knowledge in data architecture, including database design, data modeling, and the development of data pipelines and ETL processes.
• Expertise in designing and deploying reliable, multi-tenant cloud services focusing on data integrity, performance, and scalability.
• Expertise in designing distributed data systems that handle massive datasets, with in-depth knowledge of batch and real-time processing frameworks.
• Advanced understanding of data modeling principles, relational and non-relational databases, and data warehousing solutions.
• Strong knowledge of cloud infrastructure platforms (e.g., Google Cloud Platform, AWS) and experience with infrastructure-as-code tools to manage data-centric environments.
• Hands-on experience with performance tuning and scaling data systems to meet the demands of enterprise-grade SaaS products.
• Understanding of designing and optimizing analytics engines, particularly leveraging tools like Spark, BigQuery, debt, and Fivetran to build robust data processing pipelines.
• A proven track record of leading large-scale data engineering projects, improving data infrastructure, and driving technical strategies for multi-tenant SaaS platforms.
• Demonstrated ability to lead and mentor other engineers in best practices for data management, data engineering, and backend development.
• Exceptional problem-solving skills with a keen eye for optimizing data architectures and backend performance.
• A mindset of continuous improvement and a passion for driving data-driven decisions and innovations within your team.
• Experience leading cross-functional initiatives focusing on data governance, security, and compliance in cloud-based environments.

Responsibilities

• Play a crucial role in architecting and building a highly scalable, multi-tenant, data-driven system.
• Leverage your expertise in open-source technologies like Apache Spark, Google BigQuery, dbt, and Fivetran to handle large-scale data processing and analytics.
• Collaborate with other senior engineers to design and implement innovative solutions to complex challenges around data architecture, high-throughput data pipelines, concurrency, and system reliability.
• Excel at balancing high-level architectural decisions with hands-on implementation and leading technical initiatives that shape the future of the data platform.
• Create rock-solid reports and clear, exciting data visualizations.
• Develop a deep understanding of the data architecture, related datasets, and interactions.
• Collaborate across multi-functional teams.
• Automate frequent reporting requests to maximize efficiency.

We are looking for a savvy Data Engineer to join our growing team of analytics experts. The hire will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler, who enjoys optimizing data systems and building them from the ground up.
The Data Engineer will support our software developers, database architects, data analysts, and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products. The right candidate will be excited about the prospect of optimizing or even re-designing our company's data architecture to support our net generation of products and data initiatives.

Main tasks and responsibilities:

• Create and maintain optimal data pipeline architecture,
• Assemble large, complete data sets that meet functional / non-functional business requirements. Identified, designed, and implemented internal process improvements: automating manual processes,
• optimizing data delivery, re-designing infrastructure for greater scalability, performance, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and big data technologies. Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, revenue management, operational efficiencies, and other key business performance metrics. Work with stakeholders including the Executive, Product, Data, and Design teams to assist with
• data-related technical issues and support their data infrastructure needs.
• Keep our data separated and secure across national boundaries through multiple data centers and regions.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our offerings.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Other duties as assigned

Must have:

• 5+ years of experience in a Data Engineer role
• Must have Experience working with relational databases and data warehouses, including Postgres, Microsoft SQL Server, Oracle, GCP BigQuery, etc.
• Experience with big data tools: Hadoop, Spark, Hive, HBase, etc.
• Experience with NoSQL data repositories, ideally, Elastic Search.
• Experience building and optimizing big data pipelines, architectures, and data sets. Familiar with workflow management tools: NiFi, Airflow, etc.
• Must have experience with stream-processing systems, such as Apache Spark and/or Kafka Streaming, Storm, etc.
• Must have Experience with programming and scripting languages: Python, Scala, OOP

Наш клієнт – це компанія, яка займається аналітичними дослідженнями фінансового ринку. Наш клієнт належить до трійки найвпливовіших міжнародних рейтингових агентств, і є провайдером потужних рішень, пов’язаних з великими даними.

Навички

• Досвід роботи в якості Big Data Engineer від 3 років
• Знання та досвід роботи з:
  • Amazon Web Services
  • Elasticsearch
  • Enterprise Search Platforms
  • OpenSearch
  • Python
  • Semantic Web and Text Analytics
• Знання письмової та усної англійської мови на рівні Upper-Intermediate і вище (В2+)

Обов'язки

• Створення аналітичної платформи, що працює з декількома джерелами даних, збирає дані, обробляє їх за допомогою NLP-пайплайнів
• Ефективний пошук даних в AWS Opensearch, а також надання даних для використання внутрішнім клієнтам та потоку ML на проєкті

Буде перевагою

• Data Science
• Kubernetes
• NLP Labelling
• NLP Preprocessing
• Terraform

We are looking for a seasoned Senior Data Engineer to help us shape Emergn’s exciting future and play an important role in our growth.

We want you to:

• Work with stakeholders including data, design, product and executive teams and assist with data-related technical issues.
• Identify, design and implement process improvements including infrastructure re-design for greater scalability, data delivery optimization, and automation of manual processes.
• Build and support required infrastructure for optimal extraction, transformation and loading of data from various data sources.
• Design and assemble large, complex sets of data that meet non-functional and functional business requirements.
• Willingness to continuously learn & share learnings with others.

This job might be for you if you have:

• 3+ years of experience in Data Engineering or Data Science;
• Experience with Python;
• Experience with Snowflake or SAP Hana;
• Experienced with SQL;
• Highly experienced and skilled at collaborating with business clients;
• An analytical and creative approach to design and problem-solving;
• Excellent communication skills, including presentation skills;
• Must be fluent in English.

We are hiring for projects in the Middle East. The team comprises front-end and back-end developers, data analysts and data scientists, architects, analysts, and project managers. You will work with modern technologies and be an integral part of continuing to evolve our high-performing platforms. If you are serious about being a part of this fast-paced and exciting journey, this project should be top of the list!
This role will be primarily responsible for closely collaborating with a business SME to implement data validation rules.

Responsibilities

• Collaborating with a business SME to implement data validation rules.
• Understand business requirements and actively provide input from a data perspective.
• Understand the underlying data and flow of data.
• Understand overall platform architecture and capabilities and how to load data effectively.
• Recognize and adapt to the changes in processes.

Must have skills

• 7 years + of relevant experience.
• Proficient hands-on experience with PySpark-based data platforms:
  • Databricks, Cloudera, Apache Spark, etc. (3y+).
  • Proficiency in Python and PySpark (5y+).
  • Advanced knowledge in SQL (5y+).
• Should be able to analyse and understand complex data.
• Knowledge of Azure data/delta lake, Azure DevOps, CI/CD, and Git is essential.
• Should clearly understand the data platform lifecycle and contribute to preparing Design documents, Unit Test plans, and Code review reports.
• Knowledge of data Lake house and business intelligence architecture.
• Experience working in an Agile environment (Scrum, Agile) is a plus.
• Interpersonal/communication skills (both oral/written) with the ability to communicate at various levels with clarity & precision.
• Bachelor's degree in IT/Computer Science or related field

Nice to have

• Knowledge of the financial markets, portfolio theory, and risk management is a plus

Our client is a successful, self-funded specialty food and wellness company, rapidly growing with a team of over 550 people. Originally established in Newark, New Jersey during the Great Depression, they began by selling premium nuts at an open-air market.

Position overview

The ideal candidate will play a key role in establishing the groundwork for diverse types of analysis. This includes conducting comprehensive cross-channel analysis of media interaction logs, optimizing paid media efforts, and distilling purchase data to gain deeper insights into customer behavior. Additionally, the candidate will evaluate warehouse and sourcing data to enhance operational efficiency. You’ll also be expected to brainstorm innovative methods to improve our data systems, supporting better business decisions and enhancing the overall shopping experience.

Responsibilities

• Apply your engineering expertise to solve large-scale business problems. Drive design and architecture discussions and build alignment on complex technical decisions
• Design, implement, validate, and deploy software solutions to enhance our reporting and analytics systems. This includes new features and architectural improvements for our data pipeline and analytics tools, such as Looker, dbt, Databricks, Fivetran, Airbyte, and AWS Redshift
• Provide technical leadership in data architecture and help establish best practices for data modeling, storage, and retrieval
• Collaborate with cross-functional teams to understand data needs and design solutions that meet business requirements
• Drive the adoption of advanced analytics techniques and technologies to unlock insights from large datasets
• Perform design and code reviews to ensure quality and scalability
• Lead the design of data governance policies and procedures. Help establish and enforce data security and privacy protocols to safeguard sensitive information
• Further agile and DevOps practices to ensure efficient and high-quality data engineering processes
• Support on-call assignments and operational management
• Collaborate with business analysts and product owners to assist with measuring experiments to vet hypotheses
• Mentor, lead, and train other members of the data engineering team

Requirements

• Bachelor's degree required in Computer Science, Software Engineering, or related field
• 3+ years of experience in data engineering working with big data systems that can process and transform data at scale
• Deep knowledge of analytics technologies, modeling techniques, data streaming and other ingestion approaches, data warehousing concepts, cloud platforms (AWS preferred), and proven experience with data architecture including data pipelines, dimensional modeling, and dependency management
• Experience developing with data SaaS and BI tooling (Fivetran, Looker, dbt, Databricks, Apache Airflow, etc.)
• Experience automating data integration using APIs
• Proficiency in programming languages such as SQL and Python in an analytics context
• Experience handling operational aspects of large-scale systems focusing on observability, tech debt management, code refactoring, and robust software development and release processes
• Experience articulating technically complex topics in an easy-to-understand way via written and verbal communication methods
• Proven track record of mentorship, leadership, and ownership within teams and organizations

Nice to have

• Master's degree preferred in a related field

Ciklum is looking for a Senior Data Engineer to join our team full-time in Ukraine.

About the role:

As a Senior Data Engineer, become a part of a cross-functional development team engineering experiences of tomorrow.

Responsibilities

• Designing, developing, and deploying Power BI solutions, providing advanced data analytics, and creating dashboards and reports that deliver key insights to the business
• Data models design and implementation
• Optimize reports and dashboards for better performance by improving query speed and ensuring efficient data model design
• Contributes design, code, configurations, manage data ingestion, closed to real-time processing, batch processing, ETL across multiple data storages
• Responsible for performance tuning of complicated SQL and DAX queries
• Identify gaps and improve the platform’s quality, robustness, maintainability, and speed
• Cross-train other team members on technologies being developed, while also continuously learning new technologies from other team members
• Contribute in the Unit’s activities and community building, participate in conferences, and provide excellence in exercise and best practices

Requirements

We know that sometimes, you can’t tick every box. We would still love to hear from you if you think you’re a good fit!

• 3+ years of experience in developing BI solutions with Power BI
• Proficient in Power BI Desktop and Power BI Service, including report and dashboard creation, data transformations, and DAX formulas
• Strong expertise in DAX (Data Analysis Expressions) for complex calculations
• Experience with Power Query for data transformations and ETL processes
• Solid understanding of data modeling principles and best practices
• 3+ years of experience coding in SQL(Azure SQL DB or MSSQL preferable), solid CS fundamentals including data structure and algorithm design
• 2+ years contributing to production deployments of large backend data processing and analysis systems
• Experience in data integration and ETL processes
• Proficient with Azure Data Services (Azure Data Factory, Azure Synapse Analytics, etc.)
• Strong problem-solving and analytical skills, with a keen attention to detail
• Excellent communication skills, both written and verbal, with the ability to present complex data in a clear and concise manner
• Experience in working with cross-functional teams and collaborating with stakeholders at various levels of the organization
• Knowledge of professional software engineering best practices for the full software
• Knowledge of Data Warehousing, design, implementation and optimization
• Knowledge of Data Quality testing, automation and results visualization
• Knowledge of the development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations
• Experience participating in an Agile software development team, e.g. SCRUM
• Experience designing, documenting, and defending designs for key components in large distributed computing systems
• A consistent track record of delivering exceptionally high-quality software on large, complex, cross-functional projects
• Demonstrated ability to learn new technologies quickly and independently
• Ability to handle multiple competing priorities in a fast-paced environment

Desirable

• Experience with Power BI Embedded
• Experience with Power BI REST or JS APIs
• Power BI Premium Capacity Management
• Experience in backend development and deployment
• Experience in CI/CD configuration
• Experience with Azure DevOps for CICD
• Experience with Tabular Editor, DAX Studio, SQL Profiler
• Experience with Powershell
• Experience with Swagger, Postman