Вакансии Data Engineer / Big Data Architect

N-iX is seeking a proactive Senior Data Engineer to join our vibrant team. As a Senior Data Engineer, you will play a critical role in designing, developing, and maintaining sophisticated data pipelines, Ontology Objects, and Foundry Functions within Palantir Foundry. Your background in machine learning and data science will be valuable in optimizing data workflows, enabling efficient model deployment, and supporting AI-driven initiatives. The ideal candidate will possess a robust background in cloud technologies, data architecture, and a passion for solving complex data challenges.

Key Responsibilities:

• Collaborate with cross-functional teams to understand data requirements, and design, implement, and maintain scalable data pipelines in Palantir Foundry, ensuring end-to-end data integrity and optimizing workflows.
• Gather and translate data requirements into robust and efficient solutions, leveraging your expertise in cloud-based data engineering. Create data models, schemas, and flow diagrams to guide development.
• Develop, implement, optimize and maintain efficient and reliable data pipelines and ETL/ELT processes to collect, process, and integrate data to ensure timely and accurate data delivery to various business applications, while implementing data governance and security best practices to safeguard sensitive information.
• Monitor data pipeline performance, identify bottlenecks, and implement improvements to optimize data processing speed and reduce latency.
• Collaborate with Data Scientists to facilitate model deployment and integration into production environments.
• Support the implementation of basic ML Ops practices, such as model versioning and monitoring.
• Assist in optimizing data pipelines to improve machine learning workflows.
• Troubleshoot and resolve issues related to data pipelines, ensuring continuous data availability and reliability to support data-driven decision-making processes.
• Stay current with emerging technologies and industry trends, incorporating innovative solutions into data engineering practices, and effectively document and communicate technical solutions and processes.

Tools and skills you will use in this role:

• Palantir Foundry
• Python
• PySpark
• SQL
• TypeScript

Required:

• 5+ years of experience in data engineering, preferably within the pharmaceutical or life sciences industry;
• Strong proficiency in Python;
• Hands-on experience with cloud services (e.g., AWS Glue, Azure Data Factory, Google Cloud Dataflow);
• Expertise in data modeling, data warehousing, and ETL/ELT concepts;
• Hands-on experience with database systems (e.g., PostgreSQL, MySQL, NoSQL, etc.);
• Hands-on experience in containerization technologies (e.g., Docker, Kubernetes);
• Familiarity with ML Ops concepts, including model deployment and monitoring.
• Basic understanding of machine learning frameworks such as TensorFlow or PyTorch.
• Exposure to cloud-based AI/ML services (e.g., AWS SageMaker, Azure ML, Google Vertex AI).
• Experience working with feature engineering and data preparation for machine learning models.
• Effective problem-solving and analytical skills, coupled with excellent communication and collaboration abilities;
• Strong communication and teamwork abilities;
• Understanding of data security and privacy best practices;
• Strong mathematical, statistical, and algorithmic skills.

Nice to have:

• Certification in Cloud platforms, or related areas;
• Experience with search engine Apache Lucene, Web Service Rest API;
• Familiarity with Veeva CRM, Reltio, SAP, and/or Palantir Foundry;
• Knowledge of pharmaceutical industry regulations, such as data privacy laws, is advantageous;
• Previous experience working with JavaScript and TypeScript.

We're looking for an experienced Backend Developer to join our growing data platform team.
As a Backend Developer, you'll work on a massive data processing pipeline, ingesting over a billion daily events from multiple sources. You'll also create the next-generation pipeline and help us scale from a billion events a day to tens of billions of events.

Responsibilities

• Own projects from initial discussions to release, including data exploration, architecture design, benchmarking new technologies, and product feedback.
• Work with massive amounts of data from different sources using state-of-the-art technology to make big data accessible in real-time.
• Develop and deploy real-time and batch data processing infrastructures.
• Manage the development of distributed data pipelines and complex software designs to support high data rates (millions of daily active users) using cloud-based tools.
• Work closely with company stakeholders on data-related issues.
• Develop unit, integration, end-to-end (e2e), and load tests.

Requirements

• 4+ years of experience as a Software Engineer, including design & development.
• Proven experience with Java or Go.
• Experience in the design and development of scalable big data solutions.
• Experience working in a cloud-based environment.
• Passionate about technologies, frameworks, and best practices.
• Ability to work in a fast-paced environment.

Advantages

• Experience with Spring / Kubernetes.
• Experience with Terraform / Helm / Argo.

We are seeking a Senior Data Engineer to join our growing team and help design, build, and maintain robust data infrastructure for our client — a leader in digital banking solutions. This role is ideal for someone who is passionate about building scalable and efficient data pipelines and enjoys working with modern data tools in a cloud environment.

This is the job

As a Senior Data Engineer, you will play a key role in the development of a modern data platform. You will collaborate closely with data scientists, analysts, and other engineers to ensure high data availability, quality, and performance.

This is you

• You have proven experience with Airflow, Python programming, and SQL / dbt.
• You’ve worked with Snowflake or similar cloud-based data warehouses.
• You have a solid understanding of AWS services and Docker.
• You hold a Bachelor’s degree in Mathematics, Computer Science, or another relevant quantitative field.
• You approach problems with strong analytical and problem-solving skills.
• You’re familiar with Data Engineering best practices, including Data Quality and Monitoring/Observability.
• You’re comfortable working in a dynamic, fast-paced environment, and take ownership of your work.
• You have a growth mindset and are eager to learn and grow through hands-on experience.

This is your role

• Design and develop scalable data pipelines to efficiently process and analyze large volumes of data, utilizing Snowflake, Looker, Airflow, and dbt.
• Collaborate with stakeholders to translate their requirements into technical steps and coordinate the projects you drive with them.
• Monitor and improve the health of our data pipelines.
• Promote knowledge sharing within the team to foster collaboration and continuous learning, and mentor junior colleagues.
• Stay updated on emerging technologies and best practices in data engineering, and bring new ideas to enhance the technical setup.

We’re opening the position of a Senior Data Engineer to work with our Client – a leading energy sector analytics company, as they expand their innovative data products that revolutionize document analysis. As a Data Engineer, you will focus on executing defined tasks to support our data product coverage. You’ll work closely with a talented team of Engineers to ensure high-quality data solutions. Here, you’ll find a culture that values collaboration and encourages learning and growth.

Responsibilities:

• Develop and maintain scalable data pipelines using Python, SQL, AWS services (Amazon Bedrock, S3), and Databricks
• Build and optimize ETL jobs in Databricks using PySpark, ensuring efficient processing of large-scale distributed datasets
• Play a pivotal role in enhancing the breadth and depth of our courthouse data products
• Utilize your Python expertise to parse complex datasets, manipulate intricate image data, and craft innovative data products that meet our customers’ evolving needs
• Champion data quality, consistency, and reliability throughout our product lifecycle
• Contribute to the development of new features and the continuous improvement of existing data systems
• Design and implement distributed Data Engineering solutions in Databricks, leveraging PySpark for optimized workflows

Requirements:

• At least 5 years of experience as a Data Engineer, with a proven track record of successful projects
• Advanced proficiency in Python and PySpark – the languages of data manipulation and analysis
• Expertise in Databricks as a distributed Data Engineering platform
• Solid experience with relational database systems, particularly MS SQL Server
• Ability to collaborate effectively within a team environment and meet project deadlines
• Strong communication skills and fluency in English
• English level – Upper-Intermediate or higher

Will be a plus:

• Knowledge of cloud platforms such as AWS, Azure, or Google Cloud
• Experience with containerization technologies like Docker and Kubernetes

Our Client is the Enterprise Worldwide Company. The product you will be working with, provides management and data processing/handling capabilities for networks of the clients scientific lab equipment such as microscopes, etc. The main goals are:

• Collection and centralized management of data outputs (measurement results, etc.) provided by clients devices
• Outdated data utilization
• Managing large volumes of data acquired from measurement devices in the cloud securely and reliably
• Seamless sharing of measurement data with collaborators
• The ability to share measurement results and accelerate customer service.

Requirements

We are looking for a Lead Data Engineer with at least 6 years of commercial experience in development of data platforms for enterprise applications. With the experience to Lead a team of engineers and take responsiility for the technical solution.

• Proficiency in Airflow for workflow orchestration, dbt for data transformation, and SQL for data querying and manipulation.
• Experience in data modeling, ETL (Extract, Transform, Load) processes, and data warehousing concepts.
• Familiarity with cloud platforms (AWS) and their data services.
• Excellent analytical and problem-solving skills with meticulous attention to detail.
• Strong communication and collaboration skills with the ability to lead and motivate cross-functional teams.Good to have ability to participate onsite meeting.

Job responsibilities

• Implement new solutions into the current system with the refactoring and from scratch methods;
• Preparing the technical documentation;
• Participating in client meetings to understand business and user requirements and estimate tasks;
• Collaborating closely with other engineers, product owners and testers to identify and solve challenging problems;
• Taking part in defect investigation, bug fixing, troubleshooting;

We’re looking for a Data Engineer to join our team in the iGaming industry, where real-time insights, affiliate performance, and marketing analytics are at the center of decision-making. In this role, you’ll own and scale our data infrastructure, working across affiliate integrations, product analytics, and experimentation workflows.
Your primary responsibilities will include building and maintaining data pipelines, implementing automated data validation, integrating external data sources via APIs, and creating dashboards to monitor data quality, consistency, and reliability. You’ll collaborate daily with the Affiliate Management team, Product Analysts, and Data Scientists to ensure the data powering our reports and models is clean, consistent, and trustworthy.

What you’ll do:

• Design, develop, and maintain ETL/ELT pipelines to transform raw, multi-source data into clean, analytics-ready tables in Google BigQuery, using tools such as dbt for modular SQL transformations, testing, and documentation.
• Emphasize data quality, consistency, and reliability by implementing robust validation checks, including schema drift detection, null/missing value tracking, and duplicate detection using tools like Great Expectations or Soda.
• Integrate and automate affiliate data workflows, replacing manual processes in collaboration with the related stakeholders.
• Proactively monitor and manage data pipelines using tools such as Airflow, Prefect, or Dagster, with proper alerting and retry mechanisms in place.
• Build a Data Consistency Dashboard (in Looker Studio, Power BI, Tableau or Grafana) to track schema mismatches, partner anomalies, and source freshness, with built-in alerts and escalation logic.
• Ensure timely availability and freshness of all critical datasets, resolving latency and reliability issues quickly and sustainably.
• Control access to cloud resources, implement data governance policies, and ensure secure, structured access across internal teams.
• Monitor and optimize data infrastructure costs, particularly related to BigQuery usage, storage, and API-based ingestion.
• Document all pipelines, dataset structures, transformation logic, and data contracts clearly to support internal alignment and knowledge sharing.
• Build and maintain postback-based ingestion pipelines to support event-level tracking and attribution across the affiliate ecosystem.
• Collaborate closely with Data Scientists and Product Analysts to deliver high-quality, structured datasets for modeling, experimentation, and KPI reporting.
• Act as a go-to resource across the organization for troubleshooting data discrepancies, supporting analytics workflows, and enabling self-service data access.

What we expect from you:

• Strong proficiency in SQL and Python.
• Proven ability to design, deploy, and scale ETL/ELT pipelines.
• Experience with Google BigQuery and other GCP tools (e.g., Cloud Storage, Cloud Functions, Composer).
• Hands-on experience integrating and automating data from various platforms.
• Familiarity with postback tracking, attribution logic, and affiliate data reconciliation.
• Skilled in orchestration tools like Airflow, Prefect, or Dagster.
• Experience with Looker Studio, Power BI, Tableau, or Grafana for building dashboards for data quality monitoring.
• Use of Git for version control and experience managing CI/CD pipelines (e.g., GitHub Actions).
• Experience with Docker to build isolated and reproducible environments for data workflows.
• Strong sense of data ownership, documentation, and operational excellence.
• Exposure to iGaming data structures and KPIs is a strong advantage.

Our project is to develop the data platform, where all the analytical data of the company will be stacked. This is a great opportunity to participate in the launch and operation of a large Kubernetes/Spark/S3 cluster and one of the most interesting BI practices in Eastern Europe.

Responsibilities:

• Support and active development of the Data Platform and Hybrid Data Platform (on-prem + Azure Fabric, in progress)
• Support for a team of data engineers and analysts

Skills:

• Understanding the advantage of GitOps/IaC over manual work
• Kubernetes, Helm, ArgoCD, Prometheus, Grafana, Loki, HashiCorp Vault
• Apache Spark in Kubernetes, Apache Kafka, Minio/S3, Apache Airflow
• Docker (BuildKit), Gitlab, Gitlab CI
• Experience with at least one of the most popular programming languages, such as Python, Golang, Java, Scala, etc.Be able writing a code

Will be a plus:

• Kerberos, Active directory
• Clickhouse
• Datahub
• Elasticsearch
• Experience with any OLAP DB, support, optimize
• Security in K8s, HashiCorp Vault, Oauth, OpenID, Keycloak

Will be a significant plus:

• Azure, Azure Fabric.

Technologies that we use:

• Kubernetes RKE2 1.31. Cilium 1.17. Gitops, Argocd, Helm, Kustomize, Kyverno.
• Gitlab, Gitlab CI, Gitlab Kubernetes Runner, Docker, Buildkit.
• Apache Airflow.
• Apache Spark, Apache Kyuubi, Hive Metastore
• Minio, Redis, PostgreSQL (Cloudnative-PG), Elasticsearch, Apache Kafka, Clickhouse.
• Datahub.
• Prometheus Stack, Grafana, Grafana Loki.
• Python, Golang metrics exporters, Datadog Vector, Fluent-bit.
• PowerBI, Azure Fabric.
• Ansible.

We are seeking an experienced developer to create a high-performance, scalable, and flexible behavioral analytics engine platform.
You will be a key member of our team, responsible for the architecture, development, and optimization of core components for processing and analyzing large volumes of data [terrabytes].

Required professional experience:

• 7+ years of experience in developing analytics platforms or big data processing systems.
• Deep knowledge of programming languages such as Java, Scala, Python, Go, C++, or Rust.
• Experience with distributed systems and big data technologies [Kafka, Flink, Spark, Apache BEAM].
• Understanding of scalable system design principles and architectures for real-time data processing.
• Experience with NoSQL databases [Cassandra, MongoDB].
• Experience with OLAP databases [ClickHouse, StarRocks, Doris].
• Knowledge of SQL.
• Understanding of statistical methods and principles of data analysis.
• Experience with Kubernetes [Helm, ArgoCD].

Desired Skills:

• Experience with open table format [Apache Iceberg/Delta Lake/Hudi].
• Experience with cloud platforms [Google Cloud, AWS, Azure].
• Knowledge of data security methods and compliance with regulatory requirements [GDPR, CCPA].

Key Responsibilities:

• Design and develop the architecture of an behavioral analytics platform for real-time big data processing.
• Implement key engine systems [data collection, event processing, aggregation, prepare data for visualization].
• Optimize the platform performance and scalability for handling large data volumes.
• Develop tools for user behavior analysis and product metrics.
• Collaborate with data analysts and product managers to integrate the engine into analytics projects.
• Research and implement new technologies and methods in data analysis.

Boosty Labs is one of the most prominent outsourcing companies in the blockchain domain. Among our clients are such well-known companies as Ledger, Consensys, Storj, Animoca brands, Walletconnect, Coinspaid, Paraswap, and others.
About project: Advanced blockchain analytics and on-the-ground intelligence to empower financial institutions, governments & regulators in the fight against cryptocurrency crime

Requirements:

• 3+ years of experience in data engineering or a similar role
• Strong programming skills in Python
• Solid hands-on experience with Apache Kafka for real-time data streaming
• Experience working with ClickHouse or other columnar databases
• Understanding of Data Lake architecture and cloud data storage solutions
• Familiarity with Argo Workflows or similar workflow orchestration tools (e.g., Airflow, Prefect)
• Experience with CI/CD processes and containerization (Docker, Kubernetes) is a plus
• Strong problem-solving skills and the ability to work independently

Responsibilities:

• Design and implement scalable, efficient, and reliable data pipelines
• Work with real-time and batch data processing using Kafka and ClickHouse
• Develop and maintain ETL/ELT processes using Python
• Manage and optimize data storage in cloud-based Data Lake environments
• Use Argo Workflows to orchestrate complex data workflows
• Collaborate with data scientists, analysts, and engineering teams to support their data needs
• Ensure data quality, consistency, and governance throughout the pipeline