Job Description:
We are looking for a highly skilled Data Infrastructure Engineer to join our team. The Data Infrastructure Engineer will be responsible for the management, monitoring, backup, cost management, performance tuning, and deployment of a variety of highly available and fault tolerant SQL, NoSQL, Kafka and Spark-based data layer systems running on AWS and GCP. The ideal candidate will have experience working with various data processing tools and cloud platforms.
Responsibilities:
Design, implement, and maintain data layer orchestration systems to expose and manage a series of self managed and PaaS data layer technologies.
Manage and monitor data layer components and services on AWS and GCP, ensuring optimal performance and availability.
Configure and deploy backup solutions for databases, Kafka, and Spark, ensuring data integrity and facilitating disaster recovery.
Implement cost management strategies for database and data processing infrastructure, optimising resource utilisation and minimising expenses.
Perform regular performance tuning for databases, Kafka, and Spark, identifying optimization opportunities and implementing improvements. Troubleshoot and resolve issues related to these systems.
Create and maintain documentation for Data Infrastructure engineering processes, infrastructure, and applications.
Work closely with other teams to integrate data sources and ensure data consistency, and to design and model databases that support application requirements and adhere to best practices for normalisation and performance.
Monitor and manage the capacity of data infrastructure, planning for future growth and scalability; analyse query performance and implement appropriate indexing strategies
Provide technical support and guidance to other team members in relation to Data Infrastructure engineering tasks and technologies.
Collaborate with stakeholders to define and prioritise Data Infrastructure engineering projects, setting realistic timelines and expectations.
About you
As an experienced Data Infrastructure Engineer, you will have:
Significant experience working with AWS and/or GCP core services
Significant experience building, scaling and tuning data systems such as Apache Kafka, Apache Spark, Elasticsearch and Postgres-compatible databases
An awareness of cost considerations when designing data systems architectures.
Some experience working with Kubernetes from a cluster admin perspective
Experience codifying infrastructure as code in Terraform and/or Helm charts
Experience working with observability tooling (logging, monitoring, tracing)
Experience with CI tooling, including test pipelines and asset generation and publishing (eg Docker images, helm charts)
Awareness of security considerations
Nice to have:
Experience running databases within Kubernetes
Experience of using build-test-run CI/CD techniques to IaC.
Our Tech Stack
ComplyAdvantage is fully cloud-based, with a modern kubernetes-focused tech stack. All compute workloads run in Kubernetes, with clusters in multiple regions to support the needs of our global client base. Our production services are currently mostly based in AWS, but expansion into GCP is taking place throughout 2023.
We make heavy use of Terraform and Helm to define our infrastructure and services, and lean heavily on GitOps paradigms - production and non-production environment are defined in git and changes to these environments (both cloud infrastructure and Kubernetes applications) are managed via git.
Kafka is used widely across teams, and is seen as a core technology in the future of our tech stack. Through 2023 we will be focusing on developments to our Kafka tooling, as well as the other data systems we rely on, such as Postgres, Elasticsearch and Databricks/Spark. To tie in with our multi-cloud plans, we’ll also be developing inter-cloud failover options to be leveraged by our development teams.
Across the wider Technology team, teams build and release containerised applications to support the wide array of activities that our teams are engaged in - from developing low latency client-facing APIs, to machine learning models and data processing pipelines.
About us
ComplyAdvantage is the financial industry’s leading source of AI-driven financial crime risk data and detection technology. Our mission is to neutralise the risk of money laundering, terrorist financing, corruption, and other financial crime.
More than 1000 companies rely on us to understand the risk of who they’re doing business with through the world’s only global, real-time database of people and companies. Our solutions identify thousands of risk events daily from millions of structured and unstructured data points.
We have four global hubs in New York, London, Singapore, Lisbon and Cluj-Napoca and are backed by Goldman Sachs, Ontario Teachers, Index Ventures, and Balderton Capital. Our largest engineering office is in London, but we also have development teams in Prague, Bratislava and Lisbon.
Since 2014, we have raised over $100 million in funding, and in 2022 alone grew by over 40% to over 500 people globally. Over the next 12 months, as our revenue increases, we plan to increase to 600.
What’s in it for you?
Competitive Salary
Equity as we want you to have a part of what we are building
Unlimited Time Off Policy- A work-life balance and focus on our wellbeing are critical to keeping us performing at our best.
Hybrid working- We want you to strike a balance and be where you do your best work. You will work with your manager to agree on what is best for you and your team
You will also get an annual budget to kit out your home office
Opportunity to work on innovative projects with smart like-minded people who are keen to share their knowledge and continuously improve
Annual learning budget to drive your performance and career development
At ComplyAdvantage diversity fuels our rocket ship and our commitment to inclusion across race, gender, age, religion, identity and experience drives us forward every day. We encourage everyone to apply and aspire to consider every application fairly.
We will handle your information in accordance with our Privacy Policy. For further information, please click here.
At ComplyAdvantage diversity fuels our rocket ship and our commitment to inclusion across race, gender, age, religion, identity and experience drives us forward every day. We encourage everyone to apply and aspire to consider every application fairly.
We will handle your information in accordance with our Privacy Policy. For further information, please click here.