Data Engineer - Elastic

Nov 08, 2023

KIN51515 Mumbai (KIN51515) We Work

Who We Are

Kyndryl is a market leader that thinks and acts like a start-up. We design, build, manage, and modernize the mission-critical technology systems that the world depends on every day. So why work at Kyndryl? We are always moving forward – always pushing ourselves to go further in our efforts to build a more equitable, inclusive world for our employees, our customers, and our communities.

The Role

  • Job Description:
    An ELK(Elastic, Logstash & Kibana) Data Engineer is responsible for developing, implementing, and maintaining the ELK stack-based solutions within an organization. The engineer plays a crucial role in developing efficient and effective log processing, indexing, and visualization for monitoring, troubleshooting, and analysis purposes.
    Key Responsibilities:
    1. Installation and Configuration:
       - Deploy Elasticsearch, Logstash, and Kibana on virtual machines, containerized environments (e.g., Docker).
       - Set up and configure Elasticsearch clusters, ensuring high availability, fault tolerance, and scalability.
       - Configure Logstash to receive, filter, and transform logs from diverse sources (e.g., servers, applications, AppDynamics, Storage, Databases and so son) before sending them to Elasticsearch.
       - Configure Kibana for data visualization, including creating visualizations, dashboards, and alerts.
       - Configure ILM policies, Index templates etc.
    2. Data Ingestion and Processing:
       - Develop Logstash configuration files to parse, enrich, and filter log data from various input sources (e.g., AppDynamics, Database, Storage and so on).
       - Implement techniques like grok patterns, regular expressions, and plugins to handle complex log formats and structures.
       - Ensure efficient and reliable data ingestion by optimizing Logstash performance, handling high data volumes, and managing throughput.
    3. Data Transformation and Enrichment:
    -Define and implement data transformation and enrichment processes to cleanse, enrich, and prepare data for analysis and visualization.
    -Utilize Logstash pipelines for data manipulation and transformation.
    4. Data visualization and reporting:
    - Utilize Kibana to create visually appealing dashboards, reports, and custom visualizations.
    - Collaborate with business users to understand their data visualization needs and translate them into actionable insights.
    - Establishing the correlation within the data and develop visualizations to detect the root cause of the   issue.
    - Integration with ticketing tools such as Service Now.
    - Hands on with ML and Watcher functionalities.

    4. Cluster Monitoring:
       - Monitor Elasticsearch clusters for health, performance, and resource utilization.
     5. Security and Access Control:
       - Implement security measures for the ELK stack, including secure transport, authentication, and authorization.
       - Configure role-based access control (RBAC) to manage user access privileges within Kibana and Elasticsearch.
    6. Documentation and Collaboration:
       - Create and maintain technical documentation, including system diagrams, deployment procedures, and troubleshooting guides.
       - Collaborate with cross-functional teams, such as developers, system administrators, and security teams, to integrate ELK stack functionalities into existing systems and workflows.
       - Stay up-to-date with industry trends and best practices related to log management, Elasticsearch, Logstash, and Kibana.

Who You Are

  • Candidate should have minimum 6+ years of experience.
  • Implement Data-related Development Methods.
  • Design Databases.
  • Apply Data Modelling Techniques to IT Solutions.
  • Apply Knowledge of Data Mining Techniques & Tools.
  • Applications, Data and AI.

Being You

Diversity is a whole lot more than what we look like or where we come from, it’s how we think and who we are. We welcome people of all cultures, backgrounds, and experiences. But we’re not doing it single-handily: Our Kyndryl Inclusion Networks are only one of many ways we create a workplace where all Kyndryls can find and provide support and advice. This dedication to welcoming everyone into our company means that Kyndryl gives you – and everyone next to you – the ability to bring your whole self to work, individually and collectively, and support the activation of our equitable culture. That’s the Kyndryl Way.

What You Can Expect

With state-of-the-art resources and Fortune 100 clients, every day is an opportunity to innovate, build new capabilities, new relationships, new processes, and new value. Our employee learning hub gives you access to the best learning in the industry to receive certifications and accreditations, including Microsoft University, AWS Cloud Center of Excellence, Udemy, and the Harvard Business Review. Through our company-wide volunteering and giving platform, you can donate, start fundraisers, volunteer, and search over 2 million non-profit organizations.  At Kyndryl, we invest heavily in you! We want you to succeed so that together, we will all succeed.

Join 27662+ Machine Learning Engineers, receiving daily job alerts.