Senior Data Engineer

June 16, 2022


About Blackpanda: Blackpanda is Asia’s premier incident response firm, helping businesses of all sizes in the APAC region protect against cyber risks, and be there to help if a breach occurs. About the role: This position reports to CTO Gregor VandTechnology is at the heart of what Blackpanda does, and is looking for a seasoned data engineer who thrives on daily hands-on projects, while excited by the opportunity to grow into leading a tight-knit, diverse data engineering team building class-leading architecture and products. Blackpanda is at an exciting stage, envisioning what cyber risk protection can look like in the APAC region, and you will be building that future. This role suits someone who is excited by the prospect of building data architecture and products from the ground up with the most appropriate technologies available today. A background in cyber security is a plus, but not required. However, an excellent grasp of ETL pipelines and Lakehouse architecture is expected. Your ultimate role for Blackpanda initially will be to create scalable pipelines that ingest, curate, and aggregate complex data in a timely and secure way. If this sounds like you, keep reading! You will be led by our CTO and expected to be able to synthesize our data and business problems into an engineered pipeline solution using Lakehouse architecture. Blackpanda is building a cross-functional engineering team where all members should be excited by the prospect of solving problems first through planning and dialogue, and only after this moving to code and infrastructure set up. What you will be working on: You will be the leading engineer for our data pipeline aspirations across the next 12 months, spanning three different technology verticals. Blackpanda is building exciting technology both for internal use and for external-facing customer engagement.Communicating with business stakeholders from time to time to help advise on strategy, according to the likely difficulty of solving the technology challenge.Engage with code reviews of your peers, especially more junior developers.Have the opportunity to lead engineering initiatives/discovery to ascertain the best path forwards before diving into writing production code.Shipping robust, maintainable, tested code and processes - often! Your work will be the backbone of several internal processes and lead to external-facing applications in the future. Qualities you likely have to be well suited to this role: At least 5 years working as part of a data/software development team, shipping production-ready pipelines, and handling multiple streams of data.Natural curiosity and you thrive on learning new thingsValue balance in your life and know when to call it a day so that you come back fresh the next morning with class-leading solutions. We do not promote sloppy code and as such, we guard against overworked, tired developers, but you must also be good at managing your own time to achieve this.Passion for efficiency and collaboration, with a history of establishing great relationships with your engineering peers.Prioritize test-driven, elegant solutions that are maintainable, aided by appropriate terse or detailed documentation, whichever makes the most sense per case.Work closely with the CTO to evaluate platforms - test driving an API or working methodically through the documentation to understand if it will solve the business need.You understand that there are significant costs associated with computing and storage on an ongoing basis of data engineering, and understand how to help keep these manageable with the help of your team.Comfortable presenting to a group over video calls periodically, describing how a solution was arrived at, and how it works in layman's terms. Technical experience/knowledge At least 5 years working in a professional capacity as a data engineer / senior developer with a data pipeline focus. This time does not include any years of study or pure ‘hobby project’ time.You have to lead the development of, or significantly contributed to products that are used daily either by customers or internal teams and form a critical function of the organization / primary revenue stream.Extremely proficient in PythonHands-on experience with SparkFamiliar with at least one containerized deployment platform, such as Google Cloud Run, Amazon ECS, Azure and/or directly with K8sExperience designing scalable ETL pipelines around the Lakehouse architecture and ideally with Delta LakeExperience using and maintaining CI/CD pipeline infrastructure. Minimal maintenance such as Github actions experience is a plus.Experience with data curations, transformation, and availability, ideally via Delta Lake architecture or a similar ACID layer.Hands-on experience with additional data pipeline technologies such as Apache stack, Kafka, etc.Bachelor’s degree in a related field or a clear history of engineering in a team-based, professional capacity for the stated experience years (5+) General requirements Business fluent written and verbal communication skills in EnglishYou can be based anywhere in the world but must be able to work Monday to Friday for at least six hours between 0900 to 1800 SGT/HKT and be able to legally work in your chosen location (Blackpanda is not able to sponsor a work permit for a chosen country)Conversational fluency and higher in any Asian language is a plus but not required Black panda offers to all team members Top of market base pay system for position and locality every yearEquity awards available based on performance40 days combined annual leave (this includes any public holidays in your location of work)No-meeting FridaysYou can enjoy up to US$5000 learning and development allowance each calendar year. Team meetups in your region, and with the wider global Blackpanda tribe (COVID travel restrictions permitting)Black panda is committed to building a culturally diverse company, and we value a broad set of opinions in our team. As we grow, we are looking to build a team with a range of viewpoints at its core, and we encourage applications from all genders as you identify (X/F/M) and minority candidates.

Built with ❤️ for the ML Community by Dom © 2022 RemoteML