Data Engineer

Responsibilities:

• Develop new and improve existing data pipeline on Big Data platform (Hadoop or equivalent)
• Build new and enhance existing data application for streaming and batch datasets
• Work on Apache Airflow data pipeline, Python, Spark, Scala, Java, etc open sources technologies
• Develop and maintain Data REST API (Spring Boot or equivalent) and Streaming API (Kafka or equivalent)
• Perform R&D and conduct PoC (proof-of-concept) for new data solution
• Perform metadata and data management, data security
• Drive optimization, testing and tooling to improve data quality
• Design high level & detailed design to ensure that the solution delivers to the business needs and align to the data & analytics architecture principles and roadmap
• Collaborate with different stakeholders from business, technical, project management and operation to design and implement the solution.
• Ensure best practices, frameworks and re-useable components are employed in the development project
• Ability to lead troubleshooting efforts for complex design and eliminate application issue faced by the project and operation team.
• Understand various data security standards, information security standard and to apply and adhere to the required data controls for user access

We are committed to a safe and healthy environment for our employees & customers and will require all prospective employees to be fully vaccinated.

The Ideal candidate should possess:

• Bachelor degree in Computer Science, Math, Data Analytics or related field with at least 2 years of hands-on development experience
• Knowledge in Big Data Platform such as Hadoop, Cloudera, Hortonworks, etc
• Hands-on experience in Programming language including Python, PysSpark, SQL, prefer to have experience in Big Data application and Linux shell scripting
• Experience in Data API exposure (Spring Boot, Flask or equivalent)
• Experience in Open Source like Apache Airflow, Zeppelin Notebook for Data Exploratory Analysis, etc
• Basic knowledge in Microsoft software e.g. Excel, Word and Power Points.
• Comfortable to work with technical and non-technical staff to explain concept and algorithms