Share this Job

Data Engineer

Date: 19-Aug-2019

Location: Macquarie Park, Australia

Company: Singtel

Data Engineer


Ever wanted to build real time data pipelines on hundreds of billions of rows of data?


At Optus Sport, we collect hundreds of millions of data points per hour during matches across dozens of tech layers and systems to ensure a consistent user experience for our customers. Your main task as the Data Engineer is to maintain and improve our data pipeline (built in AWS) to distil multi-terabytes worth of raw data into accurate and actionable insights to the business in real time.


The role

  • Design and implement robust and stable data pipeline within AWS tech stack (Kinesis Firehose)
  • Ensure data integrity across multiple sources from measurement implementation to ETL
  • Continual analysis of server and client-side log and behavioural data to improve video delivery experience
  • Work with engineering, operations and development teams to determine how to safely, easily and quickly collect additional data
  • Continued innovation in data pipeline to improve quality and depth of insights and reduce time to delivery
  • Automated models for categorizing user experience and predict adverse client-side behaviour
  • Experiment with new technologies and build proof of concepts


The perks


We offer all kinds of benefits, such as:


  • Work collaboratively in an open and agile
  • Discounts with over 400 companies Australia wide (Technology, Retail, Home, Fashion and more!)
  • Mobile and Broadband staff discounts
  • Onsite facilities at Macquarie Park such as a Gym, GP, Mini-Mart, Cafes
  • Training, Mentoring and further learning opportunities
  • Staff busses to Epping and Wynyard, and back again





About you


Preferred skills and experience include:


  • At least 4 years data industry experience
  • At least 2 years experience with Python
  • Expert experience in SQL / NoSQL/ Hadoop (Hive) databases
  • Experience with Cloud Data Warehouse Architecture: Athena, Glue, Lambda, EMR, Kinesis, Step Functions
  • Experience designing and implementing automated ETLs (preferably in AWS)
  • Data Analytics: general statistical analysis skills
  • OS: Mac, Linux (Ubuntu)


About us


At Optus, we don’t sit back and let the future happen to us – we’re out there making it.  By expanding into new technology and relentlessly improving every day, we’re working to create a better tomorrow for all Australians.


Find similar jobs: