Big Data Engineer – London

  • Freelance
  • London
  • Posted 11 months ago
  • Salary: £650 per day

SearchBI SearchBI

Professional Services firm

Big Data Engineer – London – 6 month contract – Circa £650 per day


The Project

The project seeks to replace the existing BI data warehouse system with a cloud-based data management platform to ingest, secure and provision key data sets, providing the capability to expose data in a consumable way from which users can use tools to self-serve their own BI, MI, and advanced analytics reports and dashboards.


This project requires security clearance. If you do not have security clearance you must be eligible to be security cleared which will be funded by the client.


Key Purpose of the Role

The Security cleared Big Data Engineer (BDE) will typically come from a strong IT development and cloud background and have a proven track record in responsibility for designing, building, implementing, automating and supporting Hadoop infrastructure in an AWS environment.

  • Building end to end Hadoop cluster solutions in a secure enterprise environment (Authentication / Authorisation + Encryption), AWS IAM / Sentry / AD / KMS
  • Hadoop Administration experience (Preferably selected distribution Cloudera, Hortonworks, MapR)
  • Integration with the existing Cloud Authentication technologies (IPA) and providing first grained authorisation to data and functions.
  • Big Data tools and services (HDFS, Sqoop, Oozie, YARN, HIVE, Impala, Spark, Sentry, Pig, Kerberos, Active Directory).
  • Linux Administration background and knowledge of Linux technologies like Chef, Puppet etc.
  • Implementing visualisation tooling and provisioning data to that tooling SpotFire, Tableau.


Essential experience/knowledge:


  • AWS (Networking, CloudFormation, IAM)
  • Cloudrea Hadoop Distribution (HDFS, Sqoop, Oozie, YARN, HIVE, Impala, Spark, Sentry, Kerberos, Active Directory Cloudera Director)
  • CI/CD (Git / Jenkins)
  • Orchestration and configuration management (Chef)
  • Experience with Spark clusters, both permanent (Cloudera)
  • Experience in large scale distributed processing architectures e.g. enterprising caching, low latency data driven platforms (Enterprise Data Warehouse preferably)
  • Expertise in fault tolerant systems in AWS including Clustering & multi AZ deployments
  • Working knowledge of setting up + running Big Data clusters e.g. Hadoop, Spark, Impala supported by various Databases, object stores (S3) on AWS


SearchBI are the first choice for Business Intelligence and Agile BI recruitment business in the UK and internationally. We work with Software Houses, Partners and End Users throughout the UK and Europe. As unique specialists in this niche marketplace we have positioned ourselves ideally to satisfy the needs of a variety of clients. We understand our industry and use our expansive network to position ourselves as market leaders for all Agile Business Intelligence requirements.


Keywords – Data Engineer / Platform Engineer / Big Data Engineer / AWS Solutions Architect

Candidate Application

You can apply to this job and others using your online resume. Click the link below to submit your online resume and email your application to this employer.