An exciting opportunity is available to work at the forefront of an industry-academic collaboration, by providing a combination of data engineering expertise and Azure Cloud administrator support and to a major research project working on Data Science, the creation of a Digital Twin and Machine Learning. The project is in its early stages and will require the design and set up of data pipelines in Azure from multiple sources, with several data processing steps, the creation of a data lake and data warehouse, and integration with on-site systems.

We are looking for an experienced and highly capable data engineer with AZURE experience to join our accomplished team, who can help us to design and implement this critical infrastructure to meet the needs of this large and exciting research collaboration. The candidate will be working with the Azure Cloud Architect to meet the needs of the research engineering community. This includes the creation of data pipelines to transform raw data to curated datasets and will require proven programming and data engineering experience. A good knowledge of cyber security issues, vulnerabilities, and ways to manage and address them.

Follow the below link for full details on the project-

https://www.turing.ac.uk/research/research-projects/project-bluebird-ai-...

Essential Skills - "Azure Data Engineer"

• A minimum of 3 years' experience designing successful solutions to the Cloud
• Data Engineering & pipeline design for data science applications
• Proven understanding of Data Science and Machine Learning concepts
• Minimum of 3 years hands on Data Engineering and Software experience
• Proven experience in data processing with Python, including a good working knowledge of PySpark, Sklearn, Pandas, Numpy, SciPy
• In depth understanding of available Azure services including:
  • Data Lake, Data Warehouse, DataFactory, Databricks
  • Structured and unstructured data storage
  • Notebooks
  • Active Directory
  • Database options: SQL, NoSQL
  • Virtual Machines
  • DevOps
  • Azure Container Instances, Docker
  • Azure functions and Logic Apps
• Working with unstructured data in Azure data pipelines
• Workflows involving the use of DevOps and source control
• Evidence of developing sound, explainable design decisions and judgement
• Building ETL and ELT pipelines

"Desirable Skills - Azure Data Engineer"

• Experience working in research environments in both academia and industry
• Building Continuous Integration Pipelines
• Good understanding of cyber security & implementation for sensitive datasets with experience of working with sensitive datasets
• Experience using IaC technologies, especially Terraform and ARM templates
• Implementation of data pipelines using PySpark within Databricks
• Building complex SQL queries
• Proven experience of database setup and implementation for both SQL and NoSQL databases
• Microsoft Azure Certifications
  • AZ-300 and AZ-301 (Architect Technology)
  • DP-200 and DP-201 (Data Engineering)

• Experience of Hybrid systems with cloud and on-prem elements

Qualifications: PhD in Computer Science, Mathematics, Engineering, Physics or equivalent level of qualification and/or in related fields.