Data Engineer
Job title: Data Engineer
Location: UK
Duration: 6-month contract Inside IR35
Working Mode: Fully Remote
Vetting: Active SC Clearance
Job Description
Data Engineer Skills and Technologies Data Engineering: – Strong understanding of data concepts – data types, data structures, schemas (both JSON and Spark), schema management etc – Strong understanding of complex JSON manipulation – Experience working with Data Pipelines using a custom Python/PySpark frameworks – Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data. – Strong understanding of Data Security principles – data owners, access controls – row and column level, GDPR etc including experience of handling sensitive datasets – Strong problem solving and analytical skills, particularly able to demonstrate these intuitively
Requirements
Required Skills
Languages / Frameworks – JSON – YAML – Python (as a programming language, not just able to write basic scripts. Pydantic experience would be a bonus.) – SQL – PySpark – Delta Lake – Bash (both CLI usage and scripting) – Git – Markdown – Scala (bonus, not compulsory) – Azure SQL Server as a HIVE Metastore (bonus) Technologies – Azure Databricks – Apache Spark – Delta Tables – Data processing with Python – PowerBI (Integration / Data Ingestion) – JIRA
apply to this job