Data Engineer

Overview

Helda Technologies is seeking a Data Engineer with backend experience to manage and enhance data pipelines as we integrate healthcare organizations' data onto our platform in Lagos, Nigeria. You will transform raw hospital data into clean, structured formats within Supabase to support accurate analytics.

Key Responsibilities

• Lead the technical onboarding of new healthcare organizations by receiving their data exports, evaluating data quality, aligning data columns to Helda's standard schemas, and setting up data ingestion pipelines.
• Manage diverse data formats such as inconsistent CSV files, multi-sheet Excel workbooks, database dumps, EMR exports with vendor-specific features, and FHIR API responses.
• Collaborate directly with healthcare contacts to interpret data meaning, assess field completeness, and understand formatting conventions like date and currency.
• Document each organization's data profile including source systems, delivery methods, column mappings, data quality issues, and transformation rules.
• Execute automated validation tests on all new data loads to ensure compliance with schemas, correct data types, valid value ranges, and analytics readiness.
• Diagnose and resolve validation errors such as missing columns, unexpected data types, out-of-range values, duplicates, and encoding problems.
• Update and extend validation checks when novel data scenarios arise, for example, unique claim status taxonomies or unconventional drug name formats.
• Perform manual reviews of analytics outputs post-data ingestion to confirm meaningful KPI calculations, trend analyses, and AI-generated insights.
• Oversee pipeline health for all active clients, monitoring for load failures, state inconsistencies, schema drifts, and data quality alerts.
• Troubleshoot pipeline errors including parsing failures, connection timeouts, API rate limiting, and changes in source schemas.
• Conduct incremental data updates for organizations providing new data months, corrections, or backfills.
• Maintain pipeline configurations such as securely rotated credentials, schedules, and retry strategies.
• Develop and upkeep data transformation scripts to standardize organization-specific quirks: renaming columns, converting date formats, normalizing currencies, and standardizing categorical fields.
• Create and manage mapping tables supporting categorical standardization.
• Cleanse and deduplicate data where source quality issues exist, thoroughly documenting all transformations applied.