Open to new opportunities

Osama Shahid

Senior Data Engineer

I build data platforms that move 50M+ records a day — reliably, observably, and on time.

0years in production data
0TB migrated with <5 min downtime
0records processed daily
Osama Shahid

About

Data pipelines are products.
I treat them that way.

I'm a Senior Data Engineer with 4 years at TCS, building production data platforms on AWS and Azure Databricks for a Fortune 500 pharmaceutical client — from CDC-based migrations of 15+ TB to medallion lakehouses serving 40+ dashboards.

My happy place is the unglamorous middle: schema evolution that doesn't page anyone at 3 AM, backfills that just work, and streaming pipelines with honest latency numbers.

0fewer schema incidents
0fewer pipeline failures
0faster Spark jobs

Experience

Where I've shipped

IT Analyst — Senior Data Engineer, Tech Lead

Nov 2022 — Present

Tata Consultancy Services · Client: Fortune 500 pharmaceutical (Life Sciences) · Noida, India

  • Migrated 15+ TB of biomedical and commercial data from SQL Server to Aurora PostgreSQL with under 5 minutes of downtime — CDC pipelines on DMS, Glue PySpark and Step Functions, 120+ stored procedures converted.
  • Built bronze–silver–gold lakehouses on Azure Databricks processing 50M+ records/day at 15-minute freshness — Auto Loader ingestion, Delta Live Tables, Unity Catalog governance.
  • Cut schema-related incidents ~90% with metadata-driven ingestion and data-quality frameworks adopted across 80+ pipelines.
  • Tech lead of a 4-engineer pod; cut pipeline failures 40% across 30+ Airflow DAGs with retries, backfills and SLA alerting.

Projects

Things I've built

sql-rag

Natural-language to SQL studio. Hybrid dense + BM25 retrieval with reciprocal-rank fusion, schema pruning, and a repair loop that fixes its own bad queries.

PythonRAGStreamlitSQLite

pharma-rag

Pharmaceutical Q&A over DailyMed drug labels. Hybrid retrieval, conversational follow-ups, and a ReAct agent mode — every answer fully cited.

PythonAgentsChromaDBGemini

ps5-booking

Real-time slot booking platform. Manual approval flow, overlap-safe reservations, and email confirmations on a serverless stack.

VercelSupabasePostgreSQL

privatewire-vpn

Personal WireGuard VPN. Kotlin Android client paired with one-command server provisioning scripts.

KotlinWireGuardUbuntu

Skills

The toolbox

Languages

PythonPySparkSQLSpark SQLJavaScript

AWS

GlueLambdaStep FunctionsDMSS3Aurora PostgreSQLMSKEventBridgeAthena

Azure & Databricks

DatabricksDelta LakeUnity CatalogDelta Live TablesAuto LoaderDatabricks SQLADF

Data Engineering

ETL / ELTCDCSpark Structured StreamingMedallion ArchitectureDimensional ModelingData QualityData LineageREST APIs

DevOps

AirflowGitHub ActionsDockerCloudFormationTerraformGit

Certifications

AWS Certified Data Engineer — AssociateAzure Database Administrator Associate

Contact

Let's build something
that scales.

Currently open to data engineering roles and interesting collaborations.