III. Featured Engineering Lab & Works
Production Systems & Architecture Case Studies
Real production systems built and scaled across Walmart, Twilio, and enterprise data ecosystems. High-throughput distributed platforms, autonomous agents, and lakehouses.
Agentic AIRole: Staff Architect & Lead EngineerScale: Petabyte-scale, 5,000+ daily production Spark pipelines
SparkPlug: Autonomous Spark Optimization & RCA Platform
Agentic AI platform automating Spark workload migrations, runtime tuning, and root cause failure analysis.
Measured Production Impact:Reduced average Spark incident triage from 3 hours to 4 minutes; drove 28% compute cost reduction on skewed batch pipelines.
The Engineering Challenge
Upgrading distributed Spark jobs across thousands of pipelines required hundreds of engineering hours, manual log combing, and trial-and-error memory/skew tuning.
Architecture & Solution
Designed an autonomous multi-agent system utilizing LangGraph and custom Spark log parsers. Comprises three specialized agents: (1) Upgrade Diagnostic Agent, (2) Skew & Spill Optimizer Agent, and (3) RCA Log Analyzer Agent.
Key Architectural Decisions
Separated deterministic plan analysis (EventLog parsing) from LLM reasoning to ensure zero hallucination on metrics.
Used Model Context Protocol (MCP) to securely query cluster metrics without direct shell access.
Implemented verification loops where proposed Spark config changes are simulated before PR creation.
Technology Stack & Tools:
LangGraphPythonApache Spark 3.xMCPFastAPIVector Store
Staff Engineering Lessons Learned
- •Agents need high-fidelity deterministic tools; asking an LLM to read raw 500MB logs fails context limits and inflates cost.
- •Domain-specific abstractions beat generic prompts every time.