40 min
Before
~1 min
After
97% faster
Improvement
During QA testing, our Product Owner tried importing a ~100MB Excel file for automotive inventory data. The process took up to 40 minutes with no response indicating whether it succeeded or failed. He raised the issue and I was assigned to fix it.
After reviewing the codebase, I found four root causes:
Moved processing to the application layer
JSON transformation and data shaping were happening inside SQL queries. I moved this logic to the application layer, reducing database load significantly.
Refactored queries to use optimized joins
Replaced nested and N+1 queries with proper JOIN-based queries, eliminating redundant round trips to the database.
Replaced row-by-row inserts with bulk insert
Each row was being inserted individually inside a loop. I replaced this with a bulk insert strategy that batches all rows into a single operation.
Added upfront validation
Errors were only caught mid-import, forcing full reruns. I added validation at the start of the pipeline so invalid data is rejected before any processing begins.
Import time dropped from 40 minutes to 1 minute — 97% faster. The operations team could run imports multiple times a day without their workflow being blocked.