A large fintech platform ran its partner payouts and reconciliation as a nightly batch built on database stored procedures. It took about 12 hours, and as volume grew it was failing to scale. A batch that long leaves no room for a rerun: one bad night delays payouts and creates real operational risk.
- Runtime cut from 12 hours to 30 minutes (96% faster)
- Zero missed payouts during the migration
- 100% payout matching accuracy
- Full observability and alerting
- Phase 2 base version delivered in 3 weeks
Why stored procedures hit a wall
Stored procedures are fine until the work outgrows one database. All the processing competes with everything else that database is doing, it scales only as far as the biggest instance you can buy, and the logic is hard to test or observe. Moving the work to a distributed engine fixes all three.
Phase 1: get more out of what's there
The first phase optimized the existing MySQL stored procedures. Tuning what's already in production is the fastest, lowest-risk win, and it buys breathing room while the bigger change is built.
Phase 2: move the heavy lifting off the database
The second phase changed the shape of the pipeline: data is exported from MySQL to S3, processed in PySpark on AWS Glue, and the results are written back to the database. The base version of this phase shipped in three weeks. The database goes back to serving the application instead of doing hours of batch work, and processing that ran serially now runs in parallel across the cluster.
Migrating money logic safely
The hard part of a payout migration isn't speed, it's correctness. The legacy logic encoded years of business rules, and a single wrong number is a real problem for a real partner. The bar was simple: the new pipeline had to match the old one exactly before it took over, and the cutover had to happen with no downtime and no missed payouts. It did both.
Make it observable
The old batch was a black box until it finished or failed. The new pipeline has full observability and alerting, so problems surface quickly instead of the next morning.
The takeaway
If a critical batch job has quietly become your longest-running process, tune it first to buy time, then fix the architecture. Done in phases, that can take weeks rather than quarters.