SPSS Modeler to Databricks Migration
Retire SPSS Modeler without betting the farm on it
A few hundred streams, some scoring customers every night, most of them older than the people now asking what they do. Here’s the three-tier factory we use to move them onto Databricks – deliberately, with evidence attached.
of a typical SPSS estate never needs migrating at all
of streams convert on certified, deterministic node mappings
tiers, one validation loop
Why This Is Hard
Nobody is quite sure what all of them do anymore, as the people who built them have moved on.
The licence renewal is what forces the decision. Knowing which streams still matter is what makes it hard and it’s rarely a coding problem so much as a knowledge problem with a deadline attached.
Two Approaches That Fall Short
Rules alone stall. An LLM alone can’t be trusted at volume.
Pure Rule-Based Conversion
Reliable, until it isn’t
Works well for:
Standard sources, filters, derives, merges, and aggregates – roughly 70% of a typical estate.
Breaks on:
Hand-written CLEM with side effects, scripted supernodes, exotic node types used once and never touched again.
“Just ask an LLM”
Plausible, until it drifts
Works well for:
Exactly that long tail, one-off logic no rulebook was built to cover.
Breaks on:
High-volume, well-understood patterns. Slower, costlier, and non-reproducible, you’d re-validate every output, forever.
The Factory
Deterministic Converter
Parses stream XML, translates standard node types into PySpark and Databricks Spark SQL with certified, reproducible mappings.
LLM Translation
Complex CLEM, scripted and exotic nodes – confidence-tagged, with SME questions auto-generated where intent is ambiguous.
Human Review
Engineers and SMEs spend time only where judgment is genuinely required – usually a small, high-stakes slice.
Validation Loop
Node-to-code traceability, reconciliation notebooks, parallel-run comparisons with interpreted diffs. The evidence pack already exists when model risk asks for it.
The Migration Is The Smaller Half of The Value
Inventory First
Complexity scoring, redundancy detection, dead-stream ID: scope on evidence, not folklore.
Model Migration
Artifacts land in MLflow with score-parity evidence, the same ground as our banking and financial services work.
Docs-as-code
Business logic reverse-engineered into readable, versioned documentation before the knowledge retires.
Modernize
Medallion, Delta Live Tables, and Unity Catalog lineage – not embalmed as-is.
What This Looks Like In Practice
Weeks 1–3
Assessment and census. Triage results and a per-tier breakdown of your actual estate – scope, effort, risk on one page.
Migration Waves
Factory throughput. Quick wins and retirements first, regulatory-critical streams with full SME involvement where it counts.
Parallel Run
Prove it, then cut over. Recon notebooks and interpreted diffs make the parity case, stream by stream.
See The Numbers On Your Own Estate
We’ll run the assessment on a sample of your streams and show you the per-tier breakdown.
