CHAPTERS
- 0:07 – 0:08
Problem: Predictive modeling loses relational structure when you flatten tables
Mark explains how companies often flatten relational databases into a single table for feature engineering, which discards key information. The goal is to let models directly use primary-key/foreign-key relationships without requiring a data scientist-heavy workflow.
- •Traditional approach: flatten relational data into one table
- •Flattening loses PK/FK relationship information
- •Feature engineering becomes manual and slow
- •Motivation: faster exploration and potentially better predictions
- 0:08 – 0:37
Core idea: A zero-shot predictive model that consumes the relational database directly
Mark introduces their main contribution: the first zero-shot predictive models designed for relational databases as the primary modality. The aim is to reduce dependency on bespoke data science pipelines while improving predictive performance.
- •Zero-shot predictive modeling over relational databases
- •Model input is the database itself (not a flattened dataset)
- •Reduces the need for a data scientist to craft features
- •Designed to speed up predictive exploration
- 0:37 – 1:07
Reframing relational databases as graphs of connected tables
The model treats the relational schema and table links as a graph, where tables/entities connect via relationships. This abstraction enables leveraging neighborhood context around the entity being predicted.
- •Relational database can be represented as a graph
- •Tables/entities become nodes; relationships become edges
- •Focus on predicting for a specific entity/row (e.g., an article)
- •Use the surrounding graph context as predictive signal
- 1:07 – 1:37
Building context: Sampling a neighborhood subgraph around the target entity
To make prediction feasible, the system samples a local neighborhood graph around the entity of interest. This sampled context captures relevant information from related tables without requiring full-database traversal.
- •Collect neighboring context around the target row/entity
- •Neighborhood sampling builds a manageable subgraph
- •Context includes linked tables and their attributes
- •Emphasis on relational context rather than manual joins
- 1:37 – 1:56
Turning graph context into model input via traversal-based feature collection
Mark describes walking the neighborhood graph (DFS/BFS) to collect values/features along paths. This produces structured information that can later be transformed into a sequence suitable for transformer-style modeling.
- •Traverse the sampled graph to collect features
- •Use DFS/BFS-style walks to gather context
- •Features come from connected records across tables
- •Prepares relational context for downstream serialization
- 1:56 – 2:26
Serialization: Converting relational context into a sequence of cells
The sampled and traversed context is serialized into an ordered sequence of cell values. Each cell is then embedded using type-appropriate encoders so heterogeneous database fields can be modeled together.
- •Serialize neighborhood context into a sequence
- •Cells can include categorical, numeric, boolean, and text values
- •Use different encoders per data type (number/text/boolean)
- •Produces embeddings for transformer consumption
- 2:26 – 2:49
Novel attention mechanisms: Column attention to learn distributions across context
A key innovation is masked attention tailored to relational data. Column attention lets the model attend over values from a specific column across sampled neighbors, effectively learning useful distributions (e.g., gender proportions).
- •Use masked/specialized attention mechanisms
- •Column attention focuses on one column across neighborhood samples
- •Learns contextual distributions (e.g., 90/10 split)
- •Distributional context improves downstream prediction
- 2:49 – 3:23
Clarifying column attention: Attending over column values in the neighborhood
Ankit asks whether attention is over the column itself or its values; Mark clarifies it attends over values from that column across the sampled neighborhood. This enables learning patterns like salary distributions among friends/peers.
- •Attention is over observed column values in context
- •Neighborhood sampling can include related entities (e.g., friends)
- •Model can isolate a single feature/column across entities
- •Supports better inference using relational peer information
- 3:23 – 4:15
Additional attentions: Feature and neighborhood attention (and dropping pool attention)
Mark outlines feature attention to concentrate on the target entity’s own attributes and neighborhood attention to emphasize neighbor-derived signals. Pool attention was tested but removed because it didn’t add meaningful gains.
- •Feature attention aggregates target-entity information
- •Neighborhood attention aggregates relevant neighbor information
- •Pool (non-mask) attention showed little benefit
- •Final representation feeds classification/regression heads
- 4:15 – 4:48
Performance results: Outperforming LLM baselines on relational prediction tasks
They compare against LLMs and show strong gains, emphasizing that general LLMs struggle with relational data. Mark cites an example where their smaller model achieves substantially higher AUROC than a 4B-parameter baseline.
- •LLMs are described as weak on relational data tasks
- •Example: item-churn AUROC ~62 (Gemma 4B) vs ~73 (their model)
- •Their model is far smaller (~22M parameters)
- •Demonstrates efficiency + accuracy benefits
- 4:48 – 5:22
Deployment vision: Pairing a small relational model with larger agentic systems
Ankit proposes using this model as a specialist component for high-performance database prediction, alongside a larger model for other tasks. Mark references related work (Kumu.ai) using agentic protocols to ask questions and generate SQL for predictive queries.
- •Use specialist relational model for predictive quantities
- •Combine with larger models/agents for broader workflows
- •Agentic systems can generate SQL to query predictions
- •Industry-scale direction inspired by Kumu.ai work
- 5:22 – 5:30
Wrap-up and acknowledgments
The conversation concludes with thanks and encouragement to continue the work. This closing segment reinforces enthusiasm for the approach and its future developments.
- •Host thanks Mark for presenting
- •Positive feedback on the research direction
- •Conversation ends with closing remarks
