By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Zero-shot transformer models predict directly from relational database structures accurately
- The speaker presents a model that uses relational databases (including foreign/primary key structure) directly rather than flattening tables for traditional feature engineering.
- Relational data is represented as a graph, and a neighborhood around the prediction target is sampled (e.g., via DFS/BFS) to gather relevant context from connected tables.
- The sampled graph context is serialized into a sequence of cells and embedded with type-specific encoders (numeric, text, Boolean) before being processed by a transformer.
- Novel masked attention mechanisms (column, feature, neighborhood attention) help the model learn distributions and selectively aggregate information from columns and neighboring entities.
- Results shown indicate the specialized small model can outperform general LLMs on relational prediction tasks (e.g., higher AUROC with ~22M parameters vs multi-billion-parameter LLMs).
IDEAS WORTH REMEMBERING
5 ideasKeeping relational structure avoids information loss from flattening.
Traditional workflows often denormalize tables and hand-engineer features, which can discard important primary/foreign key relationships; treating the database as a graph preserves these signals.
Local neighborhood sampling provides task-relevant context efficiently.
Instead of ingesting the entire database, the model samples a context subgraph around the entity being predicted, capturing related records (e.g., a user and nearby connected entities) with controllable scope.
Serializing graph context enables transformer-based learning on databases.
After sampling, the system converts the neighborhood into a sequence of cells, making the input compatible with sequence models while still reflecting multi-table context.
Type-aware encoders are essential for mixed-schema data.
Relational databases contain heterogeneous value types; separate numeric, text, and Boolean encoders produce embeddings suited to each modality before attention-based aggregation.
Masked attention lets the model learn distributions and isolate signal sources.
Column attention can focus on a specific field across sampled neighbors to infer distributions, while feature and neighborhood attention separately emphasize the target entity’s attributes vs. connected entities’ information.
WORDS WORTH SAVING
5 quotesSo, um, we built the first, um, zero-shot predictive models for relational databases.
— Mark Žnidar
But we can say, "Hey, what about if we can make a model that doesn't require a data scientist, which kind of speeds up the exploration of your data, um, and maybe even gives you better predictions?"
— Mark Žnidar
So essentially, yeah, we develop a model which consumes the relational database as a primary modality.
— Mark Žnidar
It does very, very well. Like in comparison, for example, um, and this is also saying that LLMs are very bad- working with relational data.
— Mark Žnidar
So what is the kind of probability that, um, an item will churn is like the classification performance like 62 AUROC for Gemma, a four billion model, while it's at like 73 for our model.
— Mark Žnidar
SPEAKERS
Ankit Gupta
hostHost/interviewer on YC Root Access (Y Combinator).
Mark Žnidar
guestResearcher/engineer presenting zero-shot predictive modeling for relational databases using transformer-style architectures and specialized attention.
High quality AI-generated summary created from speaker-labeled transcript.
