TransactionGPT as a Payments Foundation Model

This episode explores TransactionGPT, a Visa Research paper that argues for a foundation-model approach to consumer transaction data spanning generation, anomaly detection, and representation learning. It explains why payment histories are fundamentally different from text or simple time series: each event mixes merchant IDs, amounts, timestamps, and engineered risk signals, creating a multi-modal, temporal, tabular structure that demands more schema-aware modeling. The discussion walks through the paper’s 1D, 2D, and 3D architecture progression, highlighting how separate transformers for transaction metadata, downstream features, and behavioral sequences aim to avoid the pitfalls of flattening everything into a single embedding space. Listeners would find it interesting for its clear debate over whether TransactionGPT is a genuine reusable backbone for payments or mainly a strong engineering response to real-world constraints like heterogeneity, scale, regulation, and low-latency fraud decisioning. Sources: 1. TransactionGPT — Yingtong Dou, Zhimeng Jiang, Tianyi Zhang, Mingzhi Hu, Zhichao Xu, Shubham Jain, Uday Singh Saini, Xiran Fan, Jiarui Sun, Menghai Pan, Junpeng Wang, Xin Dai, Liang Wang, Chin-Chia Michael Yeh, Yujie Fan, Yan Zheng, Vineeth Rakesh, Huiyuan Chen, Guanchu Wang, Mangesh Bendre, Zhongfang Zhuang, Xiaoting Li, Prince Aboagye, Vivian Lai, Minghua Xu, Hao Yang, Yiwei Cai, Mahashweta Das, Yuzhong Chen, 2025 http://arxiv.org/abs/2511.08939 2. TabDPT: Scaling Tabular Foundation Models on Real Data — Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Alex Labach, Hamidreza Kamkari, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L. Caterini, Maksims Volkovs, 2024 https://arxiv.org/abs/2410.18164 3. Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences — Piotr Skalski, David Sutton, Stuart Burrell, Iker Perez, Jason Wong, 2024 https://arxiv.org/abs/2401.01641 4. Open Banking Foundational Model: Learning Language Representations from Few Financial Transactions — Gustavo Polleti, Marlesson Santana, Eduardo Fontes, 2025 https://arxiv.org/abs/2511.12154 5. TransactionGPT — Yingtong Dou, Zhimeng Jiang, Tianyi Zhang, Mingzhi Hu, Zhichao Xu, Shubham Jain, Uday Singh Saini, Xiran Fan, Jiarui Sun, Menghai Pan, Junpeng Wang, Xin Dai, Liang Wang, Chin-Chia Michael Yeh, Yujie Fan, Vineeth Rakesh, Huiyuan Chen, Mangesh Bendre, Zhongfang Zhuang, Xiaoting Li, Prince Aboagye, Vivian Lai, Minghua Xu, Hao Yang, Yiwei Cai, Mahashweta Das, Yuzhong Chen, 2025 https://arxiv.org/abs/2511.08939 6. TabTransformer: Tabular Data Modeling Using Contextual Embeddings — Xin Huang, Ashish Khetan, Milan Cvitkovic, Zohar Karnin, 2020 https://arxiv.org/abs/2012.06678 7. Tabular Transformers for Modeling Multivariate Time Series — Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, Erik Altman, 2021 https://arxiv.org/abs/2011.01843 8. FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data — Dongyu Zhang, Liang Wang, Xin Dai, Shubham Jain, Junpeng Wang, Yujie Fan, Chin-Chia Michael Yeh, Yan Zheng, Zhongfang Zhuang, Wei Zhang, 2023 https://arxiv.org/abs/2310.13818 9. Multi-modal Time Series Analysis: A Tutorial and Survey — Yushan Jiang, Kanghui Ning, Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, Dongjin Song, 2025 https://arxiv.org/abs/2503.13709 10. Credit card fraud detection using machine learning: A survey — Yvan Lucas, Johannes Jurgovsky, 2020 https://arxiv.org/abs/2010.06479 11. Interleaved Sequence RNNs for Fraud Detection — Bernardo Branco, Pedro Abreu, Ana Sofia Gomes, Mariana S. C. Almeida, João Tiago Ascensão, Pedro Bizarro, 2020 https://arxiv.org/abs/2002.05988 12. FraudTransformer: Time-Aware GPT for Transaction Fraud Detection — Gholamali Aminian, Andrew Elliott, Tiger Li, Timothy Cheuk Hin Wong, Victor Claude Dehon, Lukasz Szpruch, Carsten Maple, Christopher Read, Martin Brown, Gesine Reinert, Mo Mamouei, 2025 https://arxiv.org/abs/2509.23712 13. Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings — Xiran Fan, Zhimeng Jiang, Chin-Chia Michael Yeh, Yuzhong Chen, Yingtong Dou, Menghai Pan, Yan Zheng, 2025 https://scholar.google.com/scholar?q=Enhancing+Foundation+Models+in+Transaction+Understanding+with+LLM-based+Sentence+Embeddings 14. Self-Attentive Sequential Recommendation — Wang-Cheng Kang, Julian McAuley, 2018 https://scholar.google.com/scholar?q=Self-Attentive+Sequential+Recommendation 15. TabLLM: Few-shot Classification of Tabular Data with Large Language Models — Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, David Sontag, 2022 https://scholar.google.com/scholar?q=TabLLM%3A+Few-shot+Classification+of+Tabular+Data+with+Large+Language+Models 16. TimeGPT-1 — Azul Garza, Cristian Challu, Max Mergenthaler-Canseco, 2023 https://scholar.google.com/scholar?q=TimeGPT-1 17. ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning — Junguang Jiang et al., 2023 https://scholar.google.com/scholar?q=ForkMerge%3A+Mitigating+Negative+Transfer+in+Auxiliary-Task+Learning 18. Identification of Negative Transfers in Multitask Learning Using Surrogate Models — Dongyue Li, Huy L. Nguyen, Hongyang R. Zhang, 2023 https://scholar.google.com/scholar?q=Identification+of+Negative+Transfers+in+Multitask+Learning+Using+Surrogate+Models 19. Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers — Gjergji Kasneci, Enkelejda Kasneci, 2024 https://scholar.google.com/scholar?q=Enriching+Tabular+Data+with+Contextual+LLM+Embeddings%3A+A+Comprehensive+Ablation+Study+for+Ensemble+Classifiers 20. LLM Embeddings Improve Test-time Adaptation to Tabular Y|X-Shifts — Yibo Zeng et al., 2024 https://scholar.google.com/scholar?q=LLM+Embeddings+Improve+Test-time+Adaptation+to+Tabular+Y%7CX-Shifts 21. LLM Embeddings for Deep Learning on Tabular Data — Boshko Koloski et al., 2025 https://scholar.google.com/scholar?q=LLM+Embeddings+for+Deep+Learning+on+Tabular+Data 22. TableGPT2: A Large Multimodal Model with Tabular Data Integration — Aofeng Su et al., 2024 https://scholar.google.com/scholar?q=TableGPT2%3A+A+Large+Multimodal+Model+with+Tabular+Data+Integration 23. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp3 24. AI Post Transformers: Relational Graph Transformer for Multi-Table Learning — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-11-relational-graph-transformer-for-multi-t-57cce3.mp3 25. AI Post Transformers: Unembedding Matrices as Feature Lenses for Embeddings — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-08-unembedding-matrices-as-feature-lenses-f-4dc415.mp3 Interactive Visualization: TransactionGPT as a Payments Foundation Model