Building GPT Without Training: Exploring Math-Driven Text Generation

The article explores the possibility of generating text without any machine learning training, using only mathematical techniques like co-occurrence matrices, PPMI, and SVD. While the resulting word embeddings capture semantic relationships well, generating coherent text proves challenging without incorporating a bigram grammar model.

Details

The author, Shivnath Tathe, set out to explore whether it's possible to generate text without any machine learning training, relying solely on mathematical techniques. He started with a vector database called nanoVectorDB, built using the WikiText-103 corpus. The pipeline involved creating a co-occurrence matrix, calculating PPMI (Positive Pointwise Mutual Information), performing SVD to get 64-dimensional word embeddings, and building a bigram grammar matrix. This purely mathematical approach was able to capture semantic relationships between words very well, as demonstrated by accurate word analogies. However, generating coherent text proved challenging using just the semantic information. Attempts at text generation using only cosine similarity or only the bigram grammar model resulted in repetitive or generic output. The breakthrough came with a two-stage approach - first using the semantic embeddings to filter a set of candidate next words, then reranking them based on the bigram grammar model. This combination of semantic and grammatical information allowed the system to produce coherent narratives, such as a military-themed sequence of events.