SayQL: Natural Language to SQL with GPT-Neo

Plenty of people who depend on a database can't get at their own data because they don't write SQL. SayQL fine-tuned an open-source language model to close that gap, turning a plain English question into a SQL query. GPT-Neo 1.3B was trained with a causal language-modeling objective on 7,000 question/SQL pairs from the Spider dataset, split 80/10/10 into train, validation, and test. Each example became a single string (the question, then the query, then a STOP marker) so the model learned to write the query after the question and stop on its own. Early stopping watched the validation set, and every reported number came from the test split, which was never touched during training. Training ran 13 of a possible 50 epochs before early stopping, restoring the best checkpoint at epoch 3 (validation loss 0.944). On the held-out test split the restored model measured loss 0.935 and perplexity 2.55. Against the un-tuned baseline over 50 random test questions, BLEU rose from 0.002 to 0.455, exact-match accuracy from 0% to 20%, and valid-SQL rate from 10% to 90%. Simple queries came out nearly exact. Complex joins failed in one specific, explainable way: the model invented table and column names, which followed directly from the fact that it never saw a database schema, only the question.

Dataset

Spider (7,000 question/SQL pairs, split 80/10/10 into train/validation/test)

Models

GPT-Neo 1.3B fine-tuned with a causal LM objective via the HuggingFace Trainer, fp16 on an A100, early stopping on validation loss with a patience of 10 epochs

Evaluation

Test-set loss and perplexity, smoothed sentence-level BLEU, exact-match accuracy, valid-SQL rate via a sqlglot parse check

Results & insight

Valid-SQL rate 10% to 90%, BLEU 0.002 to 0.455, exact match 0% to 20%, test perplexity 2.55. Remaining failures traced to missing schema context rather than broken syntax.