Type naturally. The most likely next word appears in italic ghost text, and Tab accepts it instantly.
model readyThe training plot shows how the model behaved during training and validation across epochs, making it easier to inspect convergence, overfitting, and stability.
This project is a GPT-style next-word prediction system built end to end: raw text is collected, cleaned, tokenized, converted into training sequences, trained with a causal Transformer model, saved, and served through Flask for live inference. The model is not a generic API wrapper; it uses a locally saved Keras model, tokenizer, and config file.
Language modeling depends heavily on corpus quality. The ideal dataset mixes classic prose, simple encyclopedic text, conversations, scripts, news, and easy-reader material. This gives the model narrative flow, factual modern wording, dialogue structure, formal sentences, and simpler sentence patterns. Poor data cannot be repaired by architecture alone; if the corpus is noisy or one-dimensional, the generated language will inherit that weakness.
Raw text contains headers, copyright lines, HTML, URLs, chapter labels, number-only lines, duplicated sentences, symbols, and non-language fragments. Cleaning removes these artifacts before tokenization so the vocabulary is not polluted. The pipeline lowercases text, removes markup and URLs, fixes HTML entities, filters low alphabetic-ratio lines, drops very short fragments, chunks very long lines, filters excessive OOV content, and removes duplicates.
The model sees integer IDs, not raw text. A word-level tokenizer maps each known word to an ID, with index 0 reserved for padding and index 1 reserved for OOV words. The tokenizer uses an apostrophe-safe regex so words like don't and mom's stay as single tokens instead of being split into junk fragments. The same tokenizer must be used during training and inference.
The configured vocabulary keeps the most frequent words and maps rare words to OOV. A larger vocabulary improves coverage but increases the output layer size and parameter count. A smaller vocabulary trains faster but loses rare-word specificity.
Training examples are built with a sliding window. Given a sentence, previous words become the input and the next word becomes the label. Contexts are padded to a fixed sequence length so every batch has the same tensor shape.
The architecture follows a decoder-only Transformer idea. Token embeddings convert word IDs into dense vectors. Positional embeddings add order information. Causal self-attention lets each position attend only to previous positions, never future words. Feed-forward layers refine the representation. Residual connections and layer normalization stabilize learning. A final LastToken layer extracts the last context position, and the dense softmax layer predicts probabilities over the vocabulary.
The model is trained with sparse categorical cross-entropy because the target is one next-word ID. AdamW is used for adaptive optimization with weight decay. Gradient clipping helps avoid unstable updates. Dropout reduces overfitting. ReduceLROnPlateau lowers the learning rate when validation loss stalls, while EarlyStopping can stop training when improvement no longer appears.
Validation loss measures how well the model predicts held-out next words. Accuracy is useful but limited because many next words can be valid. Perplexity, calculated as exp(loss), estimates how many plausible words the model is uncertain between. A lower perplexity means the model is more confident and generally more useful.
For live prediction, the browser sends the current editor text to /api/complete. Flask tokenizes it, keeps the latest context window, pads it, runs TensorFlow, applies temperature scaling, and returns top candidates. The UI renders the best candidate as italic ghost text. Tab on desktop and Enter on mobile can accept that suggestion.
Temperature changes the probability distribution. Lower temperature makes high-confidence words stronger and output safer but more repetitive. Higher temperature spreads probability across more options and makes output more creative but riskier. Top-K restricts sampling to the strongest K candidates, preventing very unlikely words during story or paragraph generation.
Generation repeatedly predicts one word, appends it to the same editor field, and feeds the expanded context back into the model. This continues until the max-word limit is reached. The UI auto-scrolls to the bottom as generated text grows so the newest words remain visible.
Serving requires three key artifacts: best_model.keras for architecture and weights, tokenizer.pkl for word-index mappings, and model_config.json for hyperparameters. Custom layers such as PositionalEmbedding, TransformerBlock, and LastToken must be provided when loading the model. Flask then exposes prediction, generation, perplexity, and model-info endpoints.