The PTB model can support small, medium, and large dataset configurations. The small model should be able to reach perplexity below 120 on the test set and the large one below 80, though it might take several hours to train.

We executed the model on the small size dataset. To do this, simply type in your command prompt the following:

    python ptb_word_lm --data_path=/tmp/simple-examples/data/ --model small 

This is where you extracted the previously downloaded PTB dataset in /tmp/simple-examples/data/.

The preceding perplexity value after eight hours of training and 13 training epochs are as follows:

Epoch: 1 Learning rate: 1.000 
0.004 perplexity: 5263.762 speed: 391 wps 
0.104 perplexity: 837.607 speed: 429 wps 
0.204 perplexity: 617.207 speed: 442 wps 
0.304 perplexity: 498.160 speed: 438 wps 
0.404 perplexity: 430.516 speed: 436 wps 
0.504 perplexity: 386.339 speed: 427 wps 
0.604 perplexity: 348.393 speed: 431 wps 
0.703 perplexity: 322.351 speed: 432 wps 
0.803 perplexity: 301.630 speed: 431 wps 
0.903 perplexity: 282.417 speed: 434 wps 
Epoch: 1 Train Perplexity: 268.124 
Epoch: 1 Valid Perplexity: 180.210 
Epoch: 2 Learning rate: 1.000 
0.004 perplexity: 209.082 speed: 448 wps 
0.104 perplexity: 150.589 speed: 437 wps 
0.204 perplexity: 157.965 speed: 436 wps 
0.304 perplexity: 152.896 speed: 453 wps 
0.404 perplexity: 150.299 speed: 458 wps 
0.504 perplexity: 147.984 speed: 462 wps 
0.604 perplexity: 143.367 speed: 462 wps 
0.703 perplexity: 141.246 speed: 446 wps 
0.803 perplexity: 139.299 speed: 436 wps 
0.903 perplexity: 135.632 speed: 435 wps 
Epoch: 2 Train Perplexity: 133.576 
Epoch: 2 Valid Perplexity: 143.072 
. . . . . . . . . .  . . . .  . . . . . . . . . .  . . . . 
Epoch: 12 Learning rate: 0.008 
0.004 perplexity: 57.011 speed: 347 wps 
0.104 perplexity: 41.305 speed: 356 wps 
0.204 perplexity: 45.136 speed: 356 wps 
0.304 perplexity: 43.386 speed: 357 wps 
0.404 perplexity: 42.624 speed: 358 wps 
0.504 perplexity: 41.980 speed: 358 wps 
0.604 perplexity: 40.549 speed: 357 wps 
0.703 perplexity: 39.943 speed: 357 wps 
0.803 perplexity: 39.287 speed: 358 wps 
0.903 perplexity: 37.949 speed: 359 wps 
Epoch: 12 Train Perplexity: 37.125 
Epoch: 12 Valid Perplexity: 123.571 
Epoch: 13 Learning rate: 0.004 
0.004 perplexity: 56.576 speed: 365 wps 
0.104 perplexity: 40.989 speed: 358 wps 
0.204 perplexity: 44.809 speed: 358 wps 
0.304 perplexity: 43.082 speed: 356 wps 
0.404 perplexity: 42.332 speed: 356 wps 
0.504 perplexity: 41.694 speed: 356 wps 
0.604 perplexity: 40.275 speed: 357 wps 
0.703 perplexity: 39.673 speed: 356 wps 
0.803 perplexity: 39.021 speed: 356 wps 
0.903 perplexity: 37.690 speed: 356 wps 
Epoch: 13 Train Perplexity: 36.869 
Epoch: 13 Valid Perplexity: 123.358 
Test Perplexity: 117.171