39 lines
945 B
Plaintext
39 lines
945 B
Plaintext
using ModKneserNey for 1-grams
|
|
modifying 1-gram counts for Kneser-Ney smoothing
|
|
Kneser-Ney smoothing 1-grams
|
|
n1 = 9816
|
|
n2 = 4085
|
|
n3 = 2199
|
|
n4 = 1578
|
|
D1 = 0.545758
|
|
D2 = 1.11864
|
|
D3+ = 1.43346
|
|
using ModKneserNey for 2-grams
|
|
modifying 2-gram counts for Kneser-Ney smoothing
|
|
Kneser-Ney smoothing 2-grams
|
|
n1 = 287180
|
|
n2 = 56704
|
|
n3 = 23987
|
|
n4 = 13383
|
|
D1 = 0.716896
|
|
D2 = 1.09021
|
|
D3+ = 1.4001
|
|
using ModKneserNey for 3-grams
|
|
Kneser-Ney smoothing 3-grams
|
|
n1 = 1065684
|
|
n2 = 128239
|
|
n3 = 46598
|
|
n4 = 23979
|
|
D1 = 0.806016
|
|
D2 = 1.12136
|
|
D3+ = 1.34092
|
|
discarded 1 1-gram probs predicting pseudo-events
|
|
warning: distributing 0.0130435 left-over probability mass over 6550 zeroton words
|
|
discarded 2 2-gram contexts containing pseudo-events
|
|
discarded 11 2-gram probs predicting pseudo-events
|
|
discarded 10919 3-gram contexts containing pseudo-events
|
|
discarded 9 3-gram probs predicting pseudo-events
|
|
writing 33110 1-grams
|
|
writing 425750 2-grams
|
|
writing 1334621 3-grams
|