Named-entity recognition (NER) tries to detect names of persons, organizations, locations, and other names in texts. Some NER systems are almost as good as humans, but it is not an easy task. Named entities usually start with upper case, such as Ivan. We should, therefore, not change the case of words when applying NER.
NLTK has support for the Stanford NER API. This is a Java API, so you need to have Java on your system. I tested the code with Java 1.8.0_65. The code in this recipe downloads the most recent Stanford NER archive (stanford-ner-2015-04-20.zip/3.5.2
) as of October 2015. If you want another version, take a look at http://nlp.stanford.edu/software/CRF-NER.shtml (retrieved October 2015).
Install NLTK by following the instructions in the Introduction section. You may also need to install Java.
The script is in the named_entity.py
file in this book's code bundle:
from nltk.tag.stanford import NERTagger import dautil as dl import os from zipfile import ZipFile from nltk.corpus import brown
def download_ner(): url = 'http://nlp.stanford.edu/software/stanford-ner-2015-04-20.zip' dir = os.path.join(dl.data.get_data_dir(), 'ner') if not os.path.exists(dir): os.mkdir(dir) fname = 'stanford-ner-2015-04-20.zip' out = os.path.join(dir, fname) if not dl.conf.file_exists(out): dl.data.download(url, out) with ZipFile(out) as nerzip: nerzip.extractall(path=dir) return os.path.join(dir, fname.replace('.zip', ''))
dir = download_ner() st = NERTagger(os.path.join(dir, 'classifiers', 'english.all.3class.distsim.crf.ser.gz'), os.path.join(dir, 'stanford-ner.jar')) fid = brown.fileids(categories='news')[0] printer = dl.log_api.Printer(nelems=9) tagged = [pair for pair in dl.collect.flatten(st.tag(brown.words(fid))) if pair[1] != 'O'] printer.print(tagged)
Refer to the following screenshot for the end result:
We created a NerTagger
object by specifying a pre-trained classifier and the NER JAR (Java archive). The classifier tagged words in our corpus as organization, location, person, or other. The classification is case sensitive, which means that if you lowercase all the words, you will get different results.