public class WikipediaDocumentCollection extends AbstractDocumentCollection implements Serializable
DocumentCollection corresponding to
a given set of files in the Yahoo! Wikipedia format.
This class provides a main method with a flexible syntax that serialises
into a document collection a list of (possibly gzip'd) files given on the command line or
piped into standard input. The files are to be taken from the
semantically
annotated snapshot of the english wikipedia distributed by Yahoo!.
The position of each record is stored using an EliasFanoMonotoneLongBigList per file, which gives us
random access with very little overhead.
Each column of the collection is indexed in parallel, and is accessible using its label as field name. For instance, a query like
Washington ^ WSJ:(B\-E\:PERSON | B\-I\:PERSON)will search for “Washington”, but only if the term has been annotated as a person name (note the escaping, which is necessary if you use the standard parser). See the
it.unimi.di.mg4j.search package for more info about the operators available.
See the collection page for more information about the tagging process.
| Modifier and Type | Class and Description |
|---|---|
static class |
WikipediaDocumentCollection.WhitespaceWordReader |
AbstractDocumentCollection.PropertyKeysDEFAULT_EXTENSION| Modifier | Constructor and Description |
|---|---|
|
WikipediaDocumentCollection(String[] file,
DocumentFactory factory,
boolean phrase)
Builds a document collection corresponding to a given set of Wikipedia files specified as an array.
|
|
WikipediaDocumentCollection(String[] file,
DocumentFactory factory,
boolean phrase,
boolean gzipped)
Builds a document collection corresponding to a given set of (possibly gzip'd) Wikipedia files specified as an array.
|
protected |
WikipediaDocumentCollection(String[] file,
DocumentFactory factory,
ObjectArrayList<EliasFanoMonotoneLongBigList> pointers,
int size,
int[] firstDocument,
boolean phrase,
boolean gzipped) |
| Modifier and Type | Method and Description |
|---|---|
WikipediaDocumentCollection |
copy() |
Document |
document(int index)
Returns the document given its index.
|
DocumentFactory |
factory()
Returns the factory used by this sequence.
|
DocumentIterator |
iterator()
Returns an iterator over the sequence of documents.
|
static void |
main(String[] arg) |
Reference2ObjectMap<Enum<?>,Object> |
metadata(int index)
Returns the metadata map for a document.
|
int |
size()
Returns the number of documents in this collection.
|
InputStream |
stream(int index)
Returns an input stream for the raw content of a document.
|
ensureDocumentIndex, printAllDocuments, toStringclose, filename, finalize, loadclone, equals, getClass, hashCode, notify, notifyAll, wait, wait, waitclose, filenamepublic WikipediaDocumentCollection(String[] file, DocumentFactory factory, boolean phrase) throws IOException
Beware. This class is not guaranteed to work if files are deleted or modified after creation!
file - an array containing the files that will be contained in the collection.factory - the factory that will be used to create documents.phrase - whether phrases should be indexed instead of documents.IOExceptionpublic WikipediaDocumentCollection(String[] file, DocumentFactory factory, boolean phrase, boolean gzipped) throws IOException
Beware. This class is not guaranteed to work if files are deleted or modified after creation!
file - an array containing the files that will be contained in the collection.factory - the factory that will be used to create documents.phrase - whether phrases should be indexed instead of documents.gzipped - the files in file are gzip'd.IOExceptionprotected WikipediaDocumentCollection(String[] file, DocumentFactory factory, ObjectArrayList<EliasFanoMonotoneLongBigList> pointers, int size, int[] firstDocument, boolean phrase, boolean gzipped)
public DocumentFactory factory()
DocumentSequenceEvery document sequence is based on a document factory that transforms raw bytes into a sequence of characters. The factory contains useful information such as the number of fields.
factory in interface DocumentSequencepublic int size()
DocumentCollectionsize in interface DocumentCollectionpublic Reference2ObjectMap<Enum<?>,Object> metadata(int index) throws IOException
DocumentCollectionmetadata in interface DocumentCollectionindex - an index between 0 (inclusive) and DocumentCollection.size() (exclusive).IOExceptionpublic Document document(int index) throws IOException
DocumentCollectiondocument in interface DocumentCollectionindex - an index between 0 (inclusive) and DocumentCollection.size() (exclusive).index-th document.IOExceptionpublic InputStream stream(int index) throws IOException
DocumentCollectionstream in interface DocumentCollectionindex - an index between 0 (inclusive) and DocumentCollection.size() (exclusive).IOExceptionpublic DocumentIterator iterator() throws IOException
DocumentSequenceWarning: this method can be safely called just one time. For instance, implementations based on standard input will usually throw an exception if this method is called twice.
Implementations may decide to override this restriction
(in particular, if they implement DocumentCollection). Usually,
however, it is not possible to obtain two iterators at the
same time on a collection.
iterator in interface DocumentSequenceiterator in class AbstractDocumentCollectionIOExceptionDocumentCollectionpublic WikipediaDocumentCollection copy()
copy in interface DocumentCollectioncopy in interface FlyweightPrototype<DocumentCollection>public static void main(String[] arg) throws IOException, com.martiansoftware.jsap.JSAPException, InstantiationException, IllegalAccessException, InvocationTargetException, NoSuchMethodException
IOExceptioncom.martiansoftware.jsap.JSAPExceptionInstantiationExceptionIllegalAccessExceptionInvocationTargetExceptionNoSuchMethodException