# Boolean and Bayesian Scenarios with explanation

This semester we examined two common techniques for representing, organizing, retrieving, and filtering information:

the deductive Booleanlogic of classes and their combinations, and the inductive Bayesianlogic of probabilities conditioned on past events.

Don't use plagiarized sources. Get Your Custom Essay on
Boolean and Bayesian Scenarios with explanation
Just from \$13/Page

As Bernhard Rieder observed, each of these techniques “implies a particular way of doing things” but can “be applied to a wide array of domains.”

For example, Boolean logic can be applied to classify and filter scientific articles, works of choreography, consumer products, or potential romantic partners, ifone can come up with a way to represent these things as belonging to classes defined by necessary and su?cient properties.

Likewise, Bayesian logic can be applied to these same domains and many others, ifone can represent these things as having quantified properties suitable for training a statistical model. In either case, for any particular domain of application, there are “many decisions to be made concerning the units to take into account, the parameters to specify, the tweaks to apply, the outputs to produce, and so forth” (Rieder).

For this question, you will identify and describe two different applications of information classification and filtering.The first should be an application for which you believe Boolean logic to be the preferable technique, and the second should be an application for which you believe Bayesian logic to be the preferable technique.

For each of the two applicationsthat you identify, you should:

• Describe the application briefly but with su?cient detail to provide context for your argument in favor of the chosen technique. Who uses it, and why? How do they use it? What is the “stu? ” being classified and filtered? How are the outputs of classification and
• Explain how the technique you’ve chosen could be applied. If the application is the one for which you believe Boolean logic would be preferable, explain how the “stu? ” could be represented as classes defined by necessary and su?cient properties, and how someone using the application would use these classes to classify and filter. If the application is the one for which you believe Bayesian logic would be preferable, explain how the “stu? ” could be represented as things having quantified properties, and how these representations could be used to train a statistical model, and how the estimated
• Give two reasons why your chosen technique is preferable for this particular application. Again, do not just identify general strengths of the technique—instead give specific reasons why the technique might be preferable for this particular application.

filtering presented or used? The application may be purely hypothetical, or it may be based on a real application that you are aware of. See the next page for an example.

probabilities produced by the model could be used to classify and filter. Do not just give a

general explanation of how the technique works—instead focus on some of the specific decisions to be made for this particular application.

4.Finally, identify one potential problemwith applying the technique in this way. What limitations might the technique impose, or what negative consequences might result? One more time: do not just identify general limitations or drawbacks of the technique—instead describe a specific negative outcome that could result from using this technique for this particular application.

Exampleof a description of an application, like you might write for part 1 above (do not

DRAFT! © April 1, 2009 Cambridge University Press. Feedback welcome.
1
INFORMATION
RETRIEVAL
1
Boolean retrieval
The meaning of the term information retrieval can be very broad. Just getting
a credit card out of your wallet so that you can type in the card number
is a form of information retrieval. However, as an academic field of study,
information retrieval might be defined thus:
Information retrieval (IR) is finding material (usually documents) of
an unstructured nature (usually text) that satisfies an information need
from within large collections (usually stored on computers).
As defined in this way, information retrieval used to be an activity that only
a few people engaged in: reference librarians, paralegals, and similar professional searchers. Now the world has changed, and hundreds of millions
of people engage in information retrieval every day when they use a web
search engine or search their email.1 Information retrieval is fast becoming
the dominant form of information access, overtaking traditional databasestyle searching (the sort that is going on when a clerk says to you: “I’m sorry,
I can only look up your order if you can give me your Order ID”).
IR can also cover other kinds of data and information problems beyond
that specified in the core definition above. The term “unstructured data”
refers to data which does not have clear, semantically overt, easy-for-a-computer
structure. It is the opposite of structured data, the canonical example of
which is a relational database, of the sort companies usually use to maintain product inventories and personnel records. In reality, almost no data
are truly “unstructured”. This is definitely true of all text data if you count
the latent linguistic structure of human languages. But even accepting that
the intended notion of structure is overt structure, most text has structure,
such as headings and paragraphs and footnotes, which is commonly represented in documents by explicit markup (such as the coding underlying web
1. In modern parlance, the word “search” has tended to replace “(information) retrieval”; the
term “search” is quite ambiguous, but in context we use the two synonymously.
Online edition (c) 2009 Cambridge UP
2
1 Boolean retrieval
pages). IR is also used to facilitate “semistructured” search such as finding a
document where the title contains Java and the body contains threading.
The field of information retrieval also covers supporting users in browsing
or filtering document collections or further processing a set of retrieved documents. Given a set of documents, clustering is the task of coming up with a
good grouping of the documents based on their contents. It is similar to arranging books on a bookshelf according to their topic. Given a set of topics,
standing information needs, or other categories (such as suitability of texts
for different age groups), classification is the task of deciding which class(es),
if any, each of a set of documents belongs to. It is often approached by first
manually classifying some documents and then hoping to be able to classify
new documents automatically.
Information retrieval systems can also be distinguished by the scale at
which they operate, and it is useful to distinguish three prominent scales.
In web search, the system has to provide search over billions of documents
stored on millions of computers. Distinctive issues are needing to gather
documents for indexing, being able to build systems that work efficiently
at this enormous scale, and handling particular aspects of the web, such as
the exploitation of hypertext and not being fooled by site providers manipulating page content in an attempt to boost their search engine rankings,
given the commercial importance of the web. We focus on all these issues
in Chapters 19–21. At the other extreme is personal information retrieval. In
the last few years, consumer operating systems have integrated information
retrieval (such as Apple’s Mac OS X Spotlight or Windows Vista’s Instant
Search). Email programs usually not only provide search but also text classification: they at least provide a spam (junk mail) filter, and commonly also
provide either manual or automatic means for classifying mail so that it can
be placed directly into particular folders. Distinctive issues here include handling the broad range of document types on a typical personal computer,
and making the search system maintenance free and sufficiently lightweight
in terms of startup, processing, and disk space usage that it can run on one
machine without annoying its owner. In between is the space of enterprise,
institutional, and domain-specific search, where retrieval might be provided for
collections such as a corporation’s internal documents, a database of patents,
or research articles on biochemistry. In this case, the documents will typically be stored on centralized file systems and one or a handful of dedicated
machines will provide search over the collection. This book contains techniques of value over this whole spectrum, but our coverage of some aspects
of parallel and distributed search in web-scale search systems is comparatively light owing to the relatively small published literature on the details
of such systems. However, outside of a handful of web search companies, a
software developer is most likely to encounter the personal search and enterprise scenarios.
Online edition (c) 2009 Cambridge UP
1.1 An example information retrieval problem
3
In this chapter we begin with a very simple example of an information
retrieval problem, and introduce the idea of a term-document matrix (Section 1.1) and the central inverted index data structure (Section 1.2). We will
then examine the Boolean retrieval model and how Boolean queries are processed (Sections 1.3 and 1.4).
1.1
GREP
An example information retrieval problem
A fat book which many people own is Shakespeare’s Collected Works. Suppose you wanted to determine which plays of Shakespeare contain the words
Brutus AND Caesar AND NOT Calpurnia. One way to do that is to start at the
beginning and to read through all the text, noting for each play whether
it contains Brutus and Caesar and excluding it from consideration if it contains Calpurnia. The simplest form of document retrieval is for a computer
to do this sort of linear scan through documents. This process is commonly
referred to as grepping through text, after the Unix command grep, which
performs this process. Grepping through text can be a very effective process,
especially given the speed of modern computers, and often allows useful
possibilities for wildcard pattern matching through the use of regular expressions. With modern computers, for simple querying of modest collections
(the size of Shakespeare’s Collected Works is a bit under one million words
of text in total), you really need nothing more.
But for many purposes, you do need more:
1. To process large document collections quickly. The amount of online data
has grown at least as quickly as the speed of computers, and we would
now like to be able to search collections that total in the order of billions
to trillions of words.
2. To allow more flexible matching operations. For example, it is impractical
to perform the query Romans NEAR countrymen with grep, where NEAR
might be defined as “within 5 words” or “within the same sentence”.
3. To allow ranked retrieval: in many cases you want the best answer to an
information need among many documents that contain certain words.
INDEX
INCIDENCE MATRIX
TERM
The way to avoid linearly scanning the texts for each query is to index the
documents in advance. Let us stick with Shakespeare’s Collected Works,
and use it to introduce the basics of the Boolean retrieval model. Suppose
we record for each document – here a play of Shakespeare’s – whether it
contains each word out of all the words Shakespeare used (Shakespeare used
about 32,000 different words). The result is a binary term-document incidence
matrix, as in Figure 1.1. Terms are the indexed units (further discussed in
Section 2.2); they are usually words, and for the moment you can think of
Online edition (c) 2009 Cambridge UP
4
1 Boolean retrieval
Antony
Brutus
Caesar
Calpurnia
Cleopatra
mercy
worser

Antony
and
Cleopatra
1
1
1
0
1
1
1
Julius
Caesar
The
Tempest
Hamlet
Othello
Macbeth
1
1
1
1
0
0
0
0
0
0
0
0
1
1
0
1
1
0
0
1
1
0
0
1
0
0
1
1
1
0
1
0
0
1
0

◮ Figure 1.1 A term-document incidence matrix. Matrix element (t, d) is 1 if the
play in column d contains the word in row t, and is 0 otherwise.
them as words, but the information retrieval literature normally speaks of
terms because some of them, such as perhaps I-9 or Hong Kong are not usually
thought of as words. Now, depending on whether we look at the matrix rows
or columns, we can have a vector for each term, which shows the documents
it appears in, or a vector for each document, showing the terms that occur in
it.2
To answer the query Brutus AND Caesar AND NOT Calpurnia, we take the
vectors for Brutus, Caesar and Calpurnia, complement the last, and then do a
bitwise AND:
110100 AND 110111 AND 101111 = 100100
B OOLEAN RETRIEVAL
MODEL
DOCUMENT
COLLECTION
CORPUS
The answers for this query are thus Antony and Cleopatra and Hamlet (Figure 1.2).
The Boolean retrieval model is a model for information retrieval in which we
can pose any query which is in the form of a Boolean expression of terms,
that is, in which terms are combined with the operators AND, OR, and NOT.
The model views each document as just a set of words.
Let us now consider a more realistic scenario, simultaneously using the
opportunity to introduce some terminology and notation. Suppose we have
N = 1 million documents. By documents we mean whatever units we have
decided to build a retrieval system over. They might be individual memos
or chapters of a book (see Section 2.1.2 (page 20) for further discussion). We
will refer to the group of documents over which we perform retrieval as the
(document) collection. It is sometimes also referred to as a corpus (a body of
texts). Suppose each document is about 1000 words long (2–3 book pages). If
2. Formally, we take the transpose of the matrix to be able to get the terms as column vectors.
Online edition (c) 2009 Cambridge UP
5
1.1 An example information retrieval problem
Antony and Cleopatra, Act III, Scene ii
Agrippa [Aside to Domitius Enobarbus]: Why, Enobarbus,
When Antony found Julius Caesar dead,
He cried almost to roaring; and he wept
When at Philippi he found Brutus slain.
Hamlet, Act III, Scene ii
Lord Polonius:
I did enact Julius Caesar: I was killed i’ the
Capitol; Brutus killed me.
◮ Figure 1.2 Results from Shakespeare for the query Brutus
AND Caesar AND NOT
Calpurnia.
INFORMATION NEED
QUERY
RELEVANCE
EFFECTIVENESS
we assume an average of 6 bytes per word including spaces and punctuation,
then this is a document collection about 6 GB in size. Typically, there might
be about M = 500,000 distinct terms in these documents. There is nothing
special about the numbers we have chosen, and they might vary by an order
of magnitude or more, but they give us some idea of the dimensions of the
kinds of problems we need to handle. We will discuss and model these size
assumptions in Section 5.1 (page 86).
the most standard IR task. In it, a system aims to provide documents from
within the collection that are relevant to an arbitrary user information need,
communicated to the system by means of a one-off, user-initiated query. An
information need is the topic about which the user desires to know more, and
is differentiated from a query, which is what the user conveys to the computer in an attempt to communicate the information need. A document is
relevant if it is one that the user perceives as containing information of value
with respect to their personal information need. Our example above was
rather artificial in that the information need was defined in terms of particular words, whereas usually a user is interested in a topic like “pipeline
leaks” and would like to find relevant documents regardless of whether they
precisely use those words or express the concept with other words such as
pipeline rupture. To assess the effectiveness of an IR system (i.e., the quality of
its search results), a user will usually want to know two key statistics about
the system’s returned results for a query:
PRECISION
Precision: What fraction of the returned results are relevant to the information need?
RECALL
Recall: What fraction of the relevant documents in the collection were returned by the system?
Online edition (c) 2009 Cambridge UP
6
1 Boolean retrieval
INVERTED INDEX
DICTIONARY
VOCABULARY
LEXICON
POSTING
POSTINGS LIST
POSTINGS
1.2
Detailed discussion of relevance and evaluation measures including precision and recall is found in Chapter 8.
We now cannot build a term-document matrix in a naive way. A 500K ×
1M matrix has half-a-trillion 0’s and 1’s – too many to fit in a computer’s
memory. But the crucial observation is that the matrix is extremely sparse,
that is, it has few non-zero entries. Because each document is 1000 words
long, the matrix has no more than one billion 1’s, so a minimum of 99.8% of
the cells are zero. A much better representation is to record only the things
that do occur, that is, the 1 positions.
This idea is central to the first major concept in information retrieval, the
inverted index. The name is actually redundant: an index always maps back
from terms to the parts of a document where they occur. Nevertheless, inverted index, or sometimes inverted file, has become the standard term in information retrieval.3 The basic idea of an inverted index is shown in Figure 1.3.
We keep a dictionary of terms (sometimes also referred to as a vocabulary or
lexicon; in this book, we use dictionary for the data structure and vocabulary
for the set of terms). Then for each term, we have a list that records which
documents the term occurs in. Each item in the list – which records that a
term appeared in a document (and, later, often, the positions in the document) – is conventionally called a posting.4 The list is then called a postings
list (or inverted list), and all the postings lists taken together are referred to as
the postings. The dictionary in Figure 1.3 has been sorted alphabetically and
each postings list is sorted by document ID. We will see why this is useful in
Section 1.3, below, but later we will also consider alternatives to doing this
(Section 7.1.5).
A first take at building an inverted index
To gain the speed benefits of indexing at retrieval time, we have to build the
index in advance. The major steps in this are:
1. Collect the documents to be indexed:
Friends, Romans, countrymen. So let it be with Caesar . . .
2. Tokenize the text, turning each document into a list of tokens:
Friends Romans countrymen So . . .
3. Some information retrieval researchers prefer the term inverted file, but expressions like index construction and index compression are much more common than inverted file construction and
inverted file compression. For consistency, we use (inverted) index throughout this book.
4. In a (non-positional) inverted index, a posting is just a document ID, but it is inherently
associated with a term, via the postings list it is placed on; sometimes we will also talk of a
(term, docID) pair as a posting.
Online edition (c) 2009 Cambridge UP
7
1.2 A first take at building an inverted index
Brutus
−→
1
2
4
11
31
45
173
174
Caesar
−→
1
2
4
5
6
16
57
132
Calpurnia
−→
2
31
54
101

..
.
| {z }
Dictionary
|
{z
Postings
}
◮ Figure 1.3 The two parts of an inverted index. The dictionary is commonly kept
in memory, with pointers to each postings list, which is stored on disk.
3. Do linguistic preprocessing, producing a list of normalized tokens, which
are the indexing terms: friend roman countryman so . . .
4. Index the documents that each term occurs in by creating an inverted index, consisting of a dictionary and postings.
DOC ID
SORTING
DOCUMENT
FREQUENCY
We will define and discuss the earlier stages of processing, that is, steps 1–3,
in Section 2.2 (page 22). Until then you can think of tokens and normalized
tokens as also loosely equivalent to words. Here, we assume that the first
3 steps have already been done, and we examine building a basic inverted
index by sort-based indexing.
Within a document collection, we assume that each document has a unique
serial number, known as the document identifier (docID). During index construction, we can simply assign successive integers to each new document
when it is first encountered. The input to indexing is a list of normalized
tokens for each document, which we can equally think of as a list of pairs of
term and docID, as in Figure 1.4. The core indexing step is sorting this list
so that the terms are alphabetical, giving us the representation in the middle
column of Figure 1.4. Multiple occurrences of the same term from the same
document are then merged.5 Instances of the same term are then grouped,
and the result is split into a dictionary and postings, as shown in the right
column of Figure 1.4. Since a term generally occurs in a number of documents, this data organization already reduces the storage requirements of
the index. The dictionary also records some statistics, such as the number of
documents which contain each term (the document frequency, which is here
also the length of each postings list). This information is not vital for a basic Boolean search engine, but it allows us to improve the efficiency of the
5. Unix users can note that these steps are similar to use of the sort and then uniq commands.
Online edition (c) 2009 Cambridge UP
8
1 Boolean retrieval
Doc 1
I did enact Julius Caesar: I was killed
i’ the Capitol; Brutus killed me.
Doc 2
So let it be with Caesar. The noble Brutus
hath told you Caesar was ambitious:
term
docID
term
docID
I
1
ambitious
2
term doc. freq.
did
1
be
2
ambitious 1
enact
1
brutus
1
be 1
julius
1
brutus
2
brutus 2
caesar
1
capitol
1
I
1
caesar
1
capitol 1
was
1
caesar
2
caesar 2
killed
1
caesar
2
did 1
i’
1
did
1
enact 1
the
1
enact
1
hath 1
capitol
1
hath
1
brutus
1
I
1
I 1
killed
1
I
1
i’ 1
me
1
i’
1
=⇒
=⇒ it 1
so
2
it
2
julius 1
let
2
julius
1
killed 1
it
2
killed
1
be
2
killed
1
let 1
with
2
let
2
me 1
caesar
2
me
1
noble 1
the
2
noble
2
so 1
noble
2
so
2
the 2
brutus
2
the
1
told 1
hath
2
the
2
told
2
told
2
you 1
you
2
you
2
was 2
caesar
2
was
1
with 1
was
2
was
2
ambitious
2
with
2

postings lists
2
2
1 → 2
1
1 → 2
1
1
2
1
1
2
1
1
2
1
2
2
1 → 2
2
2
1 → 2
2
◮ Figure 1.4 Building an index by sorting and grouping. The sequence of terms
in each document, tagged by their documentID (left) is sorted alphabetically (middle). Instances of the same term are then grouped by word and then by documentID.
The terms and documentIDs are then separated out (right). The dictionary stores
the terms, and has a pointer to the postings list for each term. It commonly also
stores other summary information such as, here, the document frequency of each
term. We use this information for improving query time efficiency and, later, for
weighting in ranked retrieval models. Each postings list stores the list of documents
in which a term occurs, and may store other information such as the term frequency
(the frequency of each term in each document) or the position(s) of the term in each
document.
Online edition (c) 2009 Cambridge UP
1.2 A first take at building an inverted index
9
search engine at query time, and it is a statistic later used in many ranked retrieval models. The postings are secondarily sorted by docID. This provides
the basis for efficient query processing. This inverted index structure is essentially without rivals as the most efficient structure for supporting ad hoc
text search.
In the resulting index, we pay for storage of both the dictionary and the
postings lists. The latter are much larger, but the dictionary is commonly
kept in memory, while postings lists are normally kept on disk, so the size
of each is important, and in Chapter 5 we will examine how each can be
optimized for storage and access efficiency. What data structure should be
used for a postings list? A fixed length array would be wasteful as some
words occur in many documents, and others in very few. For an in-memory
postings list, two good alternatives are singly linked lists or variable length
arrays. Singly linked lists allow cheap insertion of documents into postings
lists (following updates, such as when recrawling the web for updated documents), and naturally extend to more advanced indexing strategies such as
skip lists (Section 2.3), which require additional pointers. Variable length arrays win in space requirements by avoiding the overhead for pointers and in
time requirements because their use of contiguous memory increases speed
on modern processors with memory caches. Extra pointers can in practice be
encoded into the lists as offsets. If updates are relatively infrequent, variable
length arrays will be more compact and faster to traverse. We can also use a
hybrid scheme with a linked list of fixed length arrays for each term. When
postings lists are stored on disk, they are stored (perhaps compressed) as a
contiguous run of postings without explicit pointers (as in Figure 1.3), so as
to minimize the size of the postings list and the number of disk seeks to read
a postings list into memory.
?
Exercise 1.1
[ ⋆]
Draw the inverted index that would be built for the following document collection.
(See Figure 1.3 for an example.)
Doc 1
Doc 2
Doc 3
Doc 4
new home sales top forecasts
home sales rise in july
increase in home sales in july
july new home sales rise
Exercise 1.2
Consider these documents:
Doc 1
Doc 2
Doc 3
Doc 4
breakthrough drug for schizophrenia
new schizophrenia drug
new approach for treatment of schizophrenia
new hopes for schizophrenia patients
a. Draw the term-document incidence matrix for this document collection.
Online edition (c) 2009 Cambridge UP
[ ⋆]
10
1 Boolean retrieval
Brutus
−→
1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia
−→
2 → 31 → 54 → 101
Intersection
=⇒
2 → 31
◮ Figure 1.5 Intersecting the postings lists for Brutus and Calpurnia from Figure 1.3.
b. Draw the inverted index representation for this collection, as in Figure 1.3 (page 7).
Exercise 1.3
[⋆]
For the document collection shown in Exercise 1.2, what are the returned results for
these queries:
a. schizophrenia AND drug
b. for AND NOT (drug OR approach)
1.3
SIMPLE CONJUNCTIVE
QUERIES
(1.1)
Processing Boolean queries
How do we process a query using an inverted index and the basic Boolean
retrieval model? Consider processing the simple conjunctive query:
Brutus AND Calpurnia
over the inverted index partially shown in Figure 1.3 (page 7). We:
1. Locate Brutus in the Dictionary
2. Retrieve its postings
3. Locate Calpurnia in the Dictionary
4. Retrieve its postings
5. Intersect the two postings lists, as shown in Figure 1.5.
POSTINGS LIST
INTERSECTION
POSTINGS MERGE
The intersection operation is the crucial one: we need to efficiently intersect
postings lists so as to be able to quickly find documents that contain both
terms. (This operation is sometimes referred to as merging postings lists:
this slightly counterintuitive name reflects using the term merge algorithm for
a general family of algorithms that combine multiple sorted lists by interleaved advancing of pointers through each; here we are merging the lists
with a logical AND operation.)
There is a simple and effective method of intersecting postings lists using
the merge algorithm (see Figure 1.6): we maintain pointers into both lists
Online edition (c) 2009 Cambridge UP
1.3 Processing Boolean queries
11
I NTERSECT ( p1, p2 )
2 while p1 6= NIL and p2 6= NIL
3 do if docID ( p1) = docID ( p2)
4
then A DD ( answer, docID ( p1 ))
5
p1 ← next( p1 )
6
p2 ← next( p2 )
7

Pages (275 words)
Standard price: \$0.00
Client Reviews
4.9
Sitejabber
4.6
Trustpilot
4.8
Our Guarantees
100% Confidentiality
Information about customers is confidential and never disclosed to third parties.
Original Writing
We complete all papers from scratch. You can get a plagiarism report.
Timely Delivery
No missed deadlines – 97% of assignments are completed in time.
Money Back