Phase 3: Opinion Summary Collection (Aspect)

[Please read carefully before attempting the task!]

I. General Instructions:

• Attempt HITs if you are a native speaker of English.
• We are happy to receive feedback and improve this job accordingly. Feel free to send your comments to: s.angelidis [at] ed [dot] ac [dot] uk or use figure-eight's messaging.
• Your responses are confidential. Any publications based on these will not include your specific responses, but rather aggregate information from many individuals. We will not ask any information that can be used to identify who you are.
• Important: This task is, to some extent, subjective, so we do not use test questions to evaluate the quality of answers. However, we monitor answers to identify workers who don't follow our basic instructions or give random answers. Such workers will be rejected and banned from this and future tasks.
  
  

II. The Goal: Summarizing Customer Opinions about Hotels

The goal of this job is to generate textual summaries of the opinions expressed in a set of customer review sentences about a particular hotel. The review sentences presented to you have been previously identified as representative and useful. 

At each page, you will only see sentences which discuss a specific aspect of the customers' experience, e.g., the quality of the rooms, or the cleanliness of the hotel, or the location of the hotel, etc. This means that you will be creating aspect-specific summaries of the opinions expressed by customers.

III. The Task

More specifically, your task is to:

• Read the review sentences that have been pre-selected for you. These sentences will discuss only one aspect of the customers' experience.
• Important: At times you will see irrelevant opinions which do not relate to the specified aspect. You must ignore these and not include them in your summary.
• Identify the opinions expressed in these sentences, but ignore small details and irrelevant opinions.
• Summarize those opinions in a few simple, grammatical and coherent sentences, without exceeding the limit of 75 word (it is OK even if you use much fewer than 75 words).

A few important notes regarding your summaries:

• Note #1: You should try to cover all opinions that appear in the sentences which relate to the specified aspect. If both positive and negative opinions appear for the same facet of the customers' experience, you should try to mention both, for example:
  "The beds were comfortable, but not very clean."
   
• Note #2: If opinions in the pre-selected sentences directly contradict each other (e.g., "the beds were comfortable" vs. "the beds were very uncomfortable"), you should either:
  (a) write the most popular one, if it is expressed multiple times; or
  (b) use your judgement to form an accurate sentence, e.g.: "The beds were just ok in terms of comfort"
  
  
• Note #3: Your sentences should not be written in first person.
  For example, you should write:
  "The beds were really comfortable."
  but you should not write:
  "I found the beds really comfortable."
  
  
• Note #4: Your summaries should not include references to the reviewers, like:
  "Many reviewers found the beds comfortable" or "One person didn't like the bed"
  
  
• Note #5: If you don't see any sentences to summarize or none of the sentences discuss the specified aspect, just write:
  "There's no relevant sentences present on this one for me to summarize" 
  ... or something along these lines.

IV. The Interface 

For this task, we have prepared an interface that, hopefully, helps you generate a summary as quickly as possible. We have attempted to make it easy for you to:

• Identify the main keywords discussed in the pre-selected sentences, and group the sentences based on these.
• Discard irrelevant sentences.
• Keep track of the sentences that you have used so far to generate the summary.

Below is a sequence of screenshots highlighting the main interface features. An instruction video follows right after. Please spend a few minutes going over these to familiarize yourself with the task :)

1) Initially, the interface will look like below:

2) You can double-click on a keyword and all sentences that contain it will be grouped together at the top of the list:

3) The same can be achieved by double-clicking any word that gets highlighted when you hover the pointer over it:

4) Using these tools, you can start writing your summary. Every time you summarize a few sentences, you may use the "Already included?" check-boxed to strike them out, thus keeping track of what you have written about so far. Also, the "Is irrelevant?" check-boxes can help discard sentences that don't discuss the specified aspect. You DON'T have to use these check-boxes, they are only provided for your convenience.

5) At the end, all relevant sentences must be summarized, and all irrelevant ones must be ignored. Remember that you don't need to use all 75 words -- this is just an upper limit!

IV. Video Demonstration

V. Some Examples

There is no perfect way to write a summary of opinions, as different people might value opinions differently. This is expected and it is the reason we are collecting multiple summaries per example to get a better grasp over how people might digest customer reviews.

We understand, however, that some guidance might make your life easier. The following examples should act as suggestions -- not as a golden rule about how you should treat the task.

Note: Each example shows how you may summarize a few review sentences that discuss a specified aspect. They are not full-blown summarization examples, like the ones you might see in the task. Your summaries will often be longer than the ones provided here (because they summarize more review sentences).

Example #1: Rooms


Possible summary A:
"The rooms are basic in style, but comfortable. They contain many electrical appliances, including safe, coffee maker and hair dryer. There is air conditioning, but it is noisy."

Possible summary B:
The rooms were recently decorated with basic style, similar to a motel. They were fully-equipped with appliances and an air-conditioner, which was noisy. There was plenty of room.

Example #2: Location


Possible summary A:
"The location was good with many restaurants and shops nearby, including Super Target and Olive Garden."

Possible summary B:
"Walking distance to many restaurants and shops, like Super Target, Olive Garden and Publix. Easy access to major routes."

Example #3: Service


Possible summary:
"The staff were very helpful and friendly."

Example #4: Cleanliness


Possible summary:
"The rooms and pool were clean. Clean towels and beds made everyday."

Example #5: Food


Possible summary A:
"There was free continental breakfast, which was OK. It was sometimes hard to get a seat."

Possible summary B:
"The simple breakfast included cereal toast, coffee and juice. It was good enough and free. Sometimes it was too crowded."