[Please read carefully before attempting the task!]
I. General Instructions:
- Attempt HITs if you are a native speaker of English or a fluent speaker who can comfortably comprehend a review written in informal English.
- We are happy to receive feedback and improve this job accordingly. Feel free to send your comments to: s.angelidis [at] ed [dot] ac [dot] uk or use figure-eight's messaging.
- Your responses are confidential. Any publications based on these will not include your specific responses, but rather aggregate information from many individuals. We will not ask any information that can be used to identify who you are.
- Important: This task is, to some extent, subjective, so we do not use test questions to evaluate the quality of answers. However, we monitor answers to identify workers who don't follow our basic instructions or give random answers. Such workers will be rejected and banned from this and future tasks.
II. The Goal: Summarizing Customer Opinions about Hotels
This task is a first step towards generating summaries of the opinions expressed in Hotel reviews. Before creating actual opinion summaries, we need to identify the review sentences that contain informative and well-written opinions. For this task, you will not be asked to write your own summary -- you will simply select well-formed review sentences that you would want to see in an opinion summary.
III. The Task: Review Sentence Selection
In each page, you will see 5 reviews for a particular hotel. For every review, you should select the sentences that are well-formed and that you believe capture the most important and useful comments of the reviewer. You should judge the contents of each review independently -- don't worry if similar, identical or contradictory opinions are selected across reviews.
Even selecting none or all sentences in a single review is acceptable, if you think it's the right choice. We will not enforce limits on the number of sentences you select, but we expect you to select, on average, 20%-40% of the available sentences. To help you keep track of how many sentences you've selected per page, we have implemented a "selection tracker" bar. This shows you the percentage of sentences you've selected in the current page, and suggests whether you should select more or less, as shown in the screenshots below. You are not expected to always be in the green zone, but you should try to stay within these limits in most cases. Avoid the red zone as much as you can.
Judging whether a particular sentence is worth selecting is, to some extent, subjective and you should select the segments that you would want to see in a summary of opinions for the particular hotel. However, some basic rules should be taken into account:
- Do select sentences that express useful opinions of the reviewer or give useful details which support their opinions.
Do not select sentences that contain less important elaborations or further details about the reviewer's experience.
For example, in the following two review excerpts, only the sentences marked with a [✔] should be selected:[✔] The rooms were beautiful and massive! [useful opinion] [✔] Also, the most comfy beds I've ever slept on. [useful opinion] [ ] It was so hard to get out of bed every morning. [elaboration] [ ] I slept like a baby for a whole week :) [elaboration] [✔] The location is actually excellent. [useful opinion] [✔] Within walking distance of the Magnificent Mile. [useful details] [✔] Good restaurants and high end shops all around. [useful details] [ ] I never got bored of walking in that neighbourhood. [elaboration] - General comments about the experience of the reviewer should often be avoided, as they provide no useful information about why they liked/disliked the hotel. The same applies to comments which are not about the hotel itself. The following review example illustrates such cases:
[ ] We were looking for a family-friendly hotel. [not about the hotel] [ ] It feels harder and harder to find one these days. [not about the hotel] [ ] This one was just perfect!! [general comment] [✔] Beautiful and spacious rooms with a view. [useful opinion] [✔] And an excellent pool for the kids! [useful opinion] [ ] I'd definitely recommend this place to anyone! [general comment] - Avoid selecting poorly-written sentences (e.g., misspelled, or with wrong punctuation). Also avoid very long sentences that combine useful opinions with less interesting information, like the following.
[ ] The staff were very friendly and even though we booked through expedia to get a great deal on the hotel we were treated like a somebody and even received a room upgrade to a lake view. [mix of useful with other] [ ] First....location!close to Pike Market....a few blocks...!shopping ALLOVER. [poorly written]
IV. Example
Finally, here are 3 examples of sentence selection for the same review: (a) one with the appropriate number of selections; (b) one with more selections than necessary; and (c) one with fewer selections than necessary.
(a) Good selection:
(b) Over-selecting:
(c) Under-selecting: