Human Attention Maps for Text Classification

YELP-HAT Dataset

Motivated by human attention, computational attention mechanisms have been designed to help neural networks adjust their focus on specific parts of the input data. While attention mechanisms are claimed to achieve interpretability, little is known about the actual relationships between machine and human attention. We design and conduct a large-scale crowd-sourcing study to collect human attention maps that encode the parts of a text that humans focus on when doing text classification. Resulting YELP-HAT dataset, collected on the publicly available YELP Restaurant Reviews dataset, is available on this page. YELP-HAT Human Attention Maps are collected for the classification task of rating a review as positive or negative on Amazon Mechanical Turk. Participants are asked to complete two tasks: 1) Identify the sentiment of the review as positive, negative, or neither, and 2) Highlight the words that are indicative of the chosen sentiment. We collect attention maps for 5,000 reviews, and three different subjects annotated each review, resulting in a total of 15,000 human attention maps.

Do Humans and Neural Networks Focus on the Same Words?

Based on YELP-HAT Dataset, we perform a quantitative comparative analysis of machine attention maps created by deep learning models and human attention maps. Our analysis offers insights into the relationships between human versus machine attention maps along three dimensions: overlap in word selections, distribution over lexical categories, and context-dependency of sentiment polarity. Our findings open promising future research opportunities ranging from supervised attention to the design of human-centric attention-based explanations. Details of this study can be found in our paper, titled "Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words?" to appear at the Annual Conference of the Association for Computational Linguistics (ACL) 2020 (link below).

Examples from the YELP-HAT

Out in Twinsburg for work and wasn't expecting to find a well reviewed sushi restaurant but glad I did. It was quite busy for a Monday and the poor waitress was slammed but the sushi chef stepped in to help and was very friendly. The presentation and flavors were great.

Out in Twinsburg for work and wasn't expecting to find a well reviewed sushi restaurant but glad I did. It was quite busy for a Monday and the poor waitress was slammed but the sushi chef stepped in to help and was very friendly. The presentation and flavors were great.

Out in Twinsburg for work and wasn't expecting to find a well reviewed sushi restaurant but glad I did. It was quite busy for a Monday and the poor waitress was slammed but the sushi chef stepped in to help and was very friendly. The presentation and flavors were great.



Very slow. Never been in the drive at any other place longer then I do at Kneader's. I don't know what they are doing different but 20 min minimum sitting in a drive through for a sandwich is ridiculous. And it's not just this location. Every. Single. One. Is SLOW!!!

Very slow. Never been in the drive at any other place longer then I do at Kneader's. I don't know what they are doing different but 20 min minimum sitting in a drive through for a sandwich is ridiculous. And it's not just this location. Every. Single. One. Is SLOW!!!

Very slow. Never been in the drive at any other place longer then I do at Kneader's. I don't know what they are doing different but 20 min minimum sitting in a drive through for a sandwich is ridiculous. And it's not just this location. Every. Single. One. Is SLOW!!!



Food is good, but service terrible. They have me a to go box when I was eating in. Did not get my cup until after the fact. I had to talk to someone else for it place not clean, at least the food is good. Service kinda ruined my meal

Food is good, but service terrible. They have me a to go box when I was eating in. Did not get my cup until after the fact. I had to talk to someone else for it place not clean, at least the food is good. Service kinda ruined my meal

Food is good, but service terrible. They have me a to go box when I was eating in. Did not get my cup until after the fact. I had to talk to someone else for it place not clean, at least the food is good. Service kinda ruined my meal



Stopped by on a Sunday around 11am after a trip to Freedom Park and had a lovely experience here- such cool ambiance and the staff was friendly and helpful. Chicken salad was good. The homemade pita chips were ok...a little thick for me. Others in our group enjoyed their food.

Stopped by on a Sunday around 11am after a trip to Freedom Park and had a lovely experience here- such cool ambiance and the staff was friendly and helpful. Chicken salad was good. The homemade pita chips were ok...a little thick for me. Others in our group enjoyed their food.

Stopped by on a Sunday around 11am after a trip to Freedom Park and had a lovely experience here- such cool ambiance and the staff was friendly and helpful. Chicken salad was good. The homemade pita chips were ok...a little thick for me. Others in our group enjoyed their food.