Due: Monday, April 17 at 5pm.

You must work alone on this assignment.

You will be working with visitor logs from the White House. The attributes in this dataset are described here. Also, you can see a spreadsheet of the data here. Note there are datasets from 2009/2010, 2011, 2012, 2013, 2014, 2015, and 2016. We'll work with the data from 2009/2010.

Your task is to write MapReduce code to generate the following information:

1. A list of the guests who visited at least ten times (task1.py).
2. A list of the ten most frequently-visited staff members (task2.py). The dataset has some obvious anomalies that you will want to address when you answer this question.
3. A list of the guests who visited at least once in both 2009 and 2010 (task3.py).
4. A list of the people who were both guests and staff (in other words, someone whom is visited) in 2009 and 2010 (task4.py).

The terms "guest" and "staff" simply refer to the roles: someone who is a guest is visiting the White House, while a staff member is someone who works there and is being visited. Individuals such as the First Lady, whom you may not consider technically to be "staff," still qualify in this simplistic definition. In other words, these terms are being used to provide intuition, not to create a situation where you are having to make difficult determinations about who constitutes "staff members."

For full credit, you must write combiners for each of these tasks whenever it is possible to have a meaningful combiner.

The 2009/2010 dataset is small enough that you will be able to write python code to check your answers. When you can validate the results of a MapReduce calculation using a separate, non-MapReduce approach, this gives you a high degree of confidence that your MapReduce implementation is correct. Of course, it is not particularly compelling to use MapReduce for data sets that do not need it, other than for instructional purposes. But, even for large data sets, performing this kind of double-check on a manageably small subset of the data is wise.

Write your MapReduce versions in python using mrjob. You can do your development on your VirtualBox from CS 122.

To install mrjob on your VirtualBox VM:

sudo pip2 install mrjob
sudo pip3 install mrjob

To run a particular task, if the downloaded White House data is in data.csv:

python3 taskx.py --jobconf mapreduce.job.reduces=1 data.csv

for some value of x. Note that this really is supposed to say "reduces", not "reducers". You will receive a warning message about "No configs", which you can ignore.

There will be a post on Piazza with information about setting up git and chisubmit to submit this assignment. Place your files in a directory called pa1 in your git repository and perform the git push and chisubmit steps you did in 122. You should place your code in four python files with names task1.py, task2.py, etc. as above.