On December 20th, 2011, three researchers from three different universities took off on a road trip that would take them through some of the areas that suffered the most devastation caused by the Great East Japan Earthquake of March 11, 2011.

I first met Yugo Shobugawa at Harvard University back in May 2011.  He came to watch my presentation on “Social Media and GIS: Mapping Revolutions, History, and Catastrophe” where I spoke about the role that social media played in the days following the earthquake.  Yugo is a physician, an expert in clinical epidemiology, and is currently an assistant professor at Niigata University.  But he is also quite the GIS practitioner, fluent in using spatial analysis to provide amazing insights on how space is directly correlated with the outbreak, diffusion and control of various epidemics.  We bonded immediately over mutual excitements over things like temporal mapping, spatial regressions, and why our 4 year old children never listen to us.  But I think the real connection happened when we both acknowledged our feelings of despair and helplessness of being Japanese, and yet not being in Japan during the earthquake.  For many Japanese nationals living abroad, experiencing the earthquake from a distance was excruciating.  It is this sense of despair that drove many of us to pour our hearts and souls into the recovery effort, in any small way that we could, remotely.

And so it was that when I finally had the opportunity to visit family and friends in Japan in December, 2012, I also decided that I had to see the effects of the disaster with my own eyes.  I had been in a similar predicament in the past, having toured the town of Banda Aceh, Indonesia, shortly after the Indian Ocean Tsunami in 2004.  This trip left a huge imprint, witnessing an entire community in ruins, standing on ground zero of it all, rendered speechless and heartbroken for people I had no connections to.  With much trepidation, I understood that going to Sendai would be, in many ways, different.  Different in a sense that these were my people that had suffered, that I had a deep cultural and spiritual bond with the land that was affected.

So on the morning of December 20th, Yugo and I met up at the Shinkansen platform in Tokyo, and rode together towards Sendai. It was only the fourth time we had met, having first met at Harvard, then at UCLA, another time in San Diego, and now on a bullet train) , but it felt like we had been friends since kindergarden.  It was on this bullet train ride that Yugo told me about the Bishamon Project that he was working on.  The Bishamon team was working with local governments around the Fukushima Nuclear Plant, monitoring radiation levels in surrounding communities, focusing on commuting routes around schools.  He told me that the data they were collecting revealed what many feared, that radiation levels were still very high, at exposure levels that were dangerous to live in for prolonged periods of time.

As we rode along, I realized that all of this was interconnected in a strange way: our journey towards Sendai and Miyagi being one part of the whole, with the ongoing situation in the Fukushima Nuclear plant being the other.  Upon arrival in Sendai, we were met by a pleasant, effusive and thankfully talkative Dr. Mimura, who was currently working on his PhD studies in Tohoku University.  He had volunteered an entire day to take both of us around strategic spots in Sendai and Miyagi, all along providing excellent insight and stories that we would otherwise have no access to.  While there were many, many places he took us to (too many to recount here), I would like to focus on just a few highlights of the day.

Ishinomaki City

Our first stop was Ishinomaki City.  Driving around the city, we were initially struck by how life seemed to be going on as normal.  A vibrant community, densely populated neighborhoods, and the usual hussle bustle you expect from any mid-sized city.  But it soon dawned upon me that some things were far from normal.  The traffic lights not working, the visible damage in many buildings, the newly paved sidewalks, but most of all, it was the sign of boarded up businesses that were apparent in many areas of the city.  But it was not until we reached the Medical Center that the true impact of the disaster was felt.

Ishinomaki City Emergency Room 石巻市夜間急患センター

Kadowaki Elementary School

Okawa Elementary School, nine months after the Tsunami

Words cannot begin to describe the sadness that envelopes the story about Okawa Elementary School. Of the 108 children in attendance, 74 were lost due to the Tsunami that swept the school and much of Ishinomaki City on March 11, 2011. I had the opportunity to visit the location of this tragedy on December 20, some nine months later. It was a dreary, overcast day, with snowflakes falling from the sky in a slow swirling motion. It is said that the day of the earthquake had a similar weather, with temperatures hovering around freezing levels. As I took my camera out, I could hardly keep my hands on the shutter, as the cold wind cut deep into my exposed fingers. The drizzling snow added to the ambiance, as if to remind us of the symbology of the place where we stood. The entire school was surrounded by empty fields that showed remnants of the disaster. Squashed cars swept and left abandoned in the middle of rice paddies, mounds and mounds of cleaned up debris, an occasionally empty house that miraculously survived the waves, bulldozers continuing the cleanup effort, smoke rising from place to place signaling the burning of collected trash. It is clear that Ishinomaki City is but a shell of its former self, the devastation so vast that parts of the city may never fully recover. The school itself was wedged between the bank of a river and a small hill on its backside.

According to my guide, the earthquake manual for the school dictated that the children evacuate the building, and congregate in the open area outside the building.  This is the area that is sandwiched by the river and the hill.  The school teachers kept the children in this playground, instructing them to await their parents who were on their way to pick them up.  This is when the first warnings of the tsunami came.  They headed towards higher ground, but were too late, swept by the incoming waves.  The few who survived did so by reaching higher ground, only to have to endure the freezing temperatures for hours before being rescued.

The hill behind Okawa Elementary School

As I looked at the area that my guide referred to as the initial evacuation zone, I pointed at the hills located just 30 meters behind, and asked, “so, if they had simply run to those hills and gone to higher ground, would they have survived?”. I already knew the answer before he gave his one word response, “yes”.

What is left standing in Okawa Elementary School is the shell of the main campus, a brick structure whose main circular structure has remained intact. In front of the structure is a memorial, fully adorned with fresh flowers, fruit and drinks. In front of the memorial are two poster boards.

Two poster boards in front of the memorial

The first one reads:

The two of you had always wanted to go to the Sendai Illumination Pageant. I am so sorry, I guess I will never be able to take you after all. Every year you had said “let’s go!”. If I had known this was going to happen, I should have made the time and taken you there. But isn’t the illumination from the christmas tree the volunteers put together beautiful? It is called “Okawa Elementary School’s Illumination Pageant”. Isn’t it wonderful. Do enjoy it with your friends in Heaven. Your mother.

The first poster board

This is when I noticed the solar panels located next to the memorial.

Solar panels next to the memorial

These panels were powering the illumination for the christmas tree that stood in the exposed wall behind it:

The christmas tree

The second poster board, apparently a note written by the students, reads:

Please look after us
Fathers
Mothers
Little by little, little by little,
We will move forward

This post is password protected. To view it please enter your password below:

Password:

Source: University of Michigan

Red states reflect higher vote counts for John McCain, and blue states reflect higher vote counts for Barack Obama.  Now, when you “cartogram-ize” this map, you get the following result:

These cartograms have been around for quite a while, and are often used in GIS presentations to emphasize the point on how easy it is to create a false visual impression when authoring thematic maps on geographies that have a high contrast between the large and small polygons. This is especially true in US based maps, where coastal areas with high population counts are represented by tiny polygons.

I have often wondered how these cartograms are created, and was told by a graduate student at UCLA that there is now a java tool called ScapeToad that generates cartograms. Moreover, ScapeToad provides a free downloadable wizard-based tool that converts any shapefile into a cartogram!  It is insanely cool.  Here is a tutorial on how to use ScapeToad.

Step 1: Download ScapeToad

Go to http://scapetoad.choros.ch/download.php to download ScapeToad, and run the executable file.

Step 2: Add a layer

Run ScapeGoat, and click on the “Add layer” button, and choose a shapefile to upload. For this example, I chose a USA County shapefile that comes with basic 2000 census data.

Step 3: Create the cartogram

Now click on the “Create cartogram” button, and go through the wizard.  For this example, I chose to create a cartogram based on the attribute for “Hispanic”.

Step 4: Export as Shapefile

Now that you have created a cartogram, you are ready to export it to something usable.  ScapeToad allows you to export to SVG (which you can import into Illustrator), or as a shapefile (which you can import into ArcGIS).  Click on “Export to Shape”, and save the cartogram as a shapefile.  Open ArcGIS and load the shapefile.

When the shapefile is loaded into ArcGIS, it will be drawn in one random color, as this is the default behavior for loading any shapefile in ArcGIS.  Right click on the cartogram layer, and modify the symbology.  I chose to symbolize by:

• Quantities -> Graduated colors
• Value: Hispanic
• Classification: Natural Breaks
• Classes: 3

Now the cartogram is color coded based on the variable it was cartogramed with: Hispanic.  As a final step, I chose to label the counties that have high values:

US counties with high hispanic population counts (Census 2000)

UCLA Volunteer Day's live site

Every year for the past 3 years, UCLA has dedicated a day for community service.  On Volunteer Day, the incoming freshman class embarks to various destinations around Los Angeles to clean, paint, beautify, mentor and engage with the community.

As a technologist in charge of the Volunteer Center website, this day provides an incredible opportunity to utilize social media as a forum to capture the many stories and moments that occur throughout the day.  What other chance does one get to have control over a mass exodus of more than 7000 people all over a city?  How can we build a platform that allows us to capture the stories and deliver them in real time?  How might the volunteers on the ground most effectively submit their stories to a centralized public interface?  The answer was to build an awareness around what we dubbed “The Mobile Campaign“.

The result was a map-based interface that “evolved” throughout the day.  Starting out as an empty map at 7:00am, it gradually populated itself as more and more pictures and videos started flowing in, coming directly from the people on the ground, the volunteers themselves!  While you may not be able to experience the day through its “live” interface, feel free to see the hundreds of photos and video’s that came in from almost 30 different locations:

http://volunteer.ucla.edu/live

The technology behind the site was using the following:

• Google Maps API v3 for the mapping
• Flickrs API for the photo and video upload and retrieval
• and lot’s and lot’s of jQuery

Click here to view the embedded video.

(view full screen)

In concordance with previous analyses, the most noticeable observations come in the “fear” emotion on April 7th.  Looking at the “Earthquake magnitude” chart, one can see that the second largest aftershock occurs on that day, bringing meaning to the notion that “fear” was a predominant emotional reaction to an already stressed nation at the time.  One can also depict that while the April 7th 7.1 magnitude earthquake was the largest aftershock since the big one on March 11th, that the country was consistently rocked throughout, averaging more than 10 earthquakes a day.  However, as the earthquake chart reveals, the number of quakes had tailored off considerably over time, perhaps causing it to expose even more shock value to the “big” 7.1 quake, at a time when the people were starting to feel a level of normality.

This infographic was created through a painstaking process that utilized almost 10 different applications to generate the final result. The main application used to create the word cluster graphic was Gephi, an open source platform that lets you visualize complex networked data elements in a visually compelling and interactive environment.  However, coming up with this particular end result was complicated by various factors, one of which was the complexity that arose from using Japanese characters in its analysis.

The Workflow

Step 1

The first step in this Japan Twitter project was to actually collect and archive the twitter data coming out of Japan after the earthquake.  For this, a cron job was written as a PhP script by David Shepard, a member of the UCLA Digital Humanities Collaborative. The script used the Twitter search API to find and filter tweets based on relevant hashtags, and dumping them into our own MySQL database.  The cron job ran every 3 minutes for 30 days, collecting over 650,000 tweets during this time period.

Once the Twitter data was safely in our MySQL database, I queried out and generated 30 separate text files, one for each day following the earthquake.  Each “day” file consisted of just the tweet text from the thousands of tweets that belonged to that day (on average there were about 20,000 tweets per day).

Here, you can see the number of tweets collected on an hourly basis:

Step 2

In order to capture the range of emotions through the different phases of recovery following the disaster, I followed a methodology employed by Eiji Aramaki from Tokyo University, who took the words from an Emotion Dictionary to extract emotion patterns in a set of text files.  Dr. Aramaki provided me with about 2000 of the most commonly used “emotion” words in the Japanese language, sub-divided into 10 different categories. A separate CSV file for each emotion was generated.

I then used WordSmith, an application that allows you to extract word patterns, to find concurrences of every emotion word against each “day” file.  Through WordSmith’s concordance tool, I was able to run a batch process that matched each of my 10 “emotion” files against each of my 30 “day” files.

Here is a screenshot of WordSmith’s concordance function:

Step 3

The data generated from WordSmith was exported as a series of spreadsheets. These spreadsheets were combined, merged, analyzed, and recalculated to produce a single matrix of emotion words by day. While I was able to do most of the work in Excel, because of varying language character problems, I was forced use Google Spreadsheets, mostly to generate the CSV file format that Gephi requires as an input source file (Excel lost the Japanese text on csv export, while Google did not).

In order to create an emotion “measure” for each day, the spreadsheet generated columns that counted the number of times each keyword was found in each of the 30 days. For example, for word 悲しみ (sadness) was found 0.5 times for every 10,000 tweets on March 11th, 3.1 times on March 12th, 325 times on March 13th, and so on.

Step 4

The heart of the word cluster analysis was conducted in Gephi.  Gephi requires you to define your data in two basic elements:  Nodes and Edges.  For this analysis, I chose to define these as follows:

Nodes:  Every emotion word, and every day was used and defined as a Gephi node

Edges:  Every connection between a “word” and a “day” was defined as an edge, and weighted by how many times that word was found for every 10,000 tweets, for each day.

Here is a screen shot of Gephi’s data view:

Once the data elements were defined, Gephi is ready to visualize (ie, the fun part!).  Gephi comes with many layout templates that you can choose from.  Each layout has its own built in algorithms that take the nodes and edges from your database to generate a network diagram.  I chose to use a layout called “Parallel Force Atlas” (it sure sounds good).  You can choose to size and/or color each node by different data attributes, and do the same for the edges, which serve as the connectors between the nodes.  You then press a button, configure a few parameters (such as “gravity”), and voila! you are introduced to a beautiful infographic.

Step 5

What I then thought would be an easy step to export the graphic and create a web viewer (for panning and zooming the huge image) turned out to be a much bigger task than I anticipated. First of all, the Gephi exporters failed to export the Japanese characters… with one exception: SVG format. For some reason, SVG was the only export format that allowed the Japanese characters to survive. Since I wanted to provide a web interface that allows for zooming and panning the graphic, I ended up choosing one that uses the OpenLayers javascript API, which is predominantly used for geo-spatial data visualizations, but also allows you to use on images.  In order to get the image ready for OpenLayers, I used MapTiler, an application that generates the different image “tiles” that are needed for the different zoom levels.  You can see a full screen version of the final infographic here.

One key feature of social media is that it provides a snapshot of a moment’s mood, reflected by the content of what people are tweeting about in real time.  In order to analyze the emotional and psychological state of the nation in the days after the disaster, I have taken the tweet content text in the UCLA archive, and divided them into 30 text files, one for each day following the Earthquake, starting on March 11, 2011.  To measure day to day fluctuations of emotions, I will use a similar methodology employed by Eiji Aramaki PhD (Tokyo University) which takes words from an “Emotion Dictionary” (感情表現辞典) and matches it against the tweet content.  The dictionary classifies different emotions into 10 groups:

1. 喜び – Happiness
2. 怒る – Anger
3. 哀しい – Sad
4. 怖い – Fear
5. 恥 – Shame
6. 好き – Like
7. 厭 – Unpleasant
8. 昻 – Nervous
9. 安 – Relief
10. 驚く – Surprise

(view full screen)

Top 20 emotion words:

Emotions by Day

The following animated chart (press the play button to start it), shows the changes for each emotion category over the 30 days.

(view full screen)