Visual Display of Quantitative Information

I just finished Edward Tufte’s Visual Display of Quantitative Information (2^nd ed), a classic modern text (or so I hear) on how to design data graphics, which “visually display measured quantities by means of the combined use of points, lines, a coordinate system, numbers, symbols, words, shading, and color”.

This is a great book that I’m sorry I didn’t read sooner. Some key points I took away:

• Tables are better suited for displaying data-sets with 20 objects or less, while visual graphics are better suited for summarizing a lot of information.
• Color often muddles rather than clarifies data graphics, as the human eye does not easily give visual ordering to colors. Gray-scale shading, however, does convey a natural visual hierarchy, and so better represents varying quantities than color does.
• If color is used, avoid red/green contrasts in consideration of color-blind viewers (5-10% of population). I am VERY guilty of this. Green/yellow/red scales are often my default. Contrasts with blue are a safer bet: color-blind people can generally differentiate blue from all other colors.
• In regards to typography: the more that letters are differentiated, the easier the reading. This means that “serif” rather than “sans serif” fonts are preferable, and all-caps writing writing should be avoided (the more equal height/width/volume of capital letters makes for more difficult reading).
• Do not vary the design of the graphic (ie the scale, symbology, etc), because this distorts how the viewer perceives variation in the data. Variation in the data is after all what the graphic is there to illustrate – and truthfully so.
• The number of dimensions in the graphic should not exceed the number of dimensions in the data. So for example, don’t use area (2D, e.g through differently-sized circles) to represent a 1-D measure, such as the value of a dollar over time.
• Less is more, or, maximize the Data-Ink Ratio. This is the ratio of ink used to represent the data (essential to the graphic) and total ink used to print the graphic (includes grid, frame, axis/scale bar ticks, etc)
• Maximize the “data density” of the graphic – or the number of entries in the data matrix within the area of the graphic.
• “Small multiples”, or a series/block of small graphics indexed by changes in a particular variable, are an effective graphic format because they are inherently comparative and tend to have high data densities.
• Pie charts should never be used, because of their low data-density, and because the human eye is not adept at detecting differences in angles.

Maps in the History of Information Visualization:

One of the most interesting parts of the book outlines the history of data graphics . If you have any doubts that geography is awesome and always has been, consider this. Before charts, before graphs and plots, there were MAPS!

Geographic maps were the first form of data graphics, at least as far as historians can tell. While the first maps found on clay tablets date prior to 3500 BC, thousands of years passed before precise cartographic maps with full grids were created (1100′s AD in China, and 1550 AD in Western civilization), and it wasn’t until 1686 until cartography and statistics merged to create the first thematic map (which Tufte refers to as “data map”, but this has come to mean something else in the IT world, so I avoid the term for clarity’s sake). This early thematic map, courtesy of Edmond Halley, shows the location of trade winds and monsoons. Geographic analysis really blossomed after John Snow’s famous, even mythologized 1854 map of cholera deaths and water pumps in London – some say this kick-started the fields of health geography and spatial epidemiology. Charles Joseph Minard’s multivariate map (published 1869, shown below) of Napoleon’s 1812 Russian campaign is another famous merger of cartography and data visualization that Tufte says “may well be the best statistical graph ever drawn.”

Geocoding in Ruby

Sans confidence, competence, I program! Computer languages have always seemed prohibitively complex and very boring. Given my interest in working with data and ‘information’, taking the leap to explore Information Technology is a predictable and necessary next step…that I’ve avoided for years. Having I.T bigshots for both a boyfriend and a brother has done little to alleviate my dread of the terminal and its command lines.

To bulk up my GIS muscles, I tried and failed to teach myself Python from a book this past Fall. The failure was purely one of attention span…how to trudge through mundane code rules, vaguely applicable to matters of interest, when much friendlier reading material was calling to me from the library shelves? Then in January I started to learn Ruby at awesome (& free!) Learning-to-Code classes held every other Monday night at the Co-Work Buffalo office, which uses Chris Pines’ Learn to Program book/website as a guide. This learning attempt was much more successful, in that I actually accomplished stuff – geocoding being the biggest deal to me and my mapping.

Geocoding is one of the most important processes in spatial analysis: matching a place-descriptive data field (like an address, city name) to a precise location in terms of latitude/longitude. To create a geocoder in Ruby I first installed Ruby Gems, which is a “framework” for managing other packages/libraries of code. I then installed a geocoding package aptly named Geocoder, which operates within the Ruby Gems framework and uses Google’s Geocoding API by default to look up addresses and/or geographic coordinates.

I used these packages to write the program below, which reads in a list of addresses (the file Addresses.txt), and writes a new file (LatLong.txt) listing the latitude and longitude coordinates for each address.

My next programming goal is to venture into the endless possibilities of web-scraping, using the Ruby Mechanize package. Inspired by this hilarious map, I want to mine Craigslist for all the rich social data it has to offer.

Criminal Geography

Even if you do not have a mother/relative who is Jewish and from the Bronx, with a maiden-name that literally translates from German as “worry,” I’ll bet that you can empathize in some way with a childhood spent lovingly smothered by warnings of the dangers lurking within any given scenario – with special emphasis on random and violent crime. Abductions! muggings! rape! oh my!! (Love you mom, you’re great.)

More than crime itself, fear of crime has extraordinary power to transform the character of a place. Based on hearsay, news reports, or even personal judgment of people and/or the built environment, we come to know certain parks, alleys, and stretches of blocks as ‘safe’ or ‘unsafe’. If you accept how Mr. Maslow prioritizes human needs, then you agree that the need for personal safety is a very fundamental one – right above basic physiological functions like breathing, heart-beating, etc. Thus, few things affect our behavior and world-view as pervasively as fear of crime. Collectively, these attitudes help drive the presence/absence of disinvestment and casual surveillance that make neighborhoods thrive or…dive?

I obtained crime data from the past 6 months within the city of Buffalo from crimereports.com, a web service that Buffalo’s police department has partnered with in making crime data publicly available. However, this service only allows users to see 500 crimes at a time, and/or up to a maximum of 30 days in time period. I wanted to see more general trends, and so compiled this data for the past 6 months (the farthest back for which data is available) by copying and pasting the listings in 500-crime increments into a text file, which I then geocoded.

OVERVIEW:

Of the 8,084 crimes reported within Buffalo within the past 6 months (9/24/2012-3/23/2013), 3,644 or 45% of them were thefts. The most crime-ridden day was 10/10/2012 (81 crimes reported), and the most common time associated with a crime was 12 pm (692 crimes), followed by 9 am (207). This time data is a little funky – a huge chunk of the day is missing (1pm to midnight), and I can’t find any metadata to specify whether this field referred to the time the crime was committed or when it was reported. So I haven’t looked  at any ‘time of day’ patterns too closely.

CRIMES BY TYPE:

Buffalo’s crimes are classified into the following categories, noted with the % of crimes reported  within the past 6 months that fall within each category. I paraphrased crime type definitions from the FBI’s Uniform Crime Reporting Handbook.

• Theft (45%): Completed or attempted theft of property or cash without personal contact.
• Breaking & Entering (22%): the unlawful entry of a structure to commit a felony or a theft.
• Assault (18%): Attack on one person by another. Includes both aggravated assault (for the purpose of inflicting severe or aggravated bodily injury) and simple assault (does not feature weapons or serious injury)
• Robbery (7%): involves a theft/larceny but is aggravated by the element of force or threat of force.
• Theft of Vehicle (7%): …yep, like it sounds, theft of a vehicle.
  
• Other (1%): Includes Theft from vehicle, Homicide, Sexual Offense, Property Crime-both commercial and residential
  

CRIMES BY DATE:

The chart below shows the number of crimes reported by week, from late September to late March. I expected to see a spike in crime (particularly thefts) in November and December, preceding the holiday season, but this was not the case. In fact, there has been an overall though slight downward trend in the number of weekly crimes reported since late September.

CRIMES BY LOCATION:

The single location with the most crimes reported (42) was the 2100 block of Elmwood Ave – which is not in the Elmwood Village, but north of Hertel and surrounded by the parking lots of a Home Depot and other stripmall outlets. The 2600 block of Delaware Ave came in 2nd place (41 crimes), and is a similarly sprawled parking-lot-scape, as is the 600 block of Amherst Street (35 crimes). In each of these top-three crime locales, over 85% of reported crimes were thefts, presumably from the surrounding big-box stores.

If you are interested in zooming in to particular parts of the city to look at the data in greater detail, see the google map below. Crimes are color-coded by type (the three most common types), and if you click on each point you can also see the crime date, time, other notes, and the number of crimes committed at that location. At this time, Google maps is unable to differentiate points that occur in the same location. So, for example, the 42 crimes that occurred on the 2100 block of Elmwood Ave are in this map represented as one point, though the #Crimes@Location field does indicate that 42 crimes were reported there.

Crimes Reported in Buffalo, NY, 9/24/2012 – 3/23/2013:

To examine the distribution of crime beyond the scope of singular addresses, I created heat maps. These show the relative concentrations of crime incidents within a half-kilometer radius from each point throughout the city. The map below represents total crime, and is followed by maps showing the relative concentrations of the three most common types of crimes: thefts, breaking & entering, and assaults. Across all crime types, the two main hot spots in the city are the center of downtown (centering around City Hall), and the intersection of Genesee and the railroad east of Bailey Ave.

Data Analysis in R: Loan Interest Rates

If you like school and are not already familiar with Coursera, I highly recommend you check it out. The internet has come a long way in evolving beyond its primordial functions (wasting time and checking facts), but I was surprised to discover that it can now be your free college education. Through Coursera, you sign up for courses being taught at universities around the world, and each week, you go through the lectures, quizzes, and assignments at your own pace. Of course, you don’t get actual college credit for your efforts (though that may soon change), just a certificate of completion.  But that doesn’t really matter to school-lovers like myself, who often question the cognitive payoff of the time, money, and stress vested in pursuit of post-secondary degrees (ie could I have made off like Will Hunting if I just spent enough time in the library? This is a foolish thought, as I spend most of my time in the library reading cookbooks, as well as Margaret Atwood and Zadie Smith novels, but I can’t help but wonder anyway).

I’m currently taking a Data Analysis course, taught by Jeff Leek through the John Hopkins Bloomberg School of Public Health. I took a couple of stats courses in college and am comfortable with most of the concepts, but I wanted to be better. Specifically, I wanted to learn how to use R, the open-source software for statistical analysis and computing.

The first assignment was due this morning: an exploration of the relationships between peer-to-peer loan interest rates and other characteristics of the loans and their applicants. If you’re interested (don’t worry, I’m not offended if you’re not) An excerpt of my own analysis is below:

Loan Amount and Length Affect Interest Rates for Applicants with Same Credit Scores

Introduction

Peer-to-peer lending involves money lent between individuals, without involvement of traditional financial institutions, such as banks [1]. Like all lending, peer-to-peer loans involve different risk levels to the investor, depending on the characteristics of the loan and applicant. Higher risk is reflected in higher interest rates, which are undesirable for applicants as they increase the overall cost of the loan. An important characteristic reflecting risk to lenders is the applicant’s FICO credit score: lower scores reflect poor credit history and may therefore warrant the lender to charge higher interest rates. Given this relationship between FICO score and interest rates, the purpose of this data analysis is to identify associations between peer-to-peer loans’ interest rates with other characteristics of the loans and their applicants (controlling for the applicants’ FICO scores). These associations are important, as they give insight as to how peer-to-peer lenders consider characteristics of an applicant and prospective loan in determining a loan’s interest rate.

Methods

Data on 2,500 peer-to-peer loans was obtained from the Lending Club [3], downloaded as a csv file on 2/10/2013 [4], and analyzed using R Statistical Computing Software [5].

To evaluate the relationship between the loans’ interest rate and characteristics of the borrowers, I used multivariate linear regression, pairing average credit score with each other independent variable, with interest rate as the outcome variable.

Multivariate linear regression demonstrates the unique contribution of each categorical or continuous independent variable to variation in the continuous dependent variable: loan interest rate (IntRate) [6].

I began with a uni-variate linear regression model, estimating interest rates (IntRate) as a function of average FICO score (FICOavg). I interpreted the relative contribution of each variable to the model through its adjusted coefficient of determination, or Adjusted-R² value, which only increases in multiple regression if a new variable added to the model increases the model’s explanatory powermore than would be expected by chance [7].

Results

The data set includes records of loans granted in 46 states across the U.S. Besides state, other categorical variables in the data set included: applicants’ employment length, home ownership status, and FICO score range, as well as the loan purpose, and loan length. Quantitative variables included: applicants’ monthly income, debt-to-income ratio, revolving credit balance, the number of open credit lines, the number of inquiries within the last 6 months, the loan amount requested by applicants, and the loan amount funded by investors.

Loans funded by investors range in value from $0 to $35,000, with the mean value being $12,000. Loans are either 36 or 60 months in duration, and are categorized into 14 purposes, the most common being debt consolidation (n=1307) and credit card (n=444).

To determine the baseline explanatory power of average FICO score (FICOavg) for interest rates (IntRate, measured as a decimal from 0 to 1 rather than a percentage), I tested the following univariate linear regression model:

IntRate = b₁ FICOavg + error

where b₁ = -8.46e-4, indicating that an increase of 1 point in the applicant’s average FICO score corresponds to a reduction of 0.0846% in the loan’s interest rate. This relationship was highly significant at the P<0.0001 level, and the adjusted R² value of the model was 0.5026.

I then tested each other variable’s effect (independent of FICOavg) on IntRate by adding them to the regression model, one at a time:

IntRate = b₁ FICOavg + b₂ Var2 + error

Where Var2 was the added or “new” variable. If Var2 was a categorical variable, then it was added to the regression model as a factor. A variable’s independent effect on interest rates was considered “strong” according to the following two criteria: (1) the new variable coefficient’s P value was less than 0.05, and (2) that variable contributed to the model’s predictive power of interest rates (interpreted in this analysis as an increase of at least 0.10 to the baseline adjusted R² value of 0.5026. Thus, the adjusted R² value must be at least 0.6026).

Finding 1: Longer loans have higher interest rates

The length of the loan (Loan.Length) was the only categorical variable that met these criteria. Loan.Length had two possible values: 36 months and 60 months. Including Loan.Length as a factor in the regression model increased the adjusted R² value by to 0.6896 (an increase of 0.187 from the uni-variate model’s adjusted R² value). The b₂ coefficient for the Loan.Length factor of 60 months was 0.0427, indicating that 60-month loans have the effect of increasing interest rates by 4.37% (P<0.0001) as opposed to 36-month loans among applicants with the same FICO score. The Figure below represents the differential effects of loan length on interest rates, while controlling for applicants’ FICO scores.

Finding 2: The more money that is requested/funded, the higher the interest rates

Among quantitative variables, only two contributed to the model’s adjusted R² value by at least 0.10: the value of the loan requested by the applicant (Amount.Requested), and the value of the loan funded by the investors (Amount.Funded). It is intuitive that these variables are highly correlated (larger requests are more likely to result in larger amounts funded), and indeed these variables have a Pearson’s correlation coefficient of 0.9698 (P<0.0001).

The b₂ coefficient was exactly the same for when Amount.Requested and Amount.Funded were added to the multivariate regression model: 2.11e-6 (P<0.0001 for each variable). This means that for each $1 increase in the loan amount (whether the amount is being requested by the applicant or funded by the investor), the interest rate will increase by .000211%. The adjusted R² value for the model when Amount.Requested was Var2 was 0.6564, and was 0.6551 when Amount.Funded was Var2. Thus, each of these variables increased the adjusted R² value by at least 0.15 compared to the baseline model considering only FICOavg. The figure below demonstrates visually how the value of the loan requested (Amount.Requested, broken into quartiles) stratifies a scatterplot of interest rates versus credit scores.

Other (Weaker) Relationships

When added to the multivariate regression model, other variables in the data set did have statistically significant (P<0.05) b₂ coefficients, though none contributed to the baseline adjusted R² of 0.5026 by more than 0.001, and thus did not significantly increase the model’s predictive power of interest rates, independent of average FICO scores. Variables that had positive and significant b₂ coefficients (and thus were associated with increased interest rates for applicants with the same average FICO score) included monthly income, revolving credit balance, inquiries in the past 6 months, employment length of 10+ years, and loan purposes that were “house” or “small business”. Only one factor had a significant negative relationship with interest rates: applicants whose home ownership status was “rent”.

Conclusions

Loan length and the value of the loan (amount requested and amount funded) were the variables with the strongest effects on interest rates, independent of credit scores. Loans with a length of 60 months were shown to increase interest rates by over 4% compared to loans with a length of 36 months, while each dollar added to the loan amount requested/funded was associated with a .000211% increase in interest rates. There are several weaknesses in this analysis. The associations should be treated with caution because of the wide variation in units: interest rates varying by fractions of a percent, while FICO scores, incomes, and other variables are measured on different and larger scales. In addition, my multiple regression equations did not include interaction terms between FICO scores and other explanatory variables included in the analysis, which may affect how associations are interpreted [8].

References

[1] Wikipedia “Peer-to-peer lending” Page. URL: http://en.wikipedia.org/wiki/Peer-to-peer_lending. Accessed 2/15/2013.

[2] MyFICO “Credit Basics” Page. URL: http://www.myfico.com/CreditEducation/Articles// Accessed 2/16/2013.

[3] The Lending Club. URL: http://www.lendingclub.com/. Accessed 2/16/2013.

[4] Project Data Set. https://spark-public.s3.amazonaws.com/dataanalysis/loansData.csv. Accessed 2/12/2013.

[5] The R Project for Statistical Computing. URL: http://www.r-project.org/. Accessed 2/16/2013.

[6] Virginia Tech Department of Statistics “Interpreting Multiple Regression: A Short Overview” Page. URL: http://www.lisa.stat.vt.edu/sites/default/files/Multiple_Regression_Analysis.pdf. Accessed 2/15/2013.

[7] Wikipedia “Coefficient of Determination” Page. URL: http://en.wikipedia.org/wiki/Coefficient_of_determination. Accessed 2/16/2013.

[8] Macquarie University PSYSTAT “Testing and Interpreting Interactions in Regression – In a Nutshell” Page. URL: http://www.psy.mq.edu.au/psystat/documents/interaction.pdf/. Accessed 2/16/2012

Buffalo 2012 In Rem Auction Results

A shot of the auction I took 10/29/2012

In September, the list of tax delinquent properties that the city first published to be auctioned October 29-31 2012 had over 5,000 addresses. By auction day, that list shrunk by 40% : over 2,000 property owners paid their taxes and by the afternoon of October 31st, 3,192 properties were offered up for auction. Less than one third, or 1,090 of those properties, were bidded upon and sold to auction attendees. 1600 were “adjourned” or returned to property owners, while just over 1400 were “struck to the city,” that is, they are now city-owned.  A break-down of auctioned properties according to whether they were bought, adjourned, or struck to the city, and whether they were vacant lots or had homes/buildings, is provided in the chart below:

LOTS

Among the 265 lots that were purchased, the average winning bid was $876.

222 or 84% of those lots were sold for $500, while 29 of them (or 11%) were sold for over $1,000. The most expensive lot, at 783 Niagara Street, was purchased on behalf of D’Youville College for $29,000 (91% of its tax-appraised value of $32,000).

Below is a map of all LOTS sold at the 2012 auction, color-coded by the value of the winning bid. When you click on each lot, you will see the exact value of the winning bid along with the lot’s address and its tax-assessed property value.

Lots purchased at the 2012 auction

BUILDINGS/HOMES

Among the 826 buildings/homes that were purchased, the average winning bid was $7,071 though 50% of the winning bids were $3,000 or less.

218 or 26% of those homes were purchased for only $500. The most expensive purchase, at 1740 Hertel Avenue, was $158,000 – nearly double the property’s tax-assessed value ($82,000).

Below is a map of all BUILDINGS sold at the 2012 auction, color-coded by the value of the winning bid. When you click on each building, you will see the exact value of the winning bid along with the lot’s address and its tax-assessed property value.

Buildings purchased at the 2012 auction

PROPERTIES THAT WERE NOT BID UPON

The properties below were either adjourned or struck to the city, because nobody bid upon them. Opening bid was usually $500, though for some properties it was considerably more. For example, Brian and I had our eyes on a vacant lot on Masten Avenue for gardening – but the opening bid was $6,500 (forget THAT), but in hindsight it may have been worth it because purchasing vacant land for gardening from the city the regular way is proving much more arduous than anticipated, mainly because the city doesn’t see gardening as the “best use” of vacant residential land – they expect you to build upon it. I think that’s a little unrealistic given Buffalo’s decline and overabundance of the existing housing stock, but anyway, the map is below:

2013 [Urban] Planners

I’ve kept a weekly planner since I was 10 years old. Of course then, it was mainly an assignment book: when was math homework due? Which mornings did I have band practice? How many nights to go before the big slumber party? The contents of my planners have evolved as much as I have, so I try to save them.

I’ve had Moleskine planners for the past couple of years, which are great. But as 2012 drew to a close, and I thought about purchasing another little booklet in which to (rather compulsively) document my schedule for 2013, I decided I wanted something a bit different. I wanted a planner that played on the ever-fascinating concept of URBAN planning, with maps of my favorite cities and neighborhoods, as well as inspiring quotes about urbanism and planning in the header of each week. But none such existed!

So I designed one. Well actually, I designed four. One for myself (with Buffalo’s historic Allentown neighborhood on the cover, my current home) and three others for immediate family members with cover maps of their home cities (Burlington VT, Denver, and NYC).

I had them printed through LuLu’s self publishing service. You can order a 2013 [Urban] Planners for yourself, too! I currently have Allentown (Buffalo NY), the Elmwood Village (Buffalo again!) Burlington VT, and NYC, and Denver. Coming soon are Buffalo NY (the whole city), San Francisco, Boston, Chicago, and Minneapolis. Check it out !

Rail against the Highway

Several weeks ago, a certain article got me all disgruntled about how some congressmen do not take high-speed rail seriously as a viable future for regional transportation – particularly between such major metropolitan areas as San Francisco and Los Angelos. One Californian commented:

“It would be a fun thing to have, but can we really afford a fun thing?”

A fun thing?! Blood boiled. Another said:

“Maybe it’s time to cut our losses and pay down our debt…I have real doubts about the viability and the cost. Just because we’ve [already] invested money doesn’t mean we have to invest more.”

But what about the most costly and inefficient public works project ever undertaken: the Interstate Highway System? Just because we’ve already pumped over 6.5 trillion dollars into it doesn’t mean it has to keep dominating transportation spending, ruining urban development, and limiting who can get around and how (adults with the physical and financial means to take drivers ed and receive their license and own/operate a private vehicle).

As I show below, the direct costs come close to high-speed rail, and when you factor in the indirect costs on the environment and public health, the superiority of high-speed rail to the status-quo-interstate-highway is painfully obvious.

A break-down of state/local and federal government spending on the U.S highway system (for both construction and maintenance) in terms of 2010 USD is below. I added up the costs for each year according to this curve, and estimated a total of over 6,500 billion dollars spent from 1954 to 2010. If we divide this by the interstate’s current mileage, that amounts to a conservative cost estimate of $2,567,846 per mile per year.

Though I also want to take a stab at quantifying the health and environmental costs the highways have imposed. The CDC recently reported that highway fatalities cost the U.S 41 billion dollars each year in medical costs and work loss.  Granted, this is not a direct government expense like in the graph above.  There have been over 1,313,600 people killed on the U.S highway system from 1980-2010, which averages 65,600 per year, though annual deaths are declining and have been under the 40,000 mark since 2007. What are the fatalities associated with high-speed rail? Well, a Japanese rail line called the Shinkansen has had 0 deaths among its 9 billion passenger rides in over 40 years of operation. Costs associated with noise and congestion from highways has also been quantified in 2000: amounting to over 66 billion dollars just for that one year, again, just for costs associated with congestion and noise.

Several years ago Harvard economics professor Edward Glaeser estimated the cost of high-speed rail to be 2.5 million dollars per mile per year for capital costs plus 200,000 per mile per year for maintenance. That brings the projected total cost to be 2.7 million dollars per mile per year. He concluded that while trains are nice, that price tag just is just not feasible.

But let’s look back to the average annual cost of highway construction and maintenance: 2.57 million dollars per mile per year directly out of the government’s pocket. Glaeser estimates high-speed rail to be slightly more expensive, at 2.7 million dollars per mile per year. However, unlike highways, rail will not impose general economic costs upon Americans of over 100 billion dollars per year due to fatality, noise, and congestion. And this doesn’t include costs associated with the reduced air quality, environmental degradation, and greenhouse gas emissions that highways perpetuate. Even though our government has wasted trillions to make it convenient, driving down the highway just is not a smart way to get around. It’s about time we got serious about financing the infrastructure for alternative regional transportation.