IMAGIN
If you have the time, the inclination, and the wherewithal, then gosh I hope you will come check out the Emergency Management and Public Safety forum at the IMAGIN annual conference this May in Lansing, Mi.  I’ll be presenting on Thursday the 4th with some salacious details of public safety applications we’ve built here at IDV Solutions.

Rapturous Abstract
Spanning Scale: Driving Understanding and Action in Emergency Management Visualizations

Emergency management systems are increasingly moving into extensible and secure online environments where access, specialization, and collaboration are more readily achieved.  As these systems, particularly the geographic component within them, gain confidence and market penetration, their contribution to the overall workflow of emergency managers is quickly evolving beyond the passive display of mapped data.
This presentation stems from solution development with several emergency management clients ranging from regional to federal to international levels.  These organizations, while varying in size and mission, at their core wish to more effectively process and present data, enable their people to have a greater sense of understanding and empowerment, and equip them to undertake a more concrete actionable response.  What are the similarities between clients of these scales?  What are the differences?
Specific examples will be discussed including scale of focus, contextualization, cartographic options and implications, technology, tandem visualization tools, and design considerations.

People
Over the years, I’ve been struck by the similarity of the goals and requirements of Emergency Management / Public Safety applications, regardless of the variety of the scope and functions of those systems.  Whether an emergency manager for a county or for a global entity, we’re all people, and in a crisis (or preferably before a crisis) folks need to get what’s going on, and take decisive and confident action.  How could I put it any better than these guys…

“I’m a person. Bret’s a person. You’re a person. That person over there is a person.
And each person deserves to be treated like a person.”

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

SQL Server 2008 has great spatial capabilities, notably spatial querying.  But performing a Geographic spatial query is a different question and will return different results than a Geometric spatial query.  Neither method is wrong, but it is important to know the differences, and as such what you are asking, in order to understand the results.  Both options are valid, depending on what you are looking for.

Spatial Queries

Geography
A rectangular spatial query using the Geographic (spherical geometry) method is illustrated here.  My delicately manicured fingers are holding a rectangular shape over a globe.  It simply pulls a tight line around my four corner points, and returns anything within.  But look how hosed it appears in a planar system (Mercator)…

 
A Geographic rectangle query in its native spherical environment, and stretched over a planar system.

Geometry
A bounding box query is illustrated here in a Mercator map.  Looks fine –notice how it respects lines of latitude and longitude.  Now check out how that bounding box looks in on a real globe…

 
A Geometric bounding box query in its native planar environment, and warped onto a spherical system.

However, sometimes these results are confusing.  For example, Winnipeg is north of Vancouver and Anticosti Island (see awesome image above), but it clearly does not fall within my Geographic rectangle (because on a sphere, North is relative).

A Geographic rectangle query, with an indication of the Geometric version of the same bottom boundary.

If you are performing a Geographic query, but are anticipating the results of a Geometric query, you might, for example, be confused as to why items falling within the highlighted area above are omitted.  That red dashed line follows what would be the bottom edge of a Geometric query.  It’s all in the expectations of the question you are asking.

By the way, you’ll notice a bigger discrepancy between Geometry and Geography query results the farther you get from the equator.

What the Deuce IS a Great Circle?
Here is a really high-tech illustration of a great circle.  I’ve pulled a string tight over the globe between Vancouver and Anticosti Island.  This is the shortest distance between these two places.  If you were to fly an airplane from Vancouver directly to Anticosti Island, you’d follow this string (the straight line between the two locations).  So check out how this perfectly straight line appears in Mercator!  This is the exact same image, warped onto a Mercator projection.  You’ll also notice that in the Mercator image, the globes lines of latitude now appear straight.

An actual straight line (as illustrated by that string on the globe up there), and that same straight line in planar geometry (in this case, Mercator).

More Reading

• Potential workarounds: http://social.msdn.microsoft.com/forums/en-US/sqlspatial/thread/006785d4-8201-4765-b74c-aa4a053bdd61/
• Great little article: http://www.sharpgis.net/post/2008/01/Straight-lines-on-a-sphere.aspx

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

Divide and Color
What do each of these thematic maps have in common?  They are all mapping the exact same data.  What makes them so different?  They are using different classification methods.

One of the first things that pops up when whipping up a thematic data visualization that has discrete range brakes (commonly a choropleth map, but not necessarily) is how do I bucketize my numbers?  Picking range breaks to drive your color categorization can range anywhere from arbitrary and predefined to rigorously statistical and dynamic, and the various options will generate very different looking results.

Some Options
There are lots of ways to carve datasets into discrete classes.  I’ll go over three of them…

• Quantile
  Breaks the data into equally filled groups
• Standard Deviation
  Breaks the data into statistical chunks diverging from the mean
• Equal Interval
  Breaks the data into equally distant groups

Or you could just eyeball the data and then divide it into range breaks that look good or are easy to read.  This, actually, is probably the most common method that I’ve seen in online mapping applications.  It is also the most fertile ground for misunderstanding.  More details on these methods below, including examples and how-to’s.

Distribution
When choosing a method of classifying your dataset into discrete ranges, there a couple of things to consider off the bat.  First, what does the data distribution look like (if it is dynamic, what does it generally look like?)?  Is it skewed toward one extreme or the other?  Is it relatively normal (bell shaped on a histogram)?  Are there outliers to consider?
Applying various classification methods can create very different impressions of the data.  Any interface is a manifestation of tradeoffs, let’s take a look at some examples…

• Normal Data
  With relatively normally-distributed data, picking a classification method may not make a massive difference in your visualization.
  Your biggest concern is probably how many breaks to make, and what colors to use.  Check out this example of average age per county -nice and normal.

Normal data tends to deliver a relatively consistent message across many classification methods.

• Skewed Data
  Now it gets fun.  With a dataset like the percent of folks who consider themselves multi-ethnic, the distribution is far from normal.  In this case, there is a bulge at the lower end and a long tail that eventually pinches off around 30% multi-ethnic.  What a difference the classification methods make here!
  Am I telling the truth with the map (above) on the left?  Yes.  I can clearly see the locations of higher and lower proportions of multi-ethnic US residents, even regional trends and abrupt shifts.
  Am I telling the truth with the map on the right?  Yes.  I get a clear indication that most places in the US are, proportionally, pretty low in multi-ethnic residents.
  I’m telling the truth about two different things.

Skewed data may look way different depending on the classification method.

Examples in Detail

		Equal Interval for Normal data… Equal interval slices the data into equally distant range breaks.  Some color buckets get more counties than others, but if the distribution is wide, then the visualization will be adequate. The gist: Evenly spaced, unevenly filled buckets.
		Quantile for Normal data… Quantile yields a pretty high-contrast map, that is reliably good looking.  The fact that the data is normally distributed doesn’t really matter –each bucket has the exact same number of counties, but you’ll notice that in order to accommodate that, the ranges have to span varying distances. The gist: Unevenly spaced, evenly filled buckets.
		Standard Deviation for Normal data… Trusty old Standard Deviation.  It is going to look alright in most cases, but it really shines when applied to normal datasets.  You’ve got the mean in the middle and you chunk it out from there based on standard deviation distances in either direction from the mean.  Beautiful. Also, don’t do what I did –you should put actual values in your legend instead of the math nerd standard deviation breaks. And while we’re at it, it’s often a good idea to pick a diverging color scheme for data that is classified by Standard Deviation.  Pick a neutral color for the mean (center) range and then transition to one color on the left and another color on the right.  ColorBrewer gives some nice background here along with a rocking tool to generate your own cartographic color schemes. The gist: Evenly spaced (to a statistician), unevenly filled buckets.
		Equal Interval for Skewed data… Equal Interval falls apart pretty easily.  If the data is remotely skewed then it’s feast or famine for the evenly spaced color buckets.  In this case most of the buckets are largely empty while the low end bucket (0% – 6% multi-ethnic) is jam packed. Equal Interval is more fair to the population as a whole but does not capture smaller scale fluctuations. To be fair, just because all the eggs are in one basket and the map is largely monochromatic doesn’t mean that it’s useless.  You could argue that is a a perfectly fair treatment of the data because it illustrates the predominant characteristic of the data: it’s highly skewed to the lower end. The gist: Evenly spaced, unevenly filled buckets.
		Quantile for Skewed data… Quantile to the rescue.  When buckets are defined by an equal number of member counties, Now Devil’s Advocate.  It could be argued that this method implies a false or misleading heterogeneity if the data.  While the vast number of counties have a proportionally tiny multi-ethnic population, this method could imply a greater variance (as compared to the Equal Interval example above). It’s just not fair. Devil’s Advocate, Advocate.  How could you get any more fair than groups of equal size?  Plus the result illustrates a finer articulation of the variance. Just remember, when reading a map, read those legends and take the range breaks for what they are worth.  Quantile is a good illustration of that. The gist: Unevenly spaced, evenly filled buckets.
		Standard Deviation for Skewed data… Standard Deviation.  It is still providing valuable visual breaks when applied to highly skewed data.  But I can never get too cozy with it because it is so darn hard to explain. The gist: Evenly spaced (to a statistician), unevenly filled buckets.

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

A quick note from Scott Caulk, Dir. of Product Management here at IDV…

For those of you who troll for new web feeds, data.gov has added some new ones.  Where else can you find a World Copper Smelters SHP file?
By the way, there does not appear to be a direct correlation between copper smelting and seismic activity:

Today ESRI announced via an emailed newsletter that “ArcGIS Online Maps Migrated to Google Maps/Bing Maps Tiling Scheme.”  This is good news for anyone who has ever had to wrestle with the GIS-y world of the Platte Carre tile schemes that don’t line up with popular Mercator tile scheme (used by Google, Bing, Yahoo, and pretty much everyone else).

It looks like the service will be available for six more months (maybe more depending on pockets of outrage), with no updated content.

“Goodbye my worthy equirectangular foe.  We hardly knew yee.”

It looks like standing of our rectangular amigo has taken a welcome hit.  While both projections have their pros and cons, they just didn’t play well together and online application providers will have more options going forward as a result of the Plate Carre scheme getting tossed.

My family has been watching an excellent three-part broadcast on PBS called the Human Spark, where Alan Alda speaks with researchers and other scientists about the ‘nature of human uniqueness.’  In one segment Cognitive psychologist Elizabeth Spelke illustrates the ability of 2-year-olds to manipulate symbols and language to build spatial models of the real world, and how this is a specific example of a uniquely human ability.  So chew on that all you non-homo erectuses!

Check this out…

In the experiment above, the green dot stays a green dot until legending language turns it into a spatial model for the real world.  Neat stuff.

At IDV, a big part of what we do is to work with clients who have complicated business architectures that need to be visualized, often through thematic map symbology.  There are so many visual dimensions of point representations to consider, but equally important is their appropriate naming and meaningful organization.  Providing the language framework to bridge symbol to reality bridges the spatial model and drives insightful design.

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

http://vfdemo.idvsolutions.com/collisions/

CollisionWatch is a visualization of traffic fatalities over one year across the United States.  The data is, to say the least, sad, but putting these statistics into the context of when and where can lead to a better understanding of geographic, timely, and causal patterns.  An increased understanding, and perhaps the revelation of predictable patterns and trouble spots can lead the way to better questions.

Data
The data comes from the National Highway Traffic Safety Administration, used with permission, and denotes instances of automobile collisions in the year 2007 (US) where one of more fatality occurred, and what conditions were involved.

Because this data was rich with attributes, we were able to set up a filter set that could return some very specific results.  For example, I can set the time of day and day of week to isolate workday commute times, or weekend nights, to see the picture change.  Seasonal trends can be identified as well; cities encased in ice during the winter months have different patterns than you might expect –and each city is different.

The use of the “tri-state” filter enables the user to ask some very specific questions about factors that contributed to the incidents.  For example: show me incidents where speeding was a factor but alcohol was not.

The basemap tiles are provided via the Bing Map Service and assembled within IDV’s VFX front end.  The “Cartographic” map style is an interesting inverted treatment of the regular old road style, and invention of the clever folks at SoulSolutions, which we became aware of after we thought we had invented it.  This darker, more muted basemap provides a much better foundation over which to drape data, showcasing the data as much more vibrant and identifiable.

Visualization
This visualization uses Visual Fusion’s map viewer, timeline, and filter control.  Incidents are shown as interactive objects on the map and in the timeline.  Clicking any incident pulls up a basic report on the incident including contributing factors.  The incidents are also represented as a hot-spot analysis map overlay.  The hot-spot analysis, or heatmap, gives an immediate visual impression of clustered incidents and trouble spots where heavily stacked points may not.  The combination of these visualization methods provides the benefits of insightful visualization and deep-dive reporting interaction.

Examples

Heatmap of incidents over the Eastern United States.

 
Incidents near downtown Chicago.

Incidents in the Bay Area.

Illustrates the visual interaction link between events in the map and the timeline.

Incidents within the Los Angeles basin occurring in the midday hours and where speeding was a factor.

Incidents within the Los Angeles basin occurring on weekends in the nighttime hours and where alcohol was a factor.

Zooming to the street level can uncover physical risk areas correlated to an incident or cluster of incidents.

Incidents within the Los Angeles basin occurring on weekdays in the morning and evening commuting hours.

http://vfdemo.idvsolutions.com/collisions/

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

http://vfdemo.idvsolutions.com/hurricanes/

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

So from time to time you may want to have some kind of globe-like thing spinning on a web page –and you don’t want to go full-on 3D (expense, plug-in, weight, budget, whatever); just something to get the job done.  One way is the circle mask.  This is an old trick and can have a tendency to look pretty cheesy unless you are careful with your tomfoolery.  

Source Map
Go out and get yourself a map of the world.  Nerd alert: your life will be easier if it’s a cylindrical map projection.  Since this is a repeating animation, you’ll need to have some overlap (add at least one-half-again the original image) so there is no skipping.  The oceans should be transparent.  I’ve also made the equator the most opaque, and increased transparency approaching the poles –this will help make the flat images bulge more when it’s time to Columbus-ize them.

Map of the world for the foreground animation.  Note that an additional half of a world has been added to the width so it can loop without jumping.

And a reversed world to use in the background animation.  Note that it is fainter so it will visually recede.

Animation
This globe is made for Silverlight, so I am using Microsoft Expression Blend to create the storyboard animation.  Using the images described above, I translate the X position of the foreground map from right to left and the background map from left to right.  The use of a reversed Earth animating in the opposite direction as the foreground Earth animation goes a long way toward the illusion of a rotating sphere with depth.  Loop this animation so it rotates forever, or for however long you need it to rotate.

Then mask (clip) the image group so that only a circle is visible at any time.  The result is a really flat looking globe animation.  Not there yet.

 
Animating the two maps in opposite directions within a circular mask.

Adding Depth
Let’s get this squirrely animation looking less like a disk and more like a sphere.  It’s going to take gratuitous static decoration: tons of layers of radial gradients give the illusion of depth and material.  Here they are…

	Circular base with a general background color and a fuzz around the edge (the fuzz is optional, as you’ll add more later).
	Add a layer with a radial gradient who’s darker center tends near the bottom
	A squished ellipse at the bottom will give the illusion of a reflection on the back wall of what is starting to look suspiciously like a sphere.
	Pile onto that another semi-transparent layer who’s darker center of radial gradient tends near the top.  When you add the foreground reflection (in the last step) this layer will really pop. Now you’ve got the background gradients in place and are ready to englobalize…
	Now throw in your circularly-clipped earth animation.  This animation is the sole visual cue for rotation, and within the context of the unmoving depth decoration, goes a long way in making this totally 2D thing look like it is spinning in space.  But, see how the side of Africa is immediately peeling over to the foreground? Illusion busted. If you watch the animation at this point it will still just look kind of like a deflated beach ball flat-rolling along a track.
	Maybe a happy little atmoshere lives around the edges of this globe -a 600 year old optical illusion: atmospheric perspective! See how there is an inner haze around the perimeter that obscures some of the geography around the edges?  This is a double win: not only would Leonardo De Vinci be proud of you, but you are also covering the biggest weakness of the animation: the clip edge where the flat masked content immediately peels from background to foreground.
	The foreground reflection (semi-transparent white radial gradated ellipse near the top) is the last element to connote a glassy sphere.  Now you’ve got the next best thing to an actual 3D globe.

Here’s a short recap video to get a sense for how it looks…

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

One of the sassy new features of Visual Fusion (VFX 4.5) is native heatmapping.  Understand your data in a more visual and communicative way with the hot-spot visualizations that our frequency heatmapping provides.  I’ve been proclaiming my love for heatmaps for a while now, so here is some extremely high production-value video proof…

VFX Demo in a Minute: Heatmaps
  

The data comes from the National Highway Traffic Safety Administration.  Thanks for watching and feel free to give us some feedback.

 

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

http://vfdemo.idvsolutions.com/Piracy/

We at IDV Solutions have just released the first of a series of demos of Visual Fusion Experience (VFX 4.5 to drop November 10); this one focuses on piracy data.  Every week, the U.S. National Geospatial-Intelligence Agency (formerly NIMA and more formerly Defense Mapping) releases reports of "anti shipping activity" in an effort to increase awareness and prevent future events of hostile activity.  Oh, and if you have the time, read some of the incident descriptions…wow.

We put this data into VFX 4.5 where, along with good ol’ dots-on-a-map, piracy events are given the treatment in a historical (30 year) frequency heatmap and the new interactive timeline.

The map and the timeline work together to set the rules of what data will appear.  Zoom in to a smaller geographic location to reduce the points in the timeline.  Likewise, zooming in on a smaller time range will reduce the points you see on the map.  More on this AND relationship and others (like OR) here.

Ok, have at it.  My best friend’s sister’s boyfriend’s brother’s girlfriend heard from this guy who knows this kid who’s going with the girl who thinks you’ll find it pretty fun.  Tell us what you think!

http://vfdemo.idvsolutions.com/Piracy/

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

Here is a sneak peak at a Visual Fusion demo coming up around the corner (it’s since been released: http://vfdemo.idvsolutions.com/hurricanes/).  It takes Atlantic Hurricane data from data.gov and splats them in place and time, where it makes sense to see them.  We’ve all seen maps of hurricanes before, but when the map dipped into a piping hot timeline then there is no telling what can happen. 

Zip around in place and time in this edge of your seat demonstration of IDV Solutions’ new Timeline in Visual Fusion (VFX 4.5). Get ready to live…to truly live.

VFX Demo In A Minute: Timeline
  

 

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

The teaming up of map and timeline visualizations in VFX offer several options when it comes to the parameters used to fetch the data to populate each.  Depending on the purpose of your application and/or the nature of your data, you can configure stored procedures to fetch that data with varying combinations of space and time parameters.  I don’t want to assume which way works best for you and your application, so we’ve left it as open and flexible as possible.

Here are some notable options and what to expect of each…

	OR This is the most inclusive option -get ready for lots of items.  The data returned is a union of when or where.  An item that falls outside of the current map extents may still appear on the timeline, and items falling outside of the current timeline extents may still appear on the map. As you navigate one visualization, data in the other visualization remains unchanged. This is the most reasonable option if your perspective is that the map and timeline are peer visualization methods -each does not filter the results of the other.  It’s a real free wheeler.
	AND This is a super exclusive option -comparatively few items will appear in the map or timeline.  The data returned is an intersection of when and where.  Set up this way, items have to fall within the current map extents and time extents to appear in either visualization.  In effect, the map and the timeline filter each other.  As you navigate one visualization, data in the other visualization will update. This is probably the favored configuration if you need to see very specific slices of data based upon your current map and timeline extents.  It is a more dynamic interaction with your data.
	MAP Centric This option does not consider time parameters in the query, so if the data requested of the server (based upon the map extents) happens to have time notation, then, shoot, it will appear in the timeline -as an afterthought. As you navigate the map, data in the timeline will update with data that happens to be returned. This is the likely configuration for an application that is primarily map oriented but the timeline is available as a curious extra (or not at all).
	TIME Centric This option does not consider geography parameters in the query, so if the data requested of the server (based upon time bounds) happens to have some geographic element, then, shoot, it will appear in the map -as an afterthought. As you navigate the timeline, data in the map will update with data that happens to be returned. This is the likely configuration for an application that is primarily time oriented but the map is available as a curious extra (or not at all).

It all depends on what you want and what story your data is telling (good design is clear thinking made visible).  These stored procedures that determine if/what time and geographic parameters fetch the data are written for each individual feed -and remember that you should probably use the same method for all the feeds in your application.  Otherwise, your users will have no idea why data may or may not be appearing here or there; so be consistent.

Stay tuned for more updates on VFX 4.5, which will be released in November.  Ok, have fun -and if you generate some sweet visualizations, I’d love to see them.

(As an aside, the notions of geography and time are more intertwined than you may have realized)

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

VFX 4.5 TIMELINE

One of the features coming around the bend in IDV Solutions‘ VFX4.5 is the…Timeline!  The Timeline has been a lot of fun to work on and has not only turned out to be an outstanding contextualizer when partnered with the map and other visualizations, but it is also robust enough to stand alone as a pillar visualization.

Navigating the Timeline is as smooth and intuitive an experience as we can manage –and takes it cues from map navigation best practices to leverage the expertise of even novice web users; zoom in and out and pan back and forward through time.  If only 9^th grade could have whooshed by as quickly.

Data is dropped off right when it happened –whether it was a blip in time or a range in time.  As I zoom in, the time buckets expand and break out into finer and finer detail, or I can zoom out and start lumping them together (seconds, hours, days, months, years…).  It is agonizingly flexible.

Here, I have pulled in historic Atlantic hurricane data, including the hurricane time range (those spans at the bottom) and individual advisories.

How does it look?  There is a layout framework that consists of the stuff happens here panel where I see and interact with all the time data stacked up right when it belongs, a time machine panel that I can use to zip broadly through time or set very precise bounds, and a now button that can rocket me back to the present.

When I combine these visualization methods, the data really starts to go nuts.

Visualization Combinations

Arrrrgh!  Ok, now that I’ve scratched that itch, here is space/time visualization of incidents of piracy over the last 30 years.  Lets dive in for a closer look…

Global reports of "anti shipping activity" (piracy) in my lifetime.

We’ve all heard about the problem with piracy around east Africa lately.  Here is a timeline of that area over the past nine years. July, 2008 marks a stratospheric rise in reported events.  Check out the boatloads of events crammed into those columns.  Is it a factor of more vigilant reporting? A marked increase in regional political instability?  A shift in response policies of shipping conglomerates?  All of these things?  By the way, for some more info on the also new heatmapping feature you see here, check out some previous blog posts.

Piracy in the Gulf of Aden since 2000.

The Pirates of the Caribbean are, at least, a consistent crew (also, by the way, much more grisly, having flipped through some of the descriptions).  They have their market ups and downs but nothing like the spike around Somalia (but decidedly more treacherous than the Mediterranean).

Piracy around the Caribbean of Aden since 2000.

And, as I heard Amy Lobben say, the absence of data is data -not a lot of rogue activity in the Mediterranean in recent years.  A relatively hefty month like July of 2000 is a trip to the spa compared to other regions.

Piracy?  What piracy?

TIME AS A FILTER

Don’t need the full-blown timeline but still wish to do some time-based filtering here and there?  No problem.  You can inject a nanobot version of the timeline at the individual feed level to either restrict results to a time range…

Only Hurricanes from the 2008 season will be requested of this feed.

…or pull in archived data from a specific time in the past (or future, if you have that data, but wow that’s pretty proactive).

Sales territories will be color coded according to values circa 4/1/2009.

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

VFX is a visual product and I can think of few elements that can illustrate data as…funly as heatmaps can.

We’ve circled around a method of heatmapping that is business user-centric and appropriate for most data sets.  The overwhelming amount of business data calls for a heatmapping method we call frequency heatmapping, maybe more commonly known as hot spot analysis (which is sometimes an output visualization in eye-tracking research for HCI or Marketing).   By the way, there are many different flavors of heatmaps depending on the story your data tells and it is important that you use the right method to model truth (more on the families of heat maps and what sorts of data fits them here).

What are the benefits of incorporating heatmaps into your Visual Fusion application?  Here are five out of the gate…

When mapping large or dense data sets, consider heatmapping.  In which example are corridors of traffic danger more evident?  Heatmaps reveal geographic trends and hot spots, diminish network overhead, and they look pretty snazzy.

Here is an example where a heatmap is used as a baseline reference.  Brighter areas show relative risk of piracy over the last thirty years.  If I overlay that with more recent instances of high seas hi-jinks, I see a trend away from the Red Sea and Somali coast.

More reading if you just can’t get enough when it comes to pondering heat maps…

• Different Data Different Methods
• Visualizing Frequency
• Optical Illusions
• Product Offerings and Business Applications

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

Thematic cartography is ravaging Indonesia…

Chucking out the mouse just feels right; this is the way it should be.	Visual Fusion Experience for Surface IDV Solutions developed a WPF version of VFX for the Microsoft Surface for a security client recently.  The the geographic navigation controls native to VFX were enhanced by the fun availability of multi-touch user input, similar in feel to the intuitive drag-stretch-fling-pinch-twist sort of actions had in an iPhone -only on a ginormous screen. Unlike most touch technology, there is not a pressure sensitive membrane or monitors at the x and y axis waiting to square-out input.  In the case of the Surface, there is a near infra-red camera sitting at the bottom of the unit looking up at the screen.  And it is actually pretty stellar at allowing many sets of hands work the screen simultaneously.  One person can pinch and twist the map to navigate and spin, while another filters feed inputs, while still another dials an outbound call.  It’s like a data-driven air-hockey table.
Enabling Big Brother or promoting public safety…or promoting the big safety of an enabling brother?	Live Camera Feeds Here, I zoomed into the street level in Redmond and activated a live street camera of I-405.  I can go right up along the highway selecting the cameras and seeing the live traffic status, and contact security offices nearby if there is any trouble.
Tall things = more important things.  Hopefully this isn’t translated too directly in real life (I’m only 5’7").	Extrusions Each tracked asset is extruded vertically based on some data value, in this case, number of employees working within.  It’s a visualization that does a pretty good job of communicating a large data set at a glance and imparting relative importance. -Tall things are more important to me because it represents lots of people.
I want to know more about things right…HERE!	Spatial Query If I want to grab a set of features and show aggregate and individual info, I drag out an area of interest and, BANG, lasso them in.  I can draw as few or as many areas of interest I need.  I can also bump the perimeter nodes around to reshape the areas of interest.  And stuff.
Hold on, I’ll connect you…	WAVE Integration So you navigate to an area of security interest and identified the building with which you’d like to communicate.  Then what? TwistedPair‘s WAVE software enables unified VoIP communication between different device types.  Using the WAVE SDK, IDV Solutions created a user experience native to the Microsoft Surface where a user can communicate others or go into full on switchboard operator mode. For example, if I were one of the security coordination dudes who now own this ap, I could call a security office in Redmond, link the both of us to mobile police units’ two way radios, then patch us all in to an alert being broadcast on an FM band, and call my Mom on an unpatched line and plead with her to get out of town.  -All of that while zooming down to check on live street cam feeds and plotting courses on the highway!  WHEW!
Travis Conti and Daniel Briggs, lead surface developers, begin the sad process of packing it up.	Goodby, Huge Buddy And then, as quickly as it came into my life…it was gone.  Back into it’s enormous static-preventing ziplock bag, back into it’s cartoonish Wile E. Coyote shipping crate, my last contact a fleeting…knowing…touch of it’s dormant acrylic surface -back to its true owners.

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

In more savage days when we were constantly on the lookout for animals who would like nothing more than to pounce on and eat us, it was really helpful that our mammalian eyes were specially tuned to identify and sharpen shifts in patterns.  Maybe we could spot those lion eyes more readily in a sea of tall grass.  These days, among other things, it serves to confound designers and developers of heat map engines.

 Yes, You Are Crazy

See the slight darkening of the circles at their edges where they overlap?  Those dark bands are created in your retina/brain and aren’t really there.  Weird, right?  The illusion is thought to be caused by lateral inhibition.  When the light enters your eye it hits receptors and those receptors send the message to your brain.  But the receptors that are hit inhibit the firing of the receptors right next door so the result is a kind of artificial sharpening or increase in contrast. When your eye tracks the gradient of the circle and the regular/expected shift is interrupted by the overlap of the other circle, your eyes send your brain an artificially increased contrast signal.  The result is those phony bologna darkened bands.  The nature of the bands are very similar to the optical illusion known as Mach Bands.

What the Deuce?

So imagine our head scratching when at the beginning of the development of IDV’s heat mapping engine, we saw these confounded dark line artifacts all over our heat maps and no amount of eyedropping could identify an actual dip in luminance.  Daniel Briggs, our lead heat map developer, first suggested that we were seeing things that weren’t really there and identified them as something akin to Mach Bands, named after eclectic 19th century science philosopher Ernst Mach (the same dude who lent his name to the speed of sound stuff -and, by extension, safety razors).

The Mitigating Effect of Context

Good news.  It turns out, when you are looking at a heat map in an actual map, the pesky optical illusion isn’t so noticeable. In the illustrations to the left you can see some obvious Mach Bands at the intersection of these two luminance blobs that contribute to the creation of a frequency heat map. The banding is reduced if viewed over a pattern that helps to break up the continuous nature of the radial gradient shift. And, conveniently, when viewed over a map the banding pretty much disappears. Since I don’t care an awful lot about frequency heat maps when they aren’t draped over a base map, then I won’t stay up at nights worrying about phone calls regarding dark rings everywhere.  But if we do, there is a perfectly reasonable explanation: biology.

More on Context -it Matters

Here we have an image of the luminance score that goes into making a heat map.  You can see (or at least think you see) the Mach Bands pretty clearly.

 
When a color theme is applied to the luminance score and the heat map is generated, you can still totally see those Mach Bands -which is distracting and a real bummer.

 
But when viewed in a cartographic environment that provides some amount of background variance the Mach Bands for the most part die.  The reduction of the illusion is commensurate with the general business of the cartographic context.  The irregularity of the heat map can have a reducing effect as well.  More matrix-oriented heatmap inputs will tend to have more obvious Mach Bands.  When the input phenomena of a heat map are more organic and tend to clump or coagulate (as our data tends to do), the effect is diminished further.  So you can rest assured knowing that IDV is out there fretting to this extent over delivering the best possible visualization tools.

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

Sometimes if you have a lot of dots on a map you might not necessarily be getting the most bang for your data buck.  The goal of any visualization is effective communication and contribution to understanding.  Visualizations of frequency or density can benefit from Frequency heat maps.

Heat maps are lots of fun and can be valuable visual aids.  Be aware, though, that the frequency heat map is not a scientific treatment of data and the falloff rate of the radials are a general notion.  They are, however, among the best and most effective illustrations of density and with the appropriate data their visual impact can be powerful and revealing.

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

Let me start out with this bold statement about heat mapping: you can’t slice up people.  More on that later.

There are two categories of heat mapping that we are working on here at IDV and while their results might both look like pretty similar, they are some big differences between the two.  They are Value Interpolation and Frequency, each with their own considerations and appropriate data types…

VALUE INTERPOLATION

Summary
This is usually what we picture when we think of heats maps.  Value Interpolation is useful for filling in the gaps between known data measurements to generate a fluctuating value along a continuous surface.  Known data point measurements serve as a sampling of the whole and a weighted distance algorithm estimates the value of each inbetweenzie pixel given its proximity from known values and assigns each pixel in the resulting image a corollary color/alpha value.  Or something like that.

Increasing the frequency of input points results in a more precise and smooth surface mapping.  Relative “brightness” indicates higher data values (not more input points).

Data Type
Value Interpolation requires data that describes continuous phenomena-values that flow inabruptly from here to there like temperature or elevation or pollen.  Continuous phenomena are relatively rare in the business world and are more commonly seen in geologic or climatological visuals.  So never interpolate discrete phenomenon -like population.

Variables
Since interpolation relies on a fluctuating data value, the most important variable is the data field that is chosen for interpolation.  A more global consideration is the interpolation algorithm used.  Kriging seems to yield some of the nicest and smoothest results (and is the method we prefer at IDV) but is more complicated.  There are plenty of other interpolation algorithms each with their own pros and cons.  Beyond that, visual variables include the theming of the surface, like colors and/or transparency.

Examples
Temperature, elevation, pollen count, ozone, gas field productivity, wind speed…

The most common example is, not surprisingly, actual heat maps…

Here is a Value Interpolated surface from WeatherChannel.com of North American temperature. Temperature values are interpolated to fill in areas between the weather stations.  The physical locations of the contributing stations don’t really matter other than affecting the precision of the surface. Heat can be interpolated because it is a continuous data type.  If it is 100 degrees at one station and 50 degrees at a neighboring station, it must be 75 degrees somewhere between the two. (This image may actually be generated using a thermal infra-red sensor on a satellite -not interpolation.  But the visualization would be that same so for our purposes it’s a reasonable stand-in.)

Areas with higher numbers of weather stations won’t make the map look hotter.  The relative locations of the weather stations matter little to the visualization

FREQUENCY

Summary
Frequency is useful for illustrating the relative quantity and distribution of discrete things or events.  The result is a surface where "brighter" blobs indicate lots going on around there.  Heat map is a bit of a misnomer in this case, as heat isn’t an appropriate data type for Frequency surfaces.

Event points are assigned a defined radial buffer who’s pixels are assigned a diminishing value (radial gradation) based upon the distance from the event point.  The radial gradation of all events are summed to assign a cumulative per-pixel vale in a single merged raster surface illustrating the frequency of discrete events.

Increased frequency of event points results in a relatively higher overall value.  “Brighter” areas indicate higher frequency.  Frequency surfaces are an illustrative visual aid to indicate a general shift from less to more, rather than an estimate of what lies between known points.

Data Type
Frequency surfaces require data that describes discrete phenomena; event driven -items or events recorded at a point location but can justifiably be visualized as having an impact on the area directly around it (decreasing with distance).  The idea of an exact point is a bit of an illusion and largely a question of scale.  Many kinds of point data can be thought of as having a fuzzy effect around it. For example, the locations of cell towers have an immediate impact on their position and their signal strength generally decreases with distance.  If a murder happens in a neighborhood it also impacts the area nearby.  In many cases, a cumulative radial blur is a better visualization for data like this.

Data that fits into the Frequency surface paradigm are quite common in the business world.

Variables
Some things or events are more important than others and should have their visual representation weighted as such.  Event points can be assigned a magnitudinal data multiplier to affect the individual elements’ radial distance and/or intensity.

Examples
Broadcast towers, Lightning strikes, Criminal events, IEDs, Collisions, Retail locations, Infection incidents, Home foreclosures …

Here is an example of a Frequency surface of crime events, and a second image showing the input point locations of the events as a reference. The brightness of the surface illustrates the frequency of crime reports.  The locations of the criminal events matter greatly to the visualization (as opposed to the Value Interpolation described above). A criminal event has an impact on the nature of the place directly around it and the relative size and brightness of areas illustrate an approximate crime-iness.

John Nelson / IDV Solutions / john.nelson@idvsolutions.com

Furious activity is no substitute for understanding.
  – H. H. Williams

http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000004714

Microsoft recently released a case study of the solution built upon the Visual Fusion product for the Department of Transportation to, among other things, tell Louis Effa and the Maritime Transportation System "what was going on with marine transportation anywhere around the world—in real-time."

And you can totally email this case study to all of your friends.

Here are some snapshots stripped of reserved interface elements and some secure data feeds showing some of the fun features…

Where did you come from, where did you go?  Any vessel’s previous path going back however far can be fetched and rendered.  Each blip within the history contains logged data such as speed, bearing, destination, conditions, etc.

Aaaaaarrg!  Incidents of "anti-shipping activity" populate the Gulf of Oman.  The fluctuating cloud of incidents gives an immediate sense of trouble spots.  Users may define ad-hoc regions of persistent risk and receive alerts when tracked vessels enter and leave. 

Where do we go now? With nautical routing of any vessel to any port, a vessel can have defined any number of what-if routing plans based upon preferences of speed, risk, intermediary port conditions, canals, etc.
By the way, going through the Panama Canal can be pretty expensive -but so is steaming the extra 8,000 miles around South America.
Combining route finding with the automated weather risk zones (hurricane probability plumes) can really help out nautical emergency planners and responders.

John Nelson / IDV Solutions / john.nelson@idvsolutions.com