At the moment, I am digging through a couple of databases for an upcoming project. I did not really find a tool to quickly get an overview over a large set of categorical data. So I decided to roll my own and write a little nodebox script that walks over a CSV file, determines all the unique value attributes, counts how often they occur and plots the output as an area chart. The tool is good for getting a quick overview of categorical data, esp. missing values and the data diversity.

Download the dbcounter script including a sample data set of the Titanic passengers.
(needs nodebox – OS X only)

Sample pdf output

On a related note, you can also use the freshly released Parallel Sets application by Robert Kosara to determine relationships between the attributes. But that’s step 2 :)

On another related note, I cannot stress enough how awesome python is.

RANDOM WALK

WHAT DOES RANDOMNESS LOOK LIKE?

RANDOM WALK asks this question and presents experiments in mathematics and physics, showing the mysterious interaction of chaos and order in randomness.
The project RANDOM WALK simulates randomness in visualizations, which are easy to understand. In this way, it delivers insight into a phenomenon, which has so far remained unexplained.

Update: lecture slides here. Thanks to Eric Rodenbeck and the other see#4 speakers for some last minute inspirations!

We recently launched a whole new version of the MACE portal. MACE (Metadata for Architectural Contents in Europe) is a pan-european initiative to interconnect and disseminate digital information about architecture. The idea is to connect and enrich various databases containing eLearning material for architectural contents and to connect and make them accessible in novel ways. The project is co-funded by the European commission. If you are more interested in the background of the project, you can also view our info page here. Personally, I am working half of my time on this project here at FH Potsdam.

So, let me give you a little overview of the portal:

MACE users can search for contents in our faceted search application (using elastic lists, of course :). Contents can be filtered by origin, language, media type, expert classification and competencies.

Alternatively, the portal offers a radial tree visualization of the classification taxonomy that can be used to traverse the hierarchy and find associated contents. It was built using flare, and I added some extra-nice splines and a breadcrumb branch.

Upon clicking the header of a search result, users get to the MACE detail page, a pretty rich widget patchwork for displaying and editing the available metadata. It displays not only basic information such as language, description, usage rights etc., but also allows to associate contents with locations, classification terms, competencies, personal tags, rating and comments. The same page can be reached from a content by using the MACE bookmarklet, allowing users to tag and remember or find related contents while browsing.

By analysing the text of resources, we automatically detect persons and architectural projects. We keep the focus to architecture by additionally querying dbpedia to make sure we actually found persons and buildings of relevance in the architectural world. Dbpedia also provides the metadata for these “real world objects” as well as some pretty well-picked flickr images via their flickrwrappr script. If you are not familiar yet with the dbpedia project, be sure to check it out, these guys are up to something.

Finally, users can find their own resources, or expect those of others, on each user’s own page. It displays not only all resources and tags associated with the user, but also a personalized, automatically generated “intro text” characterizing the person’s activity on MACE.

I am responsible for concept and styling of the portal UI, implemented all the flash parts like the elastic lists and the hierarchical visualizations and also a decent amount of the crazy Javascripts driving the site. For me, it was the first major HTML/CSS/JS project in a long time, and apart from the usual frustrations with large code bases in dynamic languages, I really enjoyed working on it, and especially learning jQuery, which is almost like a new programming language on its own. Very interesting.

We will try and improve on the portal over the next 6 months, and also get a bit more experimental now that all the technology is in place.

If you have any feedback or suggestions, drop us a line or start a thread on our getsatisfaction page! And if you are interested the project and the page, make sure to register to get most out of it!

So, here they are. They all refer to situations where there is a set of resources which can be filtered according to different criteria.

Zoom–in

The most common pattern: Given a set of resources, select a subset by  adding an additional (AND-connected) filter criterion.


All photos -> zoom in (Europe) -> photos taken in Europe
Photos taken in Europe -> zoom-in (Cities) -> photos taken in Europe AND taken in cities


Zoom–out

The complementary pattern: Given a filtered view, select a superset by either

• Removing a (AND-connected) filter
  
Photos taken in Europe AND taken in cities -> zoom out (cities) -> photos taken in Europe

• or: Adding a OR-connected filter
  
Photos taken in Europe -> zoom–out-OR (Africa) -> photos taken in Europe OR Africa

• or: Replacing a filter with a more general version
  
Photos taken in Germany -> zoom–out-UP (Germany) -> photos taken in Europe


In either way, you end up with a more general query, that yields at least all of the original results, and usually more.

Shift

Zoom-in and zoom-out can be combined into a shift navigation mode. Here, one part of the filters is replaced by a concept that is neither more general nor more specific than the original one.


Photos taken in Germany -> shift (France) -> photos taken in France


Pivot

Another combination of zoom-in and zoom-out is the pivot navigation. It is very common in web applications like e.g. delicious.com. Given a filter setting and its results, you can jump to a fresh query consisting only of one of the occurring metadata terms.


Photos taken in Germany -> pivot (Moritz) -> photos taken by Moritz


This is often accomplished by adding clickable links to the results’ metadata items.

Querying by examples

This corresponds to a pivot on a number of resources and metadata fields at the same time. The most specific concepts that apply to a whole item selection are collected and used in a new query.


Clinton, Bush, Obama -> query by examples -> American presidents


Obviously, the generalization capabilities here depend a lot on the metadata structure. In some database, the example above might generalize to “persons” or “males”, in others to “Male american presidents after 1980″.

Related set

Demonstrated in parallax and humboldt, this navigation mode follows the same metadata link on the whole result set to construct a new one.


Photos taken in Europe -> related set (photographer) -> Photographers of photos taken in Europe


I realize this is quite dry material and could use some illustrations, examples, references. Nevertheless, I hope it spawns some new thoughts in those of you thinking about search and browsing in web applications!

Today was a good day, so I thought I would share its results immediately, instead of fine-tuning forever – who knows when I find the time anyways!

I built a little facet browser for the New York Times Article Search API - an impressively fast faceted search engine covering over two million articles. So, give it a spin!

Some caveats:

• Don’t look for the page navigation – there is none. Pure laziness, will update it soon.
• The initial counts are based on a search for “the” (which I figured would appear in all articles). Unfortunately, only the top 15 or so values per facet are returned, so you cannot click, e.g. the year 2008 in the beginning. Will fix.
• The API has a request limit of 5000 queries per day. So if your requests don’t work – come back tomorrow morning :)
• Unfortunately, the API seems to support only one value per facet. So, all facets are single-select.(fixed, see comments).

The code is based on my totally revamped elastic lists prototype. I used this project as a little sandbox experiment of how easy customization is possible, and especially how to make a switch from a fully client-based to a server–based filtering model.

After lots of experiments with drupal, indexhibit and wordpress, I finally settled for the simple and elegant frog cms. Highly recommended, if you want to fill a custom layout with dynamic content in a very flexible manner.

This is an experimental visualization project. We are inspired by nature or other real phenomena and try to transform formal aspects of it into data visualization systems.

Imitate and Informate is a project by Cedric Kiefer and Christopher Warnow. It emerged from a course called “Reality in Virtuality” at Potsdam University of Applied Sciences, Prof. Danijela Djokic.