PixelCarpet paper accepted!

A while ago, I got the opportunity to take part in a large, EU-wide research project on computer network security and data visualization. The goal of SASER is to lay the foundation for more reliable, efficient, and secure communication networks (I guess the recent revelations about the NSA infiltration might have played a role in the decision to work with substantial effort inside the EU on advanced networks).

Our part as a team of researchers at the Interaction Design Lab at FH Potsdam was to investigate data visualization, visual analysis tools, and dashboards as a support for computer security engineers. While watching traffic and server activity, they often have to sift through loads of data to filter out the suspicious traces of attacks and other malicious activity. Data visualization would certainly help them exploring data, especially bringing patterns to the light that they would have not expected (and therefore wouldn’t have looked for with their data mining tools).

We wrote a paper about our work and one of the resulting demonstrators (which we call Pixel Carpet) – it now got accepted to the IEEE VAST conference in Paris! (yeah!)

The following brief overview is taken from our team’s website, complexdatavisualized.

It builds on the observation that security engineers know their data and the requirements of their work very well. However, they might not be acquainted with advanced visualization techniques. Visualization researchers, on the other hand, know methods to visualize and analyze data but usually lack insight into the specific requirements of computer network security. The paper revolves around two main contributions:

  • results and learnings from a co-creative approach of jointly developing visualizations
  • a pixel oriented visualization technique that graphically represents multi-dimensional data sets (such as computer log files), reflecting ideas from the collaboration

You can get and read the full paper here:

Landstorfer, Herrmann, Stange, Dörk, Wettach (2014): Weaving a Carpet from Log Entries: a Network Security Visualization Built with Co-Creation. in Visual Analytics Science and Technology (VAST), 2014 IEEE Conference on, 2014 (to appear)
at 27MB or a smaller file (4 MB) without embedded video.

 

Co-creative Approach

User centered approaches are well known in the visualization community (although not always implemented) [D’Amico et al. 2005, Munzner et al. 2009]. Jointly developing the visualizations themselves, however, is rather rare. As we have very good experience with co-creative techniques in design and innovation, we wanted to apply them to the domain of data visualization as well. For example, we tried to experiment with data sets during a day-long workshop with a larger group of stakeholders (a session we called the “data picnic” because everyone brought his/her data and tools).

Visualization

For this paper, we focused on a pixel oriented technique [Keim 2000] to fullfill requirements such as visualization of raw data or a chronological view of data to preserve the course of events. We stack graphical representations for various parameters of a log line (such as IP, user name, request or message) so that we get small columns for each log line. Lining up these stacks produces a dense visual representation with distinct patterns. This is why we call it the Pixel Carpet. Other subgroups of our research group took different approaches that can be found at other places in this blog.

pixelmap-delete-example-2

Snapshot of the Pixel Carpet interface. Each “multi pixel” represents one log line, as it a appears at the bottom of the screen.

Data and Code

Our data sources included an ssh log (~13.000 lines, unpublished for privacy reasons) and an Apache (web server) access log (~145.000 lines, unpublished), and ~4.500 lines (raw data available, including countries from ip2geo .csv | .json ).

We implemented our ideas in a demonstrator in plain HTML/JavaScript (demo online – caution, will heavily stress your CPU). It helped us iterate quickly and evaluate the idea at various stages, also with new stakeholders. While the code achieves what we need, we are also aware that computing performance is rather bad. If you want to take a look or even improve it, you can find it on github.

To bring it closer to a productive tool, we would turn the Pixel Carpet into a plugin for state-of-the-art data processing engines such as ElasticSearch/Kibana or splunk (scriptable with d3.js since version 6).

comment!