software development – Page 3

Hallways user testing: the crow on the shoulder.

Every software-product developer should be familiar with the milestone-book "Don’t make me think" from Steve Krug, and probably lots of us read it.

I did, and I agreed!

This happened some years ago…

Nowadays my days runs faster, and it may happen that old precious hints get lost.

Some days ago, I called Massimo, an expert .NET developer, for testing our new product BugsVoice, but secretly to push it to develop, for free, an example of .NET page for catching errors (please don’t tell him).

He sat down in front of me (see our space here ) and started to enroll in BugsVoice.

I was peeping him over the monitors barrier watching his face struggling, and then I

remembered Steve’ rule, that actually is the all-month tips… IUCN_054

So I moved around the table and I sat behind him, really close, looking over the shoulder , to see both monitor and keyboard, like a crow on the shoulder…

… and the magic happens!

I discovered in a half hour session that some parts of our application are hard to understand or completely unclear.

We have already used, with satisfaction, professional testing service like UserTesting for our products; they usually provide you a video file recording user voice and screen interaction, but seeing a “human” (are programmers human?) face give you an emotional feedback that you will miss in video files.

Some hints to get the best from user testing:

give the tester a small overview of your product. Do not explain your product in detail, give only the context. Prepare a short story board: I gave my tester the following tasks: enroll, choose an error template, configure your server, collect some errors.
invite the tester not to be shy and talk while playing with your app. Explain that it is not a competition, and that you are testing the application not the tester 🙂
resist to interfere. Do not help your tester and take notes when the tester stops. Eventually hearten him and ask what’s the matter.
look at the mouse movements. If the pointer goes around the screen without a goal, probably there is something too small, in a bad place, or too far. Take notes!
look at the tester’s face. When you see self-satisfaction (seems that he/she is understanding something) ask what he/she thinks that feature is. Take notes!
let the tester play whith you app discovering features. Take note of the path the tester follows to reach the goal. Is he using different/unplanned ways? Probably there is something to be done on the interface!
look at the clock. Usually a “real” user does not give you more than few minutes before giving up. It is valuable to reach at least a rewarding step in the first minutes.
at the session end ask the tester an overall impression of your application. Ask what he feels about each phase (enroll, step 1 etc.) and take notes
then ask the tester some suggestions to refine your app. Take notes, but then throw them away, you already have a lots of better feedback.

In our case a thirty minute session gave me five A4 pages full of interesting notes.

What I wonder about user testing is that very often what misleads the user is a wrong message, a color in a title, a misplaced button, a hidden something; all things that you can fix in minutes and usually every programmer tends to underestimate. Of course sometimes you may get in serious trouble when you miss a core point; this is why it is crucial to start user testing very early. I know that it is sometimes hard to do, how to test an application if a core functionality is missing?

We started user testing quite late with BugsVoice alpha (internally), and we are continuing with the public beta.

Give yourself the opportunity of a day as a crow on the shoulder, you won’t regret it.

Solving the “halting problem”…

When I asked Gino (alias Roberto Baldi mostmobbed) a software solution for the “halting problem”, he told me “should not be so difficult”!

“In computability theory, the halting problem is a decision problem which can be stated as follows: given a description of a program and a finite input, decide whether the program finishes running or will run forever, given that input.” (from Wikipedia)

Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist; but as Gino says, Turing was a pessimist…

Why I’m scratching about the halting problem? Because I need power at BugsVoice user’s fingertips. In particular, I need to allow the execution of user’s JavaScript code on my server, and knowing if these scripts end may be comforting.

If you already heard about XSS you probably know that “third party” code execution, authorized or not, could be a nightmare even in the client… can you imagine how can be ugly on your server a “pirate” code?

Going more in detail, BugsVoice is a service that receives a “bug” from a customer’s server, process the request locally, serves a friendly feedback to the user and stores the bug in its database.

JavaScript (JS from here on) server-side execution is involved in the “processing” phase.

We supply a pre-filled and certified set of “rules” for processing bugs, but we even allow customers to create their own rules.

JS gives you the power of inspecting the error to understand what happened and gives your customer an error better than a “500 server error” in order to comfort it and recover a situation where your application credibility is going down. An interesting reading about error recovering and error feedback is the book “Defensive design for the web” from 37 Signal.

The complete BugsVoice process includes mainly three parts:

1) on the customer’s server side, an error page that catches the exception, collects as much information as possible (logged user, time, server status, database status, memory etc.) and redirects the user to our BugsVoice server (see how to configure an error trapping page on BugsVoice blog for more details).

2) our server, reading user preferences recovers the error template. Each template is fully dynamical and customizable; it introduces some “variables” that can be filled from the error happened. Then our server creates two JS objects: the “bug” object filled with the error collected and the “template” object filled from layout skeleton.

3) the JS rules are executed to fill “template” from “bug” or for rejecting the request.

4) the layout is rendered to the user by using “template” and “bug” objects. The bug is stored on our server.

5) the user feedback is collected and stored.

6) a “thank you” page is displayed to the user.

Then there is the error management but this is interesting “only” for BugsVoice’ users, not for this post. Here some error pages from BugsVoice:

So every user can create its own rules in order to inspect, for instance, the received bug’ stacktrace trying to discover if a database problem happens, or if there is a problem with the latest version of some browser.

Coming back to rules execution:
during step 3) we get rules from the user configuration and we execute them on our server. We use the Java SE 6 scripting features supplying an ECMAScript engine to run rules. A scripting engine instance is isolated from the JVM environment and you must declare the resource (libraries) you want to made available in the execution context.

Before executing them, the context is fed by “bug” and “template” objects. Then we run the rules…(drum roll!).

A basic (and friendly) rule example :

if (bug.code==404)  errorPage.errorMessage="Page missing: you get this error because of...";

Of course this code is safe, but what happens if an evil user composes a pleasant rule like

while(true);

or

function snake(s){
  return "s"+snake(s);
}
snake(":-<");

… or even worst?

Sadly Turing beats Gino 1-0, and there is no general solution to the question “does this rule ends?”.

The only possible solution is to narrow the scope of the problem by introducing some fences.

A solution is to set up an external observer using multi-threading and watch dogs in order to kill processes after a while, but the best solution is to avoid infinite loop situations.

Rules in our context are used mainly for discovering string patterns in the error stacktrace and for building better feedback; we do not need to iterate or create complex functions, so reducing the set of possible JS statement is possible without loosing “power”.

Luckily in JS there is a limited set of statements for iteration and recursion; so if we are able to “kill” bad intentions by forbidding dangerous statement like “while”, “for” or function definition we can run rules with confidence.

This way we reduce the complex halting problem to the (quite) easy problem of HTML sanitization (where you must remove some unaccepted tags. See XSS war: a Java HTML sanitizer ).

Actually identifying “while” or “for” statements in a complex code is not as easy as finding the “while” string. The find/replace approach it’s too rough, and here we need a more accurate solution in order to understand the difference between

while (true);

and

var dummy= “while(true)”;

that is obvious for us but not for a string searcher…

You must use something to analyze the code token by token.

ANTLR 3 supply all you need for tokenizing, parsing and walking your code. You need a JS grammar and then ANTLR will build all the stuff. We used the ES3 grammar from Xebic Reasearch (BSD license) based on the original work of Patrick Hulsmeyer, that fits perfectly our needs.

With the AS3 grammar we built parser, lexer and walker to analyze rule’s code to intercept every forbidden statement and avoid accepting dangerous scripts (at least I hope this). Only the rules that pass the test will be saved on the system and will be available to the script engine.

Ok, I can confess you, the post’ title is a little misleading, there is no way to solve the halting problem at least without cheating!

A jQuery text extractor (via Java proxy)

Developing online services seems always to bring about new challenges: yesterday’ problem was to extract plain text from a generic web page.

Using jQuery you can easily extract text from an element by using $(..).text(), so I wanted to put this function to use; the problem then was to make the entire web page available to jQuery.

Before extracting text, you should at first load the content of the intended page somewhere; if you want to preserve the integrity of your calling page, the only suitable solution is to use an iframe as container:

here are the html basic elements:

  <input type="text" name="FILLFROMHERE" >
  <span onclick="fillFromURL()"></span>
  <iframe id="pageExample" style="display:none;"></iframe>

In order to fill the iframe:

function fillSpamFromURL(){
  var url=$("#FILLFROMHERE").val();
  if (url && url!=""){
    $("#pageExample").attr('src', encodeURIComponent(url));
  }
}

You should wait until the iframe loads the page before starting the text extraction; bind the "load" event on the iframe and it will be raised at the right time:

  $(function(){     $("#pageExample").load(extractInfos);   })

At this point you can access the iframe content by

$(this).contents()

Sadly I discovered what probably I should have known already: cross site DOM inspection is forbidden (see Cross-Domain Communication with IFrames).

To work around this limitation I wrote a really simple ". jsp" proxy page in order to load the "external" page in my domain:

<%@ page import="java.io.InputStream,
                 org.apache.commons.httpclient.*,
                 org.apache.commons.httpclient.methods.GetMethod,
                 java.io.InputStreamReader, java.io.BufferedReader" %><%
  String url=request.getParameter("url");
  HttpClient client = new HttpClient();
  HttpMethod method = new GetMethod(url);
  client.executeMethod(method);
  InputStream bodyAsStream = method.getResponseBodyAsStream();
  StringBuffer sb = new StringBuffer();
  InputStreamReader streamReader = new InputStreamReader(bodyAsStream, "UTF-8");
  BufferedReader reader = new BufferedReader(streamReader);
  while (true) {
    int cr = reader.read();
    if (cr < 0)
      break;
    sb.append((char) cr);
  }
  String thePage= sb.toString();
  bodyAsStream.close();
  method.releaseConnection();
%><%=thePage%>

Now the iframe load function should be changed to :

$("#pageExample").attr('src', "proxy.jsp?url="+encodeURIComponent(url));

Warning: the proxied "page" can’t load relative resources, so this solution may not be a suitable for everyone.

The jQuery .text() function will retrieve text even in the script tags, so you can use a little trick to remove unwanted tags:

function extractInfos(){
  var ifraBody = $(this).contents().find("body");
  ifraBody.find("script,style,object,link,embed").remove();
  var text=ifraBody.text()+"";
  var re = new RegExp("(\\s){2,}", "g");
  text =text.replace(re,"$1");
  $("#EXCERPT").val(text);
}

The regular expression replacement will remove replications in "space like" chars.

I’m using this code to instruct anti-spam services (like Defensio or Akismet), supplying them sample contents to tune spam detection, which we will use for Patapage (patapage.com)’s comments. These services usually need to know the content of the main article before rating a comment as spam or not.

XSS war: a Java HTML sanitizer

Yesterday I was testing one of our forthcoming online services, in order to check XSS (Cross Site Scripting) robustness.

XSS and CSRF (Cross Site Request Forgery) are the two scary “black beasts” of online services in good company with scalability, but while even nerd-programmers think about scalability, almost nobody takes care of XSS and CSRF, at least until the first attack 🙂

The XSS becomes a serious problem when you allow your users to add contents on your site/service; typical situations are blogs, forums, online chats etc.: when you are about to “print” user contributions you must do it conscientiously, you may found some “easter egg”.

Of course if you limit user contributions to plain text you will solve the problem in minutes, by just encoding every html tag (the idea is to change every “>” in “>” and “<” in “<” and so on). But things quickly get harder when you try to accept some basic html tags.

In our case, Patapage is a service that allows to add comments, threads and more options to existing web sites, it is tailored to appealing interfaces so it is a MUST to allow users to insert a wide set of html tags. We have found on stackOverflow.com a good balance from admitted tags and refused ones; as usual Jeff Atwood is prodigal of precious hints, but in our case while his set is fine for “patacomments” it is too restrictive for the scope of “patacontents”.

As you can imagine, I’ve found a lots of holes, so going upstream as usual and not giving a damn on, probably, the best Jeff’ suggestion (“do not re-write you own implementation”, an advice which he too is not following) I started writing code basing my implementation on these articles.

Another aspect of the problem is the conservation of layout: users can break the page by inserting unclosed or misplaced tags. If you hope that these flaws do not impact on security you are on the wrong path. A well planned css attack can, for instance, layer your application for click stealing or simply “switch” two buttons of your application with funny(?) results.

Of course I’ve looked around to find something matching my needs, but I found only two kind of approaches:

The first approach isvery basic focused on removing “< s c r i p t>” tags from your html; obviously this is a silly approach, you can inject js code without a “script” tag: you have a bunch of ways to do this by using dom events (onclick, onload etc.).
The second approach is to parse HTML to understand exactly what is happening in the code in order to remove un-allowed tags; this approach is logically the right one, you cannot expect to do better. Your code analyses the HTML, extracts the tags, and then you walk down the tree dropping out unwanted ones. Unluckily this approach requires a HTML parser library really “strong” with respect to malicious code, and usually these libraries are built for a different scope. Another aspect is that parser, lexer and walker are quite complicated pieces of code, so it is not a joke to test them completely. I’ve tested a couple of parsers with unsatisfying results.

This is why we wrote our own sanitizer by hand. Our approach is to remove unwanted tags and properties without testing HTML correctness in deep.

First step is to tokenize the code: a token could be one of : tag start (), comment (), tag content (blah blah), a tag closing ().

For instance test generates three tokens:

test

The tokenize method looks for <…> pairs or comments, and that its fine for our scope of restricting accepted tags: if a <…> pair is badly closed the tag will be html-encoded.

Having the tokens list, we will test every single token whether it is acceptable; again, we do not perform tag matching at first, so for us test its fine, we are working on security, not on syntax – we’ll also fix that afterward, as it is easy in any case to add a tag-pair counter to close at the end unclosed tags, which you actually find done in our code.

We loop for every single token and we test it with regular expressions. The flow is:

if token is a comment discard it.
if token is a start tag ( ) extract the tag (p) and attributes (style=”color:red” align=”center”)
1. if tag is forbidden it will be removed
2. if tag is allowed we will extract every attribute performing a check
 1. check “href” and “src” for admitted tags (a, img, embed only) and check url validity (only http or https)
 2. check “style” attribute looking for “url(…)” parameter, and eventually discarding it
 3. remove every “on…” attribute – e.g. onClick, onLoad, …
 4. encode attribute’ value for unknown ones
 5. push the tag on the stack of open tags
3. else the tag is unknown and will be removed
if the token is an end tag ( ) extract the tag (p) and check if the corresponding tag is already open. Eventually close those that are still open.
else it is not a tag and we will encode it

In order to avoid js injection on user inserted URLs we will accept tag only if the “href” attribute is there and points to a valid URL; we test url correctness with Apache UrlValidator, and this will cut out every “javascript:” tries. Same approach for the and tags.

Finished my hard work coding the shelter, I give the happy news to our design department that the sanitizer was done, and after a first minute of excitement Matteo (http://pupunzi.open-lab.com) told me that they have three different usages of user input: for displaying an HTML page on the front office, for displaying a textual abstract in lists in the backoffice, and of course for storing contents on the database.

So a sanitizer needs three different outputs, html-encoded with tag, text-only without tag, and the “original” version for the database. This is why the latest version of the sanitizer returns “.html”, “.text” and “.val”. Why you should store “.val” instead of the original input or “.html”? Because the original input may be “dangerous”, and may mislead the user in believing that all tags are allowed. The encoded value is not suitable in case of subsequent modification because of double encodings (e.g. “>” –> “>” –> “>” and so on). On the other side “.val” removes only forbidden tags maintaining all other user oddities (strange tags, comments, etc.).

We have set-up a public playground for testing our sanitization code: http://patapage.com/applications/pataPage/site/test/testSanitize.jsp. This page allows you to input a text and by pressing “test” your input will be printed (sanitized) on the page.

Source of our sanitizer are released under MIT license (i.e. free as free beer, just keep the attribution); see the complete code here.

These tags will be accepted, others will be encoded, and printed. If you like challenges try to inject a js in your text and, for instance, get an alert. Tell us about your victories, if any.

BTW, I’ve tested my code using XSS Me plugin for Firefox, and it passed all (about 15o) tests 🙂

How to (not) buy a flight ticket online

I love to travel, my bank account doesn’t; how to move me (and family) for Xmas holiday from here, Florence – Italy to Lanzarote/ Isla la Graciosa – Canary islands – Spain without going in bankrupt?

Easy! Buy a low cost air ticket!

Ryanair is probably the European leader of low cost fly. I have already flown Ryanair in the past with satisfaction (sometime) and some complaints (often) and I had removed their option completely for many years.

But this time I had no choices; Lufthansa, that is usually affordable and with a great site for booking (I have used Lufthansa for my last business travel in Charlotte, USA for a Teamwork www.twproject.com bootcamp), was too expensive.

After a year reading books on interfaces and usability, and actually writing code and interfaces for a new online service (BugsVoice), I was approaching the “purchase” process with new critical eyes.

Ryanair is a leading and growing company, so they know very well how to sell tickets, but it is really necessary to make the process so painful?

I’ll describe what is, IMHO, frustrating in their web interface for buying tickets.

Here my requirement:

“a return fly for 2 adults and 2 children from Florence to Lanzarote starting about December 25 fly back January 6”. That’s all!

Ryanair doesn’t fly from Florence, the nearest airport is Pisa (about 100km far), but from Pisa there aren’t direct fly to Lanzarote.

What all other companies sites do is to allow selection of “from” “to” destinations and then they trace the routes, with necessary stops, without user effort.
No, Ryanair doesn’t. You must “find your path” in a sort of Yoga meditation.

In order to help pilgrims finding the route they supply a nice Silverlight (Silverilight!!!!) dynamic map showing all destination and related links.

Wow! Terrific! I just discover that to go from Pisa to Lanzarote I have to stop somewhere else.

At least 10 different locations are possible: London, Frankfurt, Dusseldorf, Dublin, Bournemouth, Girona, East Midland, Leeds, Liverpool etc., how to check timetables and availability?
Do not be scared, Silverlight it is not so bad, use the powerful tool; just click a location and
a popup will helpfully shows “book flight from here to here”

$C:\tmp\IUCN_044.jpg$

Just click and…. the tool will reveal its limits immediately:

your selection disappears…
(another icing on the cake: why the return date does not propose at least the same date of fly out?)

C’mon, do not chill, go on…

So, in order to check possible routes and schedule you have to select your destination many, many times, just to discover that there is no way to do the complete trip in the same day because of the usage of unusual time slots, and you have to waste a night somewhere. My “somewhere” will be Girona (almost Barcellona, as they say) for one night going and one night returning.

Now you have your routes (actually 2 with a stop) and you are ready to book and pay…
No! there is no way (apparently) to book both routes at once.
Hmm… this may be a dangerous game: remember that Xmas is high season and actually last seats are going off minute by minute. You are risking to book (==pay) first route and then starve in Girona (almost Barcellona), or even worst book the second track without find a way to escape from Pisa (almost Florence).

What can I do? OK! Great idea I open two Firefox instances and with the first one I will buy Pisa-Girona and, in the meanwhile, with the second instance I will buy Girona-Lanzarote.

So you start perky, strong and passionate filling a thirty-two fields form (for each track, 64 in total), you decide to queue with other passengers (actually if you are much Italian than me you can buy a “priority boarding” in order to skip the queue and gain the privilege of watching other passenger fighting for a place, while you are already sitting in the plane, for the risible amount of six Euros each. Oh, I forget to say that Ryanair flights are free-siting, so the “Italian” option get more intriguing), you fill other fields with some loooong selection box for country and for country phone-code (try to propose at least the same please!) both fix and mobile, you swerve to avoid buying insurance, you MUST accept the condition, wait a minute let me see what I’m about to accepting, ugly condition (if you do not believe read here, and have a look to the punishing side fees here ) but this is a low cost company, so, do not complain!

Accept it and continue, click “retain detail for next session” …… but the current one was gone, if you read the condition the session expires, we do not want pernickety customers!

So you start again less perky, strong and passionate, but indeed faster!
Filling the form you can receive a sms message confirming your booking:

One euro? What f*** contract you signed for sending sms? I can find an on-line service for less than 2 euro cent (e.g. http://www.fishtext.com), your margin is only 5000%…
C’mon, do not think go on, press “continue”, swerve and skip again insurance

Again? I’ve already checked the small button “no thanks”, it remember me the joke of electronic voting system in USA where the button “vote for Bush” was moving under your mouse pointer when you try to click somewhere else….

Finally you click “continue” in both browser instances (PSA-GRO and GRO-ACE) and you discover that your idea was not good enough:
your form’s data are roughly shared in session (not a conversation, not a token, not in the client) so your two browser instances shares the session and mix the data. As you can imagine, ‘last win never’ so you have to start again with two different browsers; in this case Firefox for Pisa-Girona, Chrome for Girona-Lanzarote.

Of course the “retain detail for next session” does not work when session expire (damn! nobody there have ever heard about cookies?), so you must re-fill the form even in already used Firefox.

Finally you fill the forms in both browsers and proceed to buying tickets; just choose your payment method.
Considering that you are buying online, there is only your credit card type to choose:

As you can see by selecting a standard visa you will charged by 40 euros, 10 for each passenger, even if you are buying 4 tickets in a single transaction.

Et voilà, a ticket advertised with a price of 29.99 € ( 59.98€ both ways) becomes about 150 €, considering taxes (40 €), boarding (10 €), ONE baggage (30 €), visa commissions (10 €). And if you forget to check in online, you will pay at check in desk 40€ (x passenger)….
But these are only low cost stories…. let’s continue with technical issues.

With two browsers ready to submit I finally pressed “continue” receiving this terrifying popup:

Press OK in “stereo” and BOOM!

I got two different error pages one for each browser stating that there was something wrong in the session, in one side, and a credit card error in the other one.

What was fun was that the “session error” page warn me to wait before re-trying because maybe the transaction was still in progress, and to check the email for a while waiting a confirmation message.

After one hour checking email I re-re-start again the entire process with two browser, then I filled every field, checked every checkbox, confirmed every double confirmable question, and finally, in short sequence, I pushed “OK” buttons in both browser, buying my tickets.

I know that ticketing applications are not trivial, but what could the effort to make a better application? Just look around and you will find better solution even for smaller air player.

Why a so rude company should survive?

Only becouse they sell my tickets for about 1000€ instead of 4000€ of Expedia?