L’arte è pericolosa!

Chissà se Remo Salvadori, l’autore di “Continuo infinito presente”, aveva previsto una simile “evoluzione” per la sua opera.

Passando come ogni mattina tra il duomo e il battistero, finalmente pedonalizzato (almeno dopo le ore 9.30, prima di tale ora è una ridda di furgoni impazziti!), sono rimasto piacevolmente sorpreso nel trovare nel cuore del tempio del Rinascimento un’opera di arte contemporanea. Stentavo a crederci, mancava poco che cadessi dalla bicicletta per lo stupore, a Firenze!

L’opera di Salvadori cinge la colonna di San Zanobi con un intreccio di cavi in un continuum senza inizio ne fine. Bella, grigia sul lastricato lucido di pioggia invernale, un cerchio di materia “nuova” posato sul passato.

Il giorno dopo noto che qualcuno, magari avvezzo al Chianti, o qualche furgoncino impazzito ha urtato l’opera spostandola.

Passa una settimana e ritrovo l’opera immersa in una bussola di cartelli stradali, uno per ogni punto cardinale. Sicuramente qualche solerte amministratore ha rilevato l’enorme pericolo che l’opera poneva al passo e preoccupato dalle terribili conseguenze di un turista rantolante ai piedi del battistero ha pensato bene di segnalarla come fosse una voragine aperta nel manto stradale.

Ma forse sono io che non ho capito l’opera. I cartelli sono il suo necessario complemento, rendono bene l’idea dei nostri tempi in cui la falsa idea della sicurezza ci rende ciechi alla realtà. “Alla maniera d’oggi”!

P.S.: Lunedì 22 Febbrario ore 8:20: l’opera non c’è più, ma i cartelli sono rimasti!

A delicious javascript tagging input field

A brief review of existing components and the presentation of the powerful “jquery.tagInput”

Once again, working on one of our new secret products (which is called Licorize 🙂 ), I was in need to select a third party component for speeding up development: this time, a tagging field.

… before continue reading do you like to go straight to live demo?

I realized in seconds that I already had in mind a delicious solution. Delicious (.com) and its wonderful tagging system its really nearly what I wanted, probably even a little more…

What I really love in this component as seen on Delicious is the smoothness of user interaction. Let me explain the concept: try to write a tag directly from the input field and the drop down will show your already used tags, in order of frequency. Amazing, mainly it saves me scrolling the list: most used on top, golden rule. This is not different from what usually the other components do, but in this case the string searched is a little bit “boldified”. This is stylish!

Another nice feature of delicious’s tagging field is the integration with “recommended” and “popular”: not only by clicking the tag it is inserted/removed like every nerd-programmer will be able to imagine, but even while you are typing in the input field, the suggested tag is magically highlighted – this is a masterpiece!

Of course the interface behavior is supported by meaningful contents: suggested tags are computed on the base of the analysis of the url contents and mainly, I suppose, on other users tagging criteria; having million of users makes the term “statistic” meaningful.

(BTW you can access this data using the Delicious api)

The only things I would change in this jewel is the tag separator from [space] to [comma]; this will give you the capability to add multiple word tags, even if it may induce the user in using the tag field as a sort of notes one. So the separator “flavor” may open a Pandora’s box: if ever in my life I’ll write a tag component – which I’ll try to avoid as much as possible.

Apart from the separator, I needed exactly this component, but unluckily it is entangled in Delicious’ js library, so I decided to have a look at available jQuery based components.

As usual this list does not pretend to be complete as I’m focused on developing products, not on writing reviews, but someone else could find my hints useful (I have already wrote some reviews on charting and grid components).

The top ranked one is jQuery tag suggestion by Remy Sharp.

This nice component is very easy to use and is based on an array of suggested tags that can be supplied via a js array or via Ajax.

A basic axample:

$('input.tagSuggest').tagSuggest({  tags: ['javascript', 'js2', 'js', 'jquery']});

An Ajax example

$('#tags').tagSuggest({  url: '/tag-suggestion'});

Layout is configurable via css, and some parameters will help you in fine tuning this component.

Even if this is a nice component it is quite far from Delicious one, at least because it does not implement multiple tag collection: mine, suggested, popular. Another little limitation is that the suggested tag list is static, even if called through Ajax.

Another cool component is Tagger by Chris Iufer.

Here the focus is on maintaining a tag list, instead of filling a field.

Usage is really simple:

<input class=”tagger” type=”text” name=”tags[]” />

<script type=”text/javascript”>

$(document).ready(function(){

// add to tagger element by index

$(‘.tagger’).eq(0).addTag(‘photo, design, creative’);

// add to a specific tagger element by Id

$(‘#places’).addTag(‘mountain view’);

$(‘#places’).addTag(‘chico’);

});

</script>

But this is even farther from my goal…

Even Pines Tags by Hunter Perrin goes in the wrong direction, at least compared with Delicious. It is focused mainly on writing / removing tags (like the previous one):

The documentation is little bit weak but usage is still easy:

$(“#tags”).ptags();

In the same flow is JTAG by Benoît Vidis

cute, but not my prefered one.

Another component that cought my attention was MagicTags by Dave Geddes.

Nice graphics, but here the core seems the transitions effects, making the usage of this component at bit annoying.

Nice look, but similar behavior for TestBoxList by Guillermo Rauch.

That’s all folks! I didn’t find other components, maybe the “tag” search key induces too much noise in search engines, but differently from usual there aren’t dozens of components, the choice is limited.

So I reached this sad conclusion:

I MUST WRITE MY OWN COMPONENT!

At least now my requirements are clear:

supporting already used tag list, eventually filled with an Ajax call
filtering results using user inputs
supporting suggested tags, with “smart” selection
supporting multiple separators
supporting frequency
having a nice-ligth-stylish look
be delicious!

After a couple of days of programming this is the result:

typing on the text field a drop down will show your used tags:

I’ve set up a live demo on BugVoice site (new secret product is offline for the moment).

The usage is quite easy at least in the minimal configuration:

<link rel=stylesheet href="jquery.tagInput.css" type="text/css">
<script type="text/javascript" src="jquery.tagInput.js"></script>

insert your input field:

tags: <input type="text" size="80" id="tag" >

then instantiate the component:

<script type="text/javascript">
var myTags=[
  {tag:"js",freq:30},{tag:"jquery",freq:25},
  {tag:"pojo",freq:10},{tag:"agile",freq:4},
  {tag:"blog",freq:3},{tag:"canvas",freq:8},
  {tag:"dialog",freq:3},{tag:"excel",freq:4} ]   $("#tag").tagInput({ tags:myTags})
</script>

This is just a static case example; in case you have thousands of tags, you should use something like a Json Ajax call.

There are some configuration parameters, this is the complete list:

tags: is a Json array of objects with “tag” and “freq” properties; freq is not mandatory if you do not use it for sorting. It is not mandatory if jsonUrl is specified.
jsonUrl: will receive the text of current tag typed in the “search” parameter.
autoFilter: true/false default=true when active shows only matching tags, “false” should be used for server-side filtering
autoStart: true/false default=false when active the dropdown will appear straight on entering the field, otherwise only while typing
sortBy: “frequency”|”tag”|”none” default=”tag”. When sorting by frequency is on, the filler should report the “freq” property for each tag
tagSeparator: default=”,” any separator char as space, comma, semicolon
boldify: true/false default true “boldify” the matching part of tag in the dropdown box
suggestedTags: callback function to retrieve an object array like [“sugestedtag1″,”sugestedtag2″,”suggestedtag3”].If this parameter is not present tag suggestion functionality will be kept hidden.
suggestedTagsPlaceHolder: jQuery proxy for suggested tag placeholder. If it is not supplied it is created below the input field. When placeholder is supplied (hence unique), tagField should be applied on a single input.Something like $(“#myTagFiled”).tagField(…) will work fine: $(“:text”).tagField(…) probably not!

so a more complex example will look like:

<link rel=stylesheet href="jquery.tagInput.css" type="text/css">
<script type="text/javascript" src="jquery.tagInput.js"></script> 

tags: <input type="text" size="80" id="tag" >
suggested tags: <span id="suggested" class="tagInputSuggestedTagList"></span> 

<script type="text/javascript">
  $(function(){
    $("#tag").tagInput({
      jsonUrl:"tags.jsp",
      sortBy:"frequency",
      suggestedTags:  ["jquery","tagging","tag","component"],
      tagSeparator:" ",
      autoFilter:false,
      autoStart:false,
      boldify:true,
      suggestedTagsPlaceHolder:$("#suggested")
    })
})
</script>

in this case the jsonUrl “tags.jsp” is a silly JSP page that will return 4 tags starting with the text typed by the user. Here the sample code:

<%@ page import="net.sf.json.JSONArray, net.sf.json.JSONObject" %><%
  JSONArray jsa= new JSONArray();
  String search=request.getParameter("search")
  search=null?"":search;
  for (int i=1;i<5;i++){
    JSONObject o= new JSONObject();
    o.element("tag",search+i);
    o.element("freq",i);
    jsa.add(o);
  }
  out.print(jsa.toString());
%>

This component is released under MIT license. Enjoy it!

Any feedback will be really appreciated.

JavaScript grid editor: I want to be Excel

A short list of my favorite JavaScript grid components.

How many times did you hear users asking you: “something simple, a grid like excel”?

When I was a VB programmer, oh yes, I have this dark spot on my career, and it is not the only one… this request threw me in panic. Usually what your “killer” is asking you is not what you, programmer, are thinking of (namely the power of cell functions, programmability, graph etc., that probably your “killer” does not even imagine): what they are thinking about is the editor flexibility, the ability to add columns, rows, move cells blocks, copy and paste from different sources.

After the ritual pointing-out that your application is NOT Excel, the VB solution was to adopt the standard flexGrid or the mythical TrueDbGrid that made happy both the killer-user and the victim-programmer.

But my VB era is long gone and nowadays I’m fighting with JavaScript and Java web applications; how to address the same old request of “something like Excel”?

I spent some days looking around for a solution. I warn you that this post does not pretend to be a complete list of available grid/table editors and that my investigation is mainly “emotional”: I want to feel on a web-app the same prickle that I experience when using Excel.

Of course it is not only romantic surfing, I needed to edit data to fill the graph snipplet for Patapage, so there are some “nice to have” requirements: to be free of charge, light, based on jQuery, appealing. Server side aspects as pagination, sorting, CRUD etc. are for the moment not the crucial points as the amount of information I need to manage is limited and a totally client side solution would be acceptable.

Actually the implementations I found can be classified in three main groups:

data driven: these are generally components with all the features needed for listing a large number for rows, with server-side pagination, scrolling, search and sort functionality; generally are lacking editing functionality, or when the feature is present is single row based, reflecting the fact that there is an underlying database. Data binding is made through XML or Json, depending the language supported server-side
light edit: here the focus is given to the “editing agility”, every row is always in editing status, search and sort are rarely available and in any case working on a limited set of rows isn’t a real limitation. Data is filled starting from a JS object, calling methods, or from an html table. Some of these implementation supply callback methods that can be used to interact server side.
spreadsheets: oh yes! there are also some really sophisticated JS clones of the glorious one! These are generally complex solution (in terms of dependencies), supporting formulas, graphs and lot of functionalities. Here the focus is the “simulation effect”, data management and server side integration are really in background.

When I started my search I didn’t imagine that there were so many solutions, so I decided that it would be better here to limit the result to the ones based on jQuery. Lets go in depth :

Data Driven

One great example of this category is jqGrid by Tony Tomov. This is a really well engineered solution released under MIT and GPL licenses.

Supports pagination, search, sort and even edit. There are two specific versions for PHP and .NET that drastically reduce the server-side implementation effort. Loading data is supported in multiple ways through XML, JSON or JS arrays and examples are supplied for PHP and MySql. There is a third party Ruby example here.

It uses jQuery.UI components for display, so the look and feel is pretty. Switching to edit mode is something perceivable, the user will see that the edit is active, but this is not necessarily bad in this class of components.

It supports sub-grids, multiple input types, master-detail, multi-selections and other useful features.
Column configuration is done in a declarative way using JS objects:

colNames:['Inv No','Date', 'Client', 'Amount'],
colModel:[ {
  name:'id',
  index:'id',
  width:55,
  editable:false,
  editoptions:{readonly:true,size:10}
},…

that gives you the flexibility you need for well tailored applications.

Seems a rock solid components, the behavior in resizing, scrolling, paging is really pleasant.

Documentation is not the strongest point but there are lots of examples that will help setting up your grid.

by Allan Jardine released under GPL v2 and BSD is a powerful data grid component.

Supports pagination, resizing, filtering, multi-column sorting, jQuery UI theme roller, and lots of interesting features.

Data feeding is provided a-la-carte, from DOM, Ajax XML or JSON sources, JS arrays… it’s your choice!

There is also the icing on the cake: cells and rows grouping, cell custom renderer, multi-row selection and much more.

Provides cell editing via jEditable cell-by-cell, not a row at once, and this in some cases could be a limitation; so you may need more that one update to change a row. Probably you can work around this trivia-point.

Configurations starts from zero-config in case of DOM feeding and can be refined to a real full-control using JS objects.

It is well-documented with a large set of examples and this is another plus!

A black spot (at least for me, Java programmer): server side examples in PHP only.

Another valid example is Flexgrid by

What I really appreciated in this component is the lightness and its simplest feeding example based on an html table. In this case the whole setup consists of:

$(“#mytable”).flexigrid();

Of course this is a quite minimalist approach to this component that supports also XML and JSON data feeding.

Paging, sorting, resizing and other nice features like button injection are supported.

What is missing is the edit capability that is planned but still not released.

Similarly to the previous component the configuration (if needed 😉 ) is done by JS objects.

Documentation: we all are programmers, we hate to write it but we love to read it… so here too is really succinct with few examples for PHP server-side both XML and JSON.

tgrid by Mateusz Mikołajczyk is a lightweight datagrid, that supports all main features like sorting, paging, filtering.

I didn’t discover the license type so be careful on usage.

Data is fed by Ajax Json calls that make this component scalable enough.

Documentation supplied is “short” but with some examples, so setup is not a big deal.

This components supports a “row rough” edit, in the sense that it is not in-place, but out of the grid: it did not give me the right feeling. Taken out the edit, tgrid runs well.

Configuration does not offer many options; something nice is the cell custom renderer that allows you to format data.

Ingrid (great name!) is a less powerful grid component, but it is really unobtrusive and cute enough to be considered. Released under GPL license, it supports column resizing, sorting, paging etc.

Data is fed only by providing an html table, and this could find some criticism.

Google Chrome support has some little creeps on display.

Multiple row selection is a nice feature that earns some points to this component.

Editing is not supported.

In the same spirit of loveliness jTps by Jim Palmer is released under MIT license.

Paging, resizing, sorting are supported; a nice feature is smooth scrolling. Even in this case editing is not supported, but the real limit of this component is in term of scalability: data feeding does not support Ajax sub-sequential loading, so all data is loaded at once. Paging is only client side limiting the meaning of this feature.

Same category for dhtmlxGrid by DHTMLX.

dhtmlxGrid is a JavaScript grid component with editing capabilities, and server-side data binding. It is a powerful component with so much advanced features (such as filtering, searching, grouping, charts integration, math formulas, etc.) that it could be included in “Spreadsheet” group too 🙂

dhtmlxGrid is distributed under GNU GPL v2, but also available under a commercial license.

It is built over a “dhtmlxcommon” js library instead of most famous jQuery or Prototye libraries, but this is not a big deal if you need such powerful component.

For the sake of completeness I have to cite TableEditor by Brice Burgess that seems a little bit outdated (even if really well indexed), and Snowcore’ jQuery Data Grid that is in Russian and this fact did not help me to understand how to use it; moreover the site icon let’s me suppose that IE testing is not a priority for Snowcore 🙂

Light Edit

This category, as before stated, is mainly focused on edit agility instead of server-side data management. This fact steers the implementation towards a all-client-side approach. Implementations in this category are generally a little bit rough with respect to the previous ones.

A real interesting exception is the excellent SlickGrid by Michael Leibman released under MIT style license.

Here the author is particularly careful in rendering performance; he uses a virtual rendering technique to limit the number of DOM elements in order to make SlickGrid really scalable. The example provided with 50000 rows really run in a flash.

The author states in the “grid vs data” paragraph that here the focus is on the grid, not on the data. Despite his intentions, sorting, resizing, filtering, resizing and edit callbacks, give this grid a place even in in the previous category.

Data feed can be done via JS arrays, or via Ajax using the Slick.Data.RemoteModel object provided.

Custom cell renderer allows you to express your creativity, as shown explicitly in the examples when injecting a Sparkline graph in the last column (see my previous post for a excursus on charting components).

Rows multi-selection, ordering, and custom row rendering using John Resig’s micro-templates makes this component really flexible.

Configuration is done as usual using JS objects.

Having all this features and also having taken care of cell movements with the keyboard and easy editing is a great achievement! Thanks Michael.

Once again Allan Jardine provides us a useful tool called keytable that injects Excel-like key movement on a generic table.

This can have multiple usages as exemplified on demos available on the site. It easy to understand how a key-movement can change the user experience when it is used on a powerful grid component like DataTables (see above). Here the author’ example.

Another really basic example of editable table is supplied by Greg Weber with uiTableEdit: there are no examples online, so I built a simple test page, and the components just add the capability to change the cell value. No key movement or other additional features are handled; it can be used as starting point for further refinement.

Same kind of “roughness” for tEditable by Josh Hundley, where edit is click based, and key-movement is not supported. This do not mean that this tools is useless, just that it is really far from a complete solution.

It converts DOM cells to text-areas on the fly when editing, and this makes this component almost setup-free.

A part of this category (the rough part 🙂 ) is my contribution, used to fill data on PataPage’ charting playground.

Here the functionality are really basic: key-movement, edit, load from url, add and remove only are supported. The grid content is “saved” on a text area with \t and \n separators: same format expected when loading from url. I’m working for supporting the ^v (paste) capability to facilitate user data insertion as block of cells.

Spreadsheet

Actually this category comprises solution wider than previous ones, in the sense that a complete solution will probably include all the features listed before; this is on one side a plus, but on the other side if you are interested in extracting some parts (e.g. table edit but not on formulas, server-side integration but not on key-movement etc.), this could be a little difficult due to the complexity of the solutions available.

Light years beyond other solutions at least as first impression, jQuery.sheet by Robert Plummer is a really wonderful library. I didn’t find the license type.

Really looks like an Excel sheet and the key-movement seems realistic.

Data feeding is done using a DOM table, eventually Ajax-loaded, or by adding row and cols programmatically.

The grid supports resizing, but the real wondering feature is the cell-function support. Not limited to SUM, AVG and other standard functions, but also chart are allowed.

Lots of callback will help you to integrate the sheet on you application.

There are some little bugs on cell selection and some minor layout issue with Chrome, but the overall behavior is impressive.

The last, but definitively not least, example of websheet (a web spreadsheet) is the Googledocs one. It is a really good live-teaching example, and even if it is not a ready-to-use library I suggest you to open it and dissect it with, for instance, Firebug.

It is interesting to see how many tricks they used: the whole grid is one iframe, the rows are grouped in small tables of 20 rows (probably in order to scale on IE), the cell value is stored in the td dom; when you edit a cell a text area is printed over the cell (not inside), and so on.

I tried to implement my grid using the same structure, but after some hours I surrendered; the effort to “master” the layout was over my actual estimation, and I fell back to the solution discussed above. Would be gorgeous to have a jQuery open-source implementation like this!

Google docs spreadsheet is the only application I found that gave me the right prickle!

Well I stopped my searching here, and I didn’t explore Mootools, Sriptacolous, Ext and maverick ones in detail.

There is a booklet for this blog post here on licorize.com. (Licorize is one of my latest creations, please take a look)

I’ll just list here some other libraries I found:

Spreadsheet Dojo Widget

Ext JS DataGrid

YUI 2 DataTable

Simple Spreadsheet

BlueShoes SpSpeadsheet editor

TrimSpreadsheet

Sigma Grid 2.2

30 years riding IT: part3

1983

Technology was running fast in that era, and just when I started studying Synthetic Programming (a way to use some more non documented function in your rpn code for HP41cv), (see previous blog) Massimo bought, in a Suisse shop, a terrific new programmable calculator the Sharp PC-1500, shouting me in panic.

Sharp PC-1500 was, in my mind, the best technology the humanity could do in that years.

PC-1500 have a real-shaped keyboard, 4Kb RAM, a dot addressable display (7×156 dot LCD), a plotter (not included) and a “standard” programming language that do not force me to think like a polish men 🙂 and “she” talks BASIC!

After two shop painted, three days as HiFi expert for Saba stand in a local electronic event, and selling of my “old” HP 41cv, I got enough money to take a train, go to Suisse and buy my “new” PC-1500.

In the meanwhile, my studies going slowly. Chemistry, Geometry, Physics, Design was interesting, but I spent a lot of days on my PC-1500, looking a way to make enough money to have the freedom to do what I really love: travel around the world taking pictures.

But my parents was more realistic than me and: “no exams? no money!”, so I enter a loop where I play with PC-1500 -> no time to study -> no exams passed -> no money from parents -> find a work to get money -> spend money in traveling -> study just a little -> pass an exam -> write a lot of code -> do money …. and so on.

This style-of-life hang me on Engineering faculty for about 14 years :-).

Back to PC1500:

Sharp PC1500 was a very well done product (was the “porting” of the Tandy/Radio Shack TRS-80 PC-2.), so was tested and robust.

PC-1500 with its printer/plotter was able to record the programs on audio cassettes. In this way we could save our production.

But the life it is not only work! Massimo and me were used to spend spare time playing the latest video game: asteroids! There we started, probably, the first experiment with “pair programming”: one to the joystick, the other one to the buttons. Sometime we get the hi-score and as there was only 3 char to write the name we create our brand: R&M.

With this brand we signed about 150 projects and product.

Back to PC1500 again:

The strongest exam on the first year for an Engineer was Mathematics I. Function studies, integral, series, and son on. Sometime have an idea on what you have to study for you exam could be helpful. So we start a production of mathematical software able to:

Plot your function on PC-1500 plotter (plotting during the exam is a little bit noisy, but you can use your coat, even in July with 30°, to cover the plotter), its first and second derived.
Know a-priori if a series on integral converge or diverge.
Know a-priori if your function can have zeros

With this tools, the written part of Mathematics I examination was like a joke.

Writing this software I learned a lot about basic, number, precision etc. So we were ready to start something more professional.

The “Medicina II” an University’s Clinic was looking for someone able to translate an old statistical program to its newest Apple IIe.

Software that we are called to translate was written “on paper” using a graphical notation and typed (or recalled) on a strange Olivetti Programma 101 “computer”

The work was amazing but we learnt for the first time that the BASIC is not exactly the same BASIC everywhere; and the calculus too, can be slightly different.

LESSON 1: STANDARDS ARE STANDARDS, REAL WORLD NOT!

But that was the very first “paid-software” we did.

Obviously at that time the code was to keep “secret” and we used several tricks to protect our production (control char like ^L in the source code, remove commands etc.).

The hungry of knowledge brought us explore wider territory.

I bought in Paris a funny book “Graphism Scientific pour l’ordinateur personnel” and we started new graphic experiment on pc1500’s plotter that was really not enough.

We discover in a dark side of S.Marta (S. Marta is were Engineering Faculty is located in Florence. It is was a big old monastery with a kind of dungeon underground) a terminal connected to a PDP server. It was definitively forbidden for newbies like us, but after some days of circuiting the administrator we get a limited access to read the on-line manual. But the manual was for “standard” user and we need something more, so we get a copy of administrator manual and we started to experimenting new powerful command. Unluckily something went wrong and we launched a process that write something in the “on-paper console” without a way to stop it 🙁 . All this happened in the late evening with nobody near the printer and the printer well locked behind a strong door.

The day after, 1 kilometer of paper was finished and our first PDP experience too.

Working on PC-1500 was really less dangerous!

The AI (artificial intelligence) is (or was…) an amazing science and we read something about experts systems, and that Lisp was the best language to implement it.

The mythical Texas Instruments’ Lisp machine was light year far for our dreams, and we have to arrange something less expensive.

We get some BASIC code for a Lisp interpreter written by I-don’t-remember-which University and we decide to “port” it to PC-1500. The article was full of theoretical induction about Lisp language, and the porting went quite smoothly.

The performances was not excellent, minutes to evaluate “(+ 1 2 3 4)”, but the experiment was interesting at least because give use the chance to realize something based on strong theory and to learn something about Lisp. Our attempt to create an expert system sunk on the performance beach.

Not completely happy with Lisp interpreter experience we decide to develop a BASIC “expert system” trained in mushroom identification.

In our expert system experience we discover, for the first time that, sometime, a simple and direct solution could be more and more effective than a solution “based on strong theoretical fundaments”.

Our expert system was used and trained with fun for years by a mycological group in Florence: the Lisp one, died before to bloom.

LESSON 2: SOMETHING RUNNING MAKES USER HAPPIER THAN A THEORY

After AI we go back studying “graphism scientific”; reading French computer’s book is really funny experience. I suggest to taste “la mémoire vive de votre ordinateur”.

Another source of inspiration was the french magazine “Sharpentiers” (that sound like “carpenter” that reflects quite well the “hand made” style of that years) dedicated to Sharp users.

In some week we develop a “3D wireframe CAD” for PC-1500 (do you know that for rotate a 3D object the algorithms uses the matrix? We do not! But we learned it some years late in our study, when the problem was completely solved: sometime studies are useful!).

The CAD was funny and even if data insertion for complex drawing was a nightmare, an Architect’s studio want to use it. So we “port” our Cad to the very best of Sharp: MZ-700.

They pay a lot of money, for that time our standard: 170.000 ITL (about 85€ )

But the luck smiles to the brave, and in a Florence book store I found a strange blue book: the “PC 1500 Technical Reference Manual” was at that time in Italy one of the most wanted book for PC-1500 owner.

This manual opens the hardware and software architecture to our hands.

Sharing that information with our friend (share doesn’t necessary means “for free”, so this manual became a great business) we meet a large group of PC-1500 users.

There we discovered that one of pins in the PC-1500 connector, was suitable for one-to-one (probably now “pear-to-pear” sounds better) communication. First of all we need a 60pin half pass connector, but it was not available in almost all electronics’ shop in Italy.

Florence is the artisan’s mainland, and with the patience of our ancestors I build my 60pinhalfpass connector. Was really disgusting to see, but was working.

In the meanwhile the German PC-1500’s user group will develop the Macro assembler compiler and a Forth interpreter. We bought both.

With Technical manual, macro assembler, and cable we are ready to start the first 1-to-1 communication software. Obviously the only way we can use to communicate is to open/close a single port. It is not so easy to use one port for bidirectional communication, and we invented our protocol. The unexpected, sometime mystical, behavior force us to learn about timing, clock cycle, slopes, micro-instruction length, and all thing that make me happy to develop now in java without knowing anything about hardware and operating system too.

But in the end the communication software was very well running, and very useful to communicate with Massimo during one of infinite Engineering exams.

http://www.pc1500.com/

JavaScript charting tools: an overview

Tons of charting tools, one better than the other, how to find yours?

I was working on adding a new widget for Patapage. Sometime you need to publish a simple graphical report on your page, a chart, based on a small set of data that you may want to change quickly online.

As requirements, I wanted to have an immediate feedback of data and configuration changes so I chose a JavaScript charting tool instead of server-side generating “chart images”.

Few years ago this approach was almost impossible due to the lack of cross-browser plotting libraries. Nowadays there are many powerful plotting tools: this article will introduce some of them and will explain how I got to choose my plotting library.

First of all I had to restrict the selection to those that are licensed for usage in a commercial products, but this is not a big deal; most of them are under LGPL, or MIT license.

My second requirement was the compatibility of the JavaScript library: we use jQuery so I had to drop the great plotr library that is based on prototype.

Actually Solutoire has also a Prototype porting of flot called Flotr.

Same technology for Protochart. For the same reason I quit in tears the amazing jsxgraph. What I loved of this library is the universe of interactive examples provided by the community. If you do not have library restrictions have a in-depth look to this jewel (if it doesn’t fit, give yourself at least a recreation playing for few minutes with the examples 🙂 ).

Also Raphael is a real impressive product, but even in this case it doesn’t relay on a “standard” JavaScript library, and mainly it is a low level vector library, in the sense that it doesn’t supply any specific chart plotting functions. If I had to write a graph library from scratch I’ll probably start from here.

MooTools user will have on their side moochart or canvas pie.

For jQuery fanatics, like me, there are lots of bullets too.

First I started having a look at the flot library. I was astonished to see how well engineered this library is . IMHO one of key point is that the whole configuration is defined by using a JavaScript object, so a basic example that will produce the cart on right side will look like:

<script>$(function () {    var d1 = [];
    for (var i = 0; i < 14; i += 0.5)
        d1.push([i, Math.sin(i)]);
    var d2 = [[0, 3], [4, 8], [8, 5], [9, 13]];
    // a null signifies separate line segments
    var d3 = [[0, 12], [7, 12], null, [7, 2.5], [12, 2.5]];

    $.plot($("#placeholder"), [ d1, d2, d3 ]);
});
</script>

This approach is very practical when you work with json objects, especially for data series. Another nice aspect of this library is that series configuration and data are on the same object. Of course when you need to use advanced charting features (time series, label formats. ticks etc.) the configuration will become quickly complex and not really readable, but on the other hand the basic approach is straight and painless. Unluckily flot at this moment does not implements pie renderers (to be honest there are some implementation from the community – see flot issues), and this was one of my widget requirements. Another little spot is the documentation that is complete but really ‘60 stylish. Even the examples are a bit poor but enough for starting quickly.

Then i moved to Sparklines that is a very easy-to-use and intriguing library; I like their site with lots of small charts. If you need to create dashboards for your site have a look at this plugin. Even in this case configuration and data uses a JavaScript object.

Sparklines was a little basic for my needs so I tested TufteGraph and jQuery Chart; I found both lacking in features.

Then I landed in the jQuery Visualize Plugin from Filament group; they have a really smart idea; leave the data on a HTML table!

So the data will look like

then the configuration is done as usual by passing a JavaScript configuration object to the drawer function. On their site there is a nice sandbox to try configuring your chart.

If you already have table displaying data just add a line of code to have your chart.

Of course this is a completely different approach from flot, where to have the same visual effect you must write tons of lines; on the other hand jQuery Visualize does not give you complete configurability, but only reasonable ones 🙂 . In most cases this could be an effective approach especially if you do not like to spend hours with JavaScript objects and arrays.

I didn’t stops at this station. Next stop jqPlot!

jqPlot is a really well engineered solution, in some sense it has an approach similar to flot, but it has more plotting types (pies too!), a great documentation online and a lot of examples.

Here both data and configuration are JavaScript objects but are separated, two parameters on the drawing calls.

I appreciated the “default renderer” configuration so you do not need to write too much in case of multiple data series (unlike flot).

The other key factor of this library is the complete pluggability of the renderer; you can have renderers for axis, labels, ticks etc. This approach for instance makes it possible to have labels rendered by canvas:

The separation between data and configuration helped me to realize my Patapage widget.

I made an editable table that refreshes on-the-fly chart values and a set of options to change the graph layout. Both data and configuration are serialized, so to be easily stored server-side.

I have set up a playground in order to test my Graph component, that covers only few of jqPlot features, but should give you an idea about this component.

For the sake of completeness I have to cite Google Chart API and Yahoo YUI Chart Controls, that require to bind tightly with these too small players 🙂 . No thanks, next time maybe.

well engineered

Hallways user testing: the crow on the shoulder.

Every software-product developer should be familiar with the milestone-book "Don’t make me think" from Steve Krug, and probably lots of us read it.

I did, and I agreed!

This happened some years ago…

Nowadays my days runs faster, and it may happen that old precious hints get lost.

Some days ago, I called Massimo, an expert .NET developer, for testing our new product BugsVoice, but secretly to push it to develop, for free, an example of .NET page for catching errors (please don’t tell him).

He sat down in front of me (see our space here ) and started to enroll in BugsVoice.

I was peeping him over the monitors barrier watching his face struggling, and then I

remembered Steve’ rule, that actually is the all-month tips…

So I moved around the table and I sat behind him, really close, looking over the shoulder , to see both monitor and keyboard, like a crow on the shoulder…

… and the magic happens!

I discovered in a half hour session that some parts of our application are hard to understand or completely unclear.

We have already used, with satisfaction, professional testing service like UserTesting for our products; they usually provide you a video file recording user voice and screen interaction, but seeing a “human” (are programmers human?) face give you an emotional feedback that you will miss in video files.

Some hints to get the best from user testing:

give the tester a small overview of your product. Do not explain your product in detail, give only the context. Prepare a short story board: I gave my tester the following tasks: enroll, choose an error template, configure your server, collect some errors.
invite the tester not to be shy and talk while playing with your app. Explain that it is not a competition, and that you are testing the application not the tester 🙂
resist to interfere. Do not help your tester and take notes when the tester stops. Eventually hearten him and ask what’s the matter.
look at the mouse movements. If the pointer goes around the screen without a goal, probably there is something too small, in a bad place, or too far. Take notes!
look at the tester’s face. When you see self-satisfaction (seems that he/she is understanding something) ask what he/she thinks that feature is. Take notes!
let the tester play whith you app discovering features. Take note of the path the tester follows to reach the goal. Is he using different/unplanned ways? Probably there is something to be done on the interface!
look at the clock. Usually a “real” user does not give you more than few minutes before giving up. It is valuable to reach at least a rewarding step in the first minutes.
at the session end ask the tester an overall impression of your application. Ask what he feels about each phase (enroll, step 1 etc.) and take notes
then ask the tester some suggestions to refine your app. Take notes, but then throw them away, you already have a lots of better feedback.

In our case a thirty minute session gave me five A4 pages full of interesting notes.

What I wonder about user testing is that very often what misleads the user is a wrong message, a color in a title, a misplaced button, a hidden something; all things that you can fix in minutes and usually every programmer tends to underestimate. Of course sometimes you may get in serious trouble when you miss a core point; this is why it is crucial to start user testing very early. I know that it is sometimes hard to do, how to test an application if a core functionality is missing?

We started user testing quite late with BugsVoice alpha (internally), and we are continuing with the public beta.

Give yourself the opportunity of a day as a crow on the shoulder, you won’t regret it.

Solving the “halting problem”…

When I asked Gino (alias Roberto Baldi mostmobbed) a software solution for the “halting problem”, he told me “should not be so difficult”!

“In computability theory, the halting problem is a decision problem which can be stated as follows: given a description of a program and a finite input, decide whether the program finishes running or will run forever, given that input.” (from Wikipedia)

Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist; but as Gino says, Turing was a pessimist…

Why I’m scratching about the halting problem? Because I need power at BugsVoice user’s fingertips. In particular, I need to allow the execution of user’s JavaScript code on my server, and knowing if these scripts end may be comforting.

If you already heard about XSS you probably know that “third party” code execution, authorized or not, could be a nightmare even in the client… can you imagine how can be ugly on your server a “pirate” code?

Going more in detail, BugsVoice is a service that receives a “bug” from a customer’s server, process the request locally, serves a friendly feedback to the user and stores the bug in its database.

JavaScript (JS from here on) server-side execution is involved in the “processing” phase.

We supply a pre-filled and certified set of “rules” for processing bugs, but we even allow customers to create their own rules.

JS gives you the power of inspecting the error to understand what happened and gives your customer an error better than a “500 server error” in order to comfort it and recover a situation where your application credibility is going down. An interesting reading about error recovering and error feedback is the book “Defensive design for the web” from 37 Signal.

The complete BugsVoice process includes mainly three parts:

1) on the customer’s server side, an error page that catches the exception, collects as much information as possible (logged user, time, server status, database status, memory etc.) and redirects the user to our BugsVoice server (see how to configure an error trapping page on BugsVoice blog for more details).

2) our server, reading user preferences recovers the error template. Each template is fully dynamical and customizable; it introduces some “variables” that can be filled from the error happened. Then our server creates two JS objects: the “bug” object filled with the error collected and the “template” object filled from layout skeleton.

3) the JS rules are executed to fill “template” from “bug” or for rejecting the request.

4) the layout is rendered to the user by using “template” and “bug” objects. The bug is stored on our server.

5) the user feedback is collected and stored.

6) a “thank you” page is displayed to the user.

Then there is the error management but this is interesting “only” for BugsVoice’ users, not for this post. Here some error pages from BugsVoice:

So every user can create its own rules in order to inspect, for instance, the received bug’ stacktrace trying to discover if a database problem happens, or if there is a problem with the latest version of some browser.

Coming back to rules execution:
during step 3) we get rules from the user configuration and we execute them on our server. We use the Java SE 6 scripting features supplying an ECMAScript engine to run rules. A scripting engine instance is isolated from the JVM environment and you must declare the resource (libraries) you want to made available in the execution context.

Before executing them, the context is fed by “bug” and “template” objects. Then we run the rules…(drum roll!).

A basic (and friendly) rule example :

if (bug.code==404)  errorPage.errorMessage="Page missing: you get this error because of...";

Of course this code is safe, but what happens if an evil user composes a pleasant rule like

while(true);

or

function snake(s){
  return "s"+snake(s);
}
snake(":-<");

… or even worst?

Sadly Turing beats Gino 1-0, and there is no general solution to the question “does this rule ends?”.

The only possible solution is to narrow the scope of the problem by introducing some fences.

A solution is to set up an external observer using multi-threading and watch dogs in order to kill processes after a while, but the best solution is to avoid infinite loop situations.

Rules in our context are used mainly for discovering string patterns in the error stacktrace and for building better feedback; we do not need to iterate or create complex functions, so reducing the set of possible JS statement is possible without loosing “power”.

Luckily in JS there is a limited set of statements for iteration and recursion; so if we are able to “kill” bad intentions by forbidding dangerous statement like “while”, “for” or function definition we can run rules with confidence.

This way we reduce the complex halting problem to the (quite) easy problem of HTML sanitization (where you must remove some unaccepted tags. See XSS war: a Java HTML sanitizer ).

Actually identifying “while” or “for” statements in a complex code is not as easy as finding the “while” string. The find/replace approach it’s too rough, and here we need a more accurate solution in order to understand the difference between

while (true);

and

var dummy= “while(true)”;

that is obvious for us but not for a string searcher…

You must use something to analyze the code token by token.

ANTLR 3 supply all you need for tokenizing, parsing and walking your code. You need a JS grammar and then ANTLR will build all the stuff. We used the ES3 grammar from Xebic Reasearch (BSD license) based on the original work of Patrick Hulsmeyer, that fits perfectly our needs.

With the AS3 grammar we built parser, lexer and walker to analyze rule’s code to intercept every forbidden statement and avoid accepting dangerous scripts (at least I hope this). Only the rules that pass the test will be saved on the system and will be available to the script engine.

Ok, I can confess you, the post’ title is a little misleading, there is no way to solve the halting problem at least without cheating!

A jQuery text extractor (via Java proxy)

Developing online services seems always to bring about new challenges: yesterday’ problem was to extract plain text from a generic web page.

Using jQuery you can easily extract text from an element by using $(..).text(), so I wanted to put this function to use; the problem then was to make the entire web page available to jQuery.

Before extracting text, you should at first load the content of the intended page somewhere; if you want to preserve the integrity of your calling page, the only suitable solution is to use an iframe as container:

here are the html basic elements:

  <input type="text" name="FILLFROMHERE" >
  <span onclick="fillFromURL()"></span>
  <iframe id="pageExample" style="display:none;"></iframe>

In order to fill the iframe:

function fillSpamFromURL(){
  var url=$("#FILLFROMHERE").val();
  if (url && url!=""){
    $("#pageExample").attr('src', encodeURIComponent(url));
  }
}

You should wait until the iframe loads the page before starting the text extraction; bind the "load" event on the iframe and it will be raised at the right time:

  $(function(){     $("#pageExample").load(extractInfos);   })

At this point you can access the iframe content by

$(this).contents()

Sadly I discovered what probably I should have known already: cross site DOM inspection is forbidden (see Cross-Domain Communication with IFrames).

To work around this limitation I wrote a really simple ". jsp" proxy page in order to load the "external" page in my domain:

<%@ page import="java.io.InputStream,
                 org.apache.commons.httpclient.*,
                 org.apache.commons.httpclient.methods.GetMethod,
                 java.io.InputStreamReader, java.io.BufferedReader" %><%
  String url=request.getParameter("url");
  HttpClient client = new HttpClient();
  HttpMethod method = new GetMethod(url);
  client.executeMethod(method);
  InputStream bodyAsStream = method.getResponseBodyAsStream();
  StringBuffer sb = new StringBuffer();
  InputStreamReader streamReader = new InputStreamReader(bodyAsStream, "UTF-8");
  BufferedReader reader = new BufferedReader(streamReader);
  while (true) {
    int cr = reader.read();
    if (cr < 0)
      break;
    sb.append((char) cr);
  }
  String thePage= sb.toString();
  bodyAsStream.close();
  method.releaseConnection();
%><%=thePage%>

Now the iframe load function should be changed to :

$("#pageExample").attr('src', "proxy.jsp?url="+encodeURIComponent(url));

Warning: the proxied "page" can’t load relative resources, so this solution may not be a suitable for everyone.

The jQuery .text() function will retrieve text even in the script tags, so you can use a little trick to remove unwanted tags:

function extractInfos(){
  var ifraBody = $(this).contents().find("body");
  ifraBody.find("script,style,object,link,embed").remove();
  var text=ifraBody.text()+"";
  var re = new RegExp("(\\s){2,}", "g");
  text =text.replace(re,"$1");
  $("#EXCERPT").val(text);
}

The regular expression replacement will remove replications in "space like" chars.

I’m using this code to instruct anti-spam services (like Defensio or Akismet), supplying them sample contents to tune spam detection, which we will use for Patapage (patapage.com)’s comments. These services usually need to know the content of the main article before rating a comment as spam or not.

XSS war: a Java HTML sanitizer

Yesterday I was testing one of our forthcoming online services, in order to check XSS (Cross Site Scripting) robustness.

XSS and CSRF (Cross Site Request Forgery) are the two scary “black beasts” of online services in good company with scalability, but while even nerd-programmers think about scalability, almost nobody takes care of XSS and CSRF, at least until the first attack 🙂

The XSS becomes a serious problem when you allow your users to add contents on your site/service; typical situations are blogs, forums, online chats etc.: when you are about to “print” user contributions you must do it conscientiously, you may found some “easter egg”.

Of course if you limit user contributions to plain text you will solve the problem in minutes, by just encoding every html tag (the idea is to change every “>” in “>” and “<” in “<” and so on). But things quickly get harder when you try to accept some basic html tags.

In our case, Patapage is a service that allows to add comments, threads and more options to existing web sites, it is tailored to appealing interfaces so it is a MUST to allow users to insert a wide set of html tags. We have found on stackOverflow.com a good balance from admitted tags and refused ones; as usual Jeff Atwood is prodigal of precious hints, but in our case while his set is fine for “patacomments” it is too restrictive for the scope of “patacontents”.

As you can imagine, I’ve found a lots of holes, so going upstream as usual and not giving a damn on, probably, the best Jeff’ suggestion (“do not re-write you own implementation”, an advice which he too is not following) I started writing code basing my implementation on these articles.

Another aspect of the problem is the conservation of layout: users can break the page by inserting unclosed or misplaced tags. If you hope that these flaws do not impact on security you are on the wrong path. A well planned css attack can, for instance, layer your application for click stealing or simply “switch” two buttons of your application with funny(?) results.

Of course I’ve looked around to find something matching my needs, but I found only two kind of approaches:

The first approach isvery basic focused on removing “< s c r i p t>” tags from your html; obviously this is a silly approach, you can inject js code without a “script” tag: you have a bunch of ways to do this by using dom events (onclick, onload etc.).
The second approach is to parse HTML to understand exactly what is happening in the code in order to remove un-allowed tags; this approach is logically the right one, you cannot expect to do better. Your code analyses the HTML, extracts the tags, and then you walk down the tree dropping out unwanted ones. Unluckily this approach requires a HTML parser library really “strong” with respect to malicious code, and usually these libraries are built for a different scope. Another aspect is that parser, lexer and walker are quite complicated pieces of code, so it is not a joke to test them completely. I’ve tested a couple of parsers with unsatisfying results.

This is why we wrote our own sanitizer by hand. Our approach is to remove unwanted tags and properties without testing HTML correctness in deep.

First step is to tokenize the code: a token could be one of : tag start (), comment (), tag content (blah blah), a tag closing ().

For instance test generates three tokens:

test

The tokenize method looks for <…> pairs or comments, and that its fine for our scope of restricting accepted tags: if a <…> pair is badly closed the tag will be html-encoded.

Having the tokens list, we will test every single token whether it is acceptable; again, we do not perform tag matching at first, so for us test its fine, we are working on security, not on syntax – we’ll also fix that afterward, as it is easy in any case to add a tag-pair counter to close at the end unclosed tags, which you actually find done in our code.

We loop for every single token and we test it with regular expressions. The flow is:

if token is a comment discard it.
if token is a start tag ( ) extract the tag (p) and attributes (style=”color:red” align=”center”)
1. if tag is forbidden it will be removed
2. if tag is allowed we will extract every attribute performing a check
 1. check “href” and “src” for admitted tags (a, img, embed only) and check url validity (only http or https)
 2. check “style” attribute looking for “url(…)” parameter, and eventually discarding it
 3. remove every “on…” attribute – e.g. onClick, onLoad, …
 4. encode attribute’ value for unknown ones
 5. push the tag on the stack of open tags
3. else the tag is unknown and will be removed
if the token is an end tag ( ) extract the tag (p) and check if the corresponding tag is already open. Eventually close those that are still open.
else it is not a tag and we will encode it

In order to avoid js injection on user inserted URLs we will accept tag only if the “href” attribute is there and points to a valid URL; we test url correctness with Apache UrlValidator, and this will cut out every “javascript:” tries. Same approach for the and tags.

Finished my hard work coding the shelter, I give the happy news to our design department that the sanitizer was done, and after a first minute of excitement Matteo (http://pupunzi.open-lab.com) told me that they have three different usages of user input: for displaying an HTML page on the front office, for displaying a textual abstract in lists in the backoffice, and of course for storing contents on the database.

So a sanitizer needs three different outputs, html-encoded with tag, text-only without tag, and the “original” version for the database. This is why the latest version of the sanitizer returns “.html”, “.text” and “.val”. Why you should store “.val” instead of the original input or “.html”? Because the original input may be “dangerous”, and may mislead the user in believing that all tags are allowed. The encoded value is not suitable in case of subsequent modification because of double encodings (e.g. “>” –> “>” –> “>” and so on). On the other side “.val” removes only forbidden tags maintaining all other user oddities (strange tags, comments, etc.).

We have set-up a public playground for testing our sanitization code: http://patapage.com/applications/pataPage/site/test/testSanitize.jsp. This page allows you to input a text and by pressing “test” your input will be printed (sanitized) on the page.

Source of our sanitizer are released under MIT license (i.e. free as free beer, just keep the attribution); see the complete code here.

These tags will be accepted, others will be encoded, and printed. If you like challenges try to inject a js in your text and, for instance, get an alert. Tell us about your victories, if any.

BTW, I’ve tested my code using XSS Me plugin for Firefox, and it passed all (about 15o) tests 🙂

30 years riding IT – part2

Year 1982, “mathematics 1st” second lesson: I meet Massimo, the red haired boy that will share my professional destiny for next 15 years.

Massimo owned an amazing programmable pocket calculator Hewlett Packard 41-CV.

“Calculator” is definitively a too modern term to define this strange tool that in order to do (1 + 2)*3 require you input 1 enter 2 enter + 3 *; this is called RPN (reverse polish notation). But the most amazing features of hp41cv where the HUGE memory (2233 bytes!), the capability of remember something like 300 steps of code and last but not least an alphanumeric display. Ok, alpha not so much, but surely numeric!

I MUST HAVE ONE!

So, after a Christmas selling Hi-Fi in my father’s shop I got enough money to by my first programmable device (second hand, but in good shape).

My HP41cv was fantastic, with its “crunching” keybord.

With this archaic device I‘ve been initiated to the art of “quasi-computer” programming.

http://www.hpmuseum.org/hp41.htm

…to be continued