Some thoughts on the web as a filesystem

This is a draft idea I have been working on. I am posting it to provide context for some other ideas I had recently.

Update: it looks like Jon Udell and the guys at Freebase are already dreaming along the same lines.

Some interesting ideas result from REST placing restrictions on you.

These constraints make web objecs similar to file-like objects. In fact, the entire URL space is a little file-like. Things rest in only one place (canonical). Many names can point to the same thing (hard links). The same object can be copied in multiple spaces.

Maybe that is why you can re-invent old Unix tools as web services?

Now, what happens if we try to merge web-based tools with a extreme file-oriented operating system?

You might end up with commands like this:


$ mount http://amazon.com
$ grep 'agile & software' /mnt/w/amazon/books > agile-books
$ cat agile-books | cut -3 | uniq -c
...
# Result: the number of books about Agile software
development, counted by publication date.
...


Or this:


$ mount http://google.com/gmail
$ mount https://acanvas.wordpress.com/
$ ls /mnt/w/gmail/inbox
...
foo.eml
...
$ cp foo.eml $home/drafts/foo-reply.txt
...
edit foo-reply.txt
...
$ cat $home/drafts/foo.txt > /mnt/w/acanvas/posts/new


Or my favorite, this:


$ mount https://acanvas.wordpress.com/
$ cat $home/drafts/post.txt | wiki2html > /mnt/w/acanvas/posts/new


I know, I’m not the first person to think of this, And there are problems like “What does it mean to run ‘cp’ or ‘mv’ on a URI, or it’s contents?”. But it is a problem worth solving – one could ‘program the web’, in the most literal sense.

Advertisement

The Agile Alliance takes a stand on certification

Jon Kern, one of the Agile Alliance founders, tells us that the Agile Alliance has come out against certification.

More specifically, the Alliance prefer skills-based certification over knowledge-based certification, but they do not believe either should be a requirement or restriction of one’s job. And I agree.

Certification: Natural versus Official Authority

This is a re-post of a comment I made during a discussion on Scrum certification at InfoQ. It relates some of the pressures that I see affecting the software industry. (And yes, I know it mentions Dave Thomas again. What can I say? The guy’s a legend.)

I agree, certification locks up access to a profession, often to the profession’s detriment. We have all encountered people with impressive official credentials who deliver less-than-impressive real world results.

It is the difference between natural and official authority. Fotunately, many of the brightest lights in the software community have great natural authority, authority gained by doing useful things, and by helping others. It is part of the culture.

I think the problem comes when you attempt to reconcile software’s culture with the rest of society as a whole. We are pressured into becoming an “official” profession in the vein of doctors, lawyers, and engineers. But what is wrong with aligning ourselves with professions that value experience and results, like sports, or the arts?

Dave Thomas, one of the Pragmatic Programmers, aludes to the problem of authority in his presentation titled “Herding Racehorses, Racing Sheep”. He notes how nursing faced similar challenges as software, and how they reconciled official and natural authority within their profession. Why can’t we do the same?

I am glad that the rise of agile and the certification debate are pushing these issues to the forefront. Software won’t realize its full potential until they are resolved.

Hilarious on so many levels…

Some questions have a perfect answer:

Cost-plus Contracting Caveats

I came across an important point when reading InfoQ’s interview with Paul Oldfield, titiled Doing Agile Right (it is an excellent interview, I highly recommend reading it.)

In an earlier post I refer to cost-plus contracting as a way to align the demands of lean software contracting with the demands of a client. But Paul brings up a very important point:

I commonly come across people who do not make the distinction between manufacturing and development. They see the best practice and highly automated production line processes, and want to apply the same sort of processes. Mentioning no names, I have come across situations where payment was for time and materials. Profit was related to head count. Here, efficiency meant reduction in profit, and was only attempted when the customer could no longer accept that the head count was necessary.

This reveals the problem with cost-plus contracts, namely, how does one calculate the “plus” part, the profit? Relating profit to head-count creates the problem Paul speaks of. Linking it to time causes problems as well. (I should note that I have seen cost-plus contracting referred to as creating “gold standard” work – the best money can buy, which is what we are trying to create with lean methods. By contrast, with fixed-price work the incentive is to cut corners whenever possible.)

Lean processes, particularly Scrum, have checks that can prevent intentional waste by the contractor. Transparency plays a large role here. Idealy you get a constant time-effort tradeoff from the velocity calculation. But that assumes that the original estimate is in-line. For example, I have heard that in some professions it is common to bill 125% of your time to the project, as a matter of course. On a lean contract one could inflate the estimate by 125% and not work as quickly as possible.

Some food for thought.

The Real Value of Twitter

So I came acrosspeople’s comments on Twitter, a communication hub service. Interesting stuff.

Some have commented on the banality and uselessness of the content, which is probably what the vast majority of people will create. But surely there are some real uses for this service?

Twitter serves an important role as a media-hub, a place where communication comes together. It could serve a very important place in the mobile world, namely, allowing people who are not plugged in to participate.

Imagine watching a twitter feed for a group of protestors. Or a group of friends doing the flash-mob thing. Twitter actually lets you watch the action on a number of different channels. SMS locks it away (and taxes it, to). This mixing gets really powerful when you combine it with other apps to do real things. (The current content’s banality is because most people aren’t doing interesting/relevant things all the time. So why are you telling everone about it?)

Just wait until some interesting event takes place that you can’t participate in, but you can watch on Twitter. Then you will see the value.

Some ideas for an evolved form:

• fluid group creation and dissolution
• more actionable system tie-ins (maps!, photo/video blogs)

Kind of reminds me of a popular IRC channel – lots of banality, the occasionsal useful tidbit. The real value on IRC is generated in a channel of 30-60 people dicussing a relevant subject area, so maybe there is some middle-ground for Twitter networks. You just need to find the right topic.

The Lore Masters

I came across Pragmatic Programmer Dave Thomas’ presentation “Herding Racehorses, Racing Sheep” while surfing the conference list for QCon.

The entire slide show is fascinating, but the slide about the fourth stage of skill development caught my eye. Dave mentions maxims as a teaching tool used by those who have attained fourth stage skills. This struck a chord with me and fits well within my experience. For example, career consultant Markus Buckingham uses maxims to great effect.

My wife and I are fond of a particular maxim:

Trust Children.

There are a number of maxims in the software world as well. They guide the design of any and every routine within a particular system. Some examples:

Python

Explicit is better than implicit.

Unix

From the Art of Unix Programming:

• Fail as early as possible.
• Always do the least surprising thing.
• K.I.S.S.

Ruby

Don’t repeat yourself.

XP

Release early, release often.

One point to note: practioners at the lower levels in the Dreyfus Model don’t necessarily understand or apply the maxims created by those at a higher levels.

I wonder what implications that has for the Ruby and Rails communities, give their surge in popularity? Or the fact that a lot of developers will be coming to the language with pre-existing maxims to guide them? The Python community may offer some insight here, as the effects of pre-existing maxims is well-known to them. (You can tell when a Java programmer has written Python code. It looks… different.)

But hey, we all had to start somewhere.

Watch This!

From David Henderson’s Blog. As it says, Watch This!

http://www.youtube.com/watch?v=6gmP4nk0EOE

P.S.

There is a final version of the video, but I prefer the original. The style is more spontaneous.

Seaching past distractions

For a while now I have wanted a “visual search engine”. A search engine that will answer when I ask ‘What was that site? The one with the gross burgundy bar down the right-hand side?’. And yesterday I came upon a workable answer to this problem!

It’s a bit of a whirlwind, so hold on:
It starts with an attention recorder, a piece of software that keeps track of the web sites I view. Normally, the recorder would store the site addresses in an attention vault of some sort. But what if I stored images from snap.com along with the addresses? You know, site preview images, those little thought bubbles that pop up when you hold the mouse over a link (this weblog uses them).

That small site preview image from Snap.com is just large enough to recognize large blocks of colour and contrast. We can use simple techniques from desktop optical character recognition software to look for patterns in the Snap images. The user would draw a simple picture using a trivial drawing program (simpler than Microsoft Paint); a coloured box, maybe a line. The drawing is broken into patterns, and the search software looks for those patterns in the attention vault images. The user gets back a list of sites they have visited that look similar to the picture they drew.

Pretty cool. This technique lets the user perform vague searches like “the page was green down left”, or “it had a circle in the middle”, or “there was a distinct blue and orange theme”. Mix in some of the artistic and design skill necessary for good data representation, and you have a real application.

So there you go. Snap.com, my attention vault, and OCR technology shine a new light on my past distractions.

Lean Software and Subway Lines

My co-worker made an interesting comment after seeing this excellent interview with Mary and Tom Poppendieck, titled “Using Lean for Competitive Advantage”. Straight to the quick: “They are right, Lean process won’t work in a contract-based environment”.

The problem is simple — customers expect you to deliver X software product, on Y date, for $Z cost, like any off-the-shelf manufactured product. However, as Mary states in the interview, software development cannot be treated this way.

What if software development was seen as a service first? Contracts would retain a team of software developers, for a fixed rate per month, for a few months. At the end of each month the team delivers working software. You can run the meter as long as you want, switch service providers at any time, or ever completely redefine the product you would like to receive. You, the customer, have complete control. Iterative development practices, such as SCRUM and Extreme Programming, make this possible.

My co-worker told me how engineering had dealt with the problem long ago by introducing “turnkey” and “cost-plus” contracts. The turnkey contract lays down a fixed cost and delivery date. A typical turnkey project would be a factory, bridge, or water treatment plant. These things have been built a thousand times before, and the challenges are well-understood.

The cost-plus contracts are the interesting part. The construction of the Shepard subway line in Toronto was done under such a contract. A team of one hundred engineers was placed at the city’s disposal, for $1 million per month, on an on-going monthly basis. The engineers would start, and later extend, a subway line under the city.

Performing any part of this project on either a fixed-cost or fixed-price basis would be crazy. Subway lines have been built before, but each one is unique. You never know what you will find while digging dozens of kilometers of tunnels under a city. You also do not know if the city will run out of money, or when the political sponsorship may change. But the subway was needed, so contracts were drawn up, financing was aquired, and the work went on until the city said “enough”.

Software projects, especially large ones, face many of the same problems and uncertainties. But customers, for whatever reasons, insist on a turnkey software solution! It is our job, as developers, to show them that there are better ways to build software. But we must also show the customer how ‘software engineering as a service’ is a better way to finance projects.

Engineers must be able to calculate per-month contract rates in order to write cost-plus contracts. And they must be able to show their clients when metered contract work is necessary. Lean software development could probably learn something by studying how engineering cost-plus contracts are drawn up and pitched.