I’m often asked how bloggers and other online content producers can begin accepting bitcoins on their website, so I thought I’d quickly put together an article to save others the trouble.

Why accept bitcoins?

The first question is why should a blogger want to use bitcoins in the first place? Well, most bloggers want to have their efforts rewarded in some way, and most resort to providing context sensitive advertising. The income from this kind of offering can be very low, especially if the blog is in a niche that does not get well served by ads. There is also a sense of being disconnected from one’s audience, which works both ways. When I appreciate someone”s work I want to donate directly to them and I don’t want some advertiser or payment processor taking a cut.

Bitcoin solves this problem, quickly and efficiently.

Getting started

The quickest way is to offer a “Bitcoin swatch” (see the MultiBit FAQ for more information) as an image link with a Bitcoin URI and a suggested donation. For example, my blog has one over there on the right using the following HTML:

<div>
<a href="bitcoin:1KzTSfqjF2iKCduwz59nv2uqh1W2JsTxZH?amount=0.5&label=Agile Stack">
<img src="http://gary-rowe.com/agilestack/wp-content/uploads/2011/10/AgileStack-0.5BTC.jpg" ></a>
<p>1KzTSfqjF2iKCduwz59nv2uqh1W2JsTxZH</p>
</div>

The Bitcoin swatch image was dragged out of the MultiBit “Receive Bitcoins” screen onto the desktop and then uploaded using the standard WordPress image import process. MultiBit is a free and open source Bitcoin client.

You’ll notice that the “href” attribute uses a different protocol than the usual “http”. If someone has installed a Bitcoin client onto their system then it will very likely be configured as a handler for that protocol. Clicking on that link will cause the appropriate application (or browser plugin) to pop up, usually in the “Send bitcoins” screen with the details provided already filled in. MultiBit does this from version 0.3 onwards.

It doesn’t matter which browser is being used because it is the operating system that manages protocol handlers. Developers interested in getting this kind of functionality to work in their own systems may want to look at this Stack Exchange answer.

The QR component of the Bitcoin swatch allows people with smartphones to make donations using a Bitcoin wallet, or to use drag and drop payment. You’ll notice that I’ve also left the raw address visible. This is to allow people who do not have a suitable Bitcoin client to be able to copy paste the address into their respective client as a last resort.

Adding a donation counter (optional)

Finally, if you’re accepting donations rather than selling a product, it might be useful to provide a “donated so far” label to give people an indication of your ongoing campaign. The Block Explorer site provides a wealth of useful information about Bitcoin addresses, and coupled with a Text-to-Image web service can give a nice result without any server-side processing.

For example, the following snippet (inspired by the Bitcoin Trader blog and offered up by Jim Burton) provides the amount sent to a given address.

<img src="http://ansrv.com/png?s=http://blockexplorer.com/q/getreceivedbyaddress/1KzTSfqjF2iKCduwz59nv2uqh1W2JsTxZH&amp;c=000000&amp;b=FFFFFF&amp;size=5" />

as shown here:

Why not see it change value by sending it a bitcoin? ;-)

By cycling your public addresses you can reset the amount as required and provide a label indicating the time span.

Known problems

It does appear that if you are using WordPress.com to host your blog, then it may not allow you to use to bitcoin: protocol and will change it to http: or omit the link altogether. If you encounter this problem then contact the WordPress.com staff and they may be able to help you. If you host your own version of WordPress, or run your blog on Blogger.com, then this is not an issue.

Final word

So, if your blog offers content that is of real value to others then they can now show their appreciation by donating a small amount (less than a dollar) very easily. No fees. No registration. No hassle. Just you and them.

To all those who have donated to my blog – I thank you, personally.

Background

A little while ago I discovered Bitcoin. Put simply, it allows people to send any amount of money, anywhere in the world in about ten minutes without incurring an excessive transaction fee. It’s based on a very effective use of digital signatures and a public ledger of transactions to avoid double spending. In short, it is money, reinvented. If you want to send someone less than a dollar on the other side of the world, you can do it with Bitcoin.

There are many Bitcoin clients out there (I contribute development effort to the MultiBit project) and some have taken advantage of the Android platform. In particular there is Andreas Schildbach’s Bitcoin Wallet (an outstanding mobile solution with many ongoing developments) and Brian Armstrong’s Bitcoin Android (an early client that has seen a reduction in recent commits). I’ve used Andreas’ code for some time now as my primary mobile wallet and had an excellent experience.

And then my smartphone died as a result of faulty networking hardware.

Well, not a proper death, just enough to cause all the non-factory applications to stop working and to put my bitcoins (small “b” for the unit of currency with the Bitcoin protocol) at risk. Bitcoin requires you to keep your private keys safe since they provide access to your bitcoins in the public ledger (called the blockchain). Lose those private keys and you lose your bitcoins. Forever.

What follows is a guide that shows what I had to do to get my bitcoins back. If you find yourself in the same situation, perhaps this will help you. It does assume that you’re very technically proficient. If the idea of installing the Android SDK and using a shell to run up adb fills you with horror, stop now.

What you have to do

1) Do not attempt to re-install your Bitcoin wallet application – it will very likely delete your local wallet file and that’s the end of your keys.

2) You absolutely have to root your phone. Mine is (at the time of writing) a HTC Desire, so I used unrevoked3 (see http://unrevoked.com/). This should not delete any data from your system, but will enable you to access otherwise protected files.

3) Install the Android SDK and get the “platform tools” variant for your platform so that you get the adb application.

4) Hook up your phone with a USB cable (you’d probably have left it in after rooting it)

5) Fire up a terminal session and enter the following command (I’m using “>”, “$” and “#” to represent where you are in the shells, don’t actually type them)

> cd <wherever you've installed Android SDK>

> adb shell

$

In the adb shell switch to become root

$ su

#

If you get a # then you’re root. If not, try again with the unrevoked3.

6) Navigate to the wallet file directory

For the Andreas Schildbach Bitcoin Wallet do this

# cd /data/data/de.schildbach.wallet/files

Copy the all-important private key file somewhere that adb can get to it

# cat key-backup-base58 > /data/local

You’ll notice that “cp” and “mv” are not options for adb against a production build of Android. Hence the sneaky use of “cat”. And, yes, I did try mounting the partition as read-write and totally failed to get it to work.

For the Brian Armstrong Android Wallet do this

# cd /data/data/com.bitcoinwallet/files

Copy the all-important private key file somewhere that adb can get to it

# cat prodnet.wallet > /data/local/prodnet.wallet

7) Exit out of the shells (root then adb)

# exit

$ exit

8) Pull the private key files off the device and somewhere local

> adb pull /data/local/key-backup-base58

> cat key-backup-base58

The Schildbach key is stored as a simple base58 format that looks a bit like this:

5dsflkjsflklnsdaflsmdflmsortofthing

The Armstrong key is more complex, it is stored as a serialized ECKey. To get the private key out of it I had to use the BitCoinJ library (version 0.2) and make use of the DumpWallet.java file.  Poking around with an IDE and debugger lead me to the private key that looked a bit like this:

42a34b31e9a4eedf56980ee0fc32fe6e675ff7007651ff3sortofthing

So now you have to get this private key back into a safe place.

9) Use Mt Gox to import the key

By far the fastest way is to just register an account with Mt Gox (a major Bitcoin exchange) and use their very flexible private key import facility. Just select “Private key” as a deposit method and copy-paste the contents of your key. The Mt Gox exchange can recognise a wide variety of formats and if it recognises your input then it’ll immediately give you a balance associated with that key. A short while later your Mt Gox account will be credited with the bitcoins and you can do with them as you wish.

10) Finally do a clean up to make sure you don’t leave the private key lying around for anyone to grab easily

> adb shell

$ su

# cd /data/local

# rm key-backup-base58

# rm prodnet.wallet

# rm prodnet.keychain

# exit

$ exit

> exit

And you’re done.

This shows what must be done to recover your private keys and is correct at the time of writing. However, you should be aware that soon all private keys will be encrypted so that in addition to the above steps, you will also need to know the passphrase to gain access to the private key.

I write web applications in Java, and have been doing so for some time now. I like to think that I’ve become pretty good at it, and I can usually get something working quickly and efficiently. This means that my choice of frameworks and tools is somewhat specialised, and others would make a different choice, perhaps Grails or Ruby on Rails. This article is not about which framework is superior to another, but rather what I’m currently using and why it’s good for me right now.

Next year, it should be different otherwise I’m not really progressing.

IDE

Intellij IDEA. I used to be an Eclipse evangelist, but then my friend Gareth introduced me to
Intellij and I never looked back. Well, maybe once or twice when I had to reorient myself within the new IDE, but once I got the hang of it then that was it. Everything just works really well, and the refactoring is amazing. Especially in XML and JavaScript. Oh, you thought refactoring was just for Java? I invite you to download the freebie version of Intellij and try out the goodness for yourself. If you’re a contractor like me you’ll find that the productivity increase pays for the IDE within a day or two.

Build systems

I’ve written about this before, but there is only one build system for me at the moment: Maven. I think it reduces the problem of creating portable builds (and environments) to a pretty simple format. It’s a pretty straightforward way of allowing a disparate team of developers to work on different aspects of a large project. In my opinion, a build system should be able to get a new developer productive within 30 minutes. That includes time reading the wiki to locate the appropriate project and check it out of version control. It also includes the time taken for the developer to figure out (using only the documentation) how to build and deploy it locally with a sample set of data and a demonstration page so they can see how the application is supposed to work. If your set up does not allow a developer to do this, then – seriously – look at your systems.

In Maven, the way I’ve got my stuff set up, you get a fully operational environment as described above by doing the following:

1) Check out your project from version control
2) Type mvn clean jetty:run at the command line in the project checkout directory
3) Navigate to localhost:8080/SomeApp in the browser (the wiki will tell you where to go)

That’s it, and it’s the same procedure for all applications at all tiers of the stack. The act of
issuing that command will automatically ensure that the correct supporting versions of any other JARs and WARs are downloaded, included and executed within the same Jetty instance. This gives a complete set of interactivity so that a developer knows they have a representative slice of the overall system. Not bad for a single memorable command, eh?

Web services with RESTEasy and JAXB

I used to use SpringMVC for my web front end. It remains an excellent implementation of the standard Model/View/Controller design pattern that made front end design much simpler than the frameworks that had gone before – I’m looking at Struts here. Now, I’m hooked on an even simpler framework that trivialises the whole creation of web services to a mere typing exercise: RESTEasy. Wow. What a revelation! I’m sure that in Spring 3+ they’ve done something similar (and I will take a look, I promise) but right now RESTEasy is my best friend. Here’s an example of creating a simple web service to respond to a GET request:

@Path("/hello")

public class Hello {

 @GET

 @Path("/{name}")

 public String example(@PathParam("name") String name) {

 return String.format("Hello, %s",name);

 }

}

HTTP GET /ExampleApp/hello/bob

Hello, bob

It’s difficult to imagine a simpler way of coding that service up. Nothing is wasted, the web.xml is trivial and the same for pretty much all the web services on the back end. And it integrates with Spring too. And JAXB. And JSON. And Maven. In fact, it’s damn near perfection.

And what’s so good about JAXB I hear you ask? Well, let me tell you. Take a POJO, apply some simple annotations and hand it over to RESTEasy. All that hard effort of turning it into XML or JSON or whatever for transmission to the outside world is all taken away. Consider a simple class that provides a list of Users formatted as a paged result set:

@XmlRootElement(name="Result")

public class Result {

 @XmlElementWrapper(name="Users")

 @XmlElement(name="User")

 private List<User> users=new ArrayList<User>();

 @XmlAttribute(name="firstResult")

 private int firstResult;

 @XmlAttribute(name="batchSize")

 public int getBatchSize() {

 return users.size();

 }

 public void setFirstResult(int firstResult) {

 this.firstResult=firstResult;

 }

}

Assume that User has been created and marked up in a similar fashion. Put this into a RESTEasy web service, like so

@Path("/search")

public class SearchService {

 UserDao userDao = new UserDao();

 @GET

 @Path("/{query}")

 public Result example(@PathParam("query") String query) {

 return userDao.search(query);

 }

}

You’ll see the following

HTTP GET /ExampleApp/search/bob

<Result firstResult="1" batchSize="1">

 <Users>

 <User>

 <Name>Bob</Name>

 </User>

 </Users>

</Result>

Neat and very obvious how it all works so maintenance issues drop off.

Database access with JPA Now I jumped on the Hibernate wagon with everyone else. It solved a huge problem in the ORM world in a neat and elegant manner. It also educated mere mortals into the proper way to think about the way our applications dealt with database issues. I’m thinking of sessions, lazy loading, avoiding the dredge anti-pattern, entity to table mappings and on and on.

In fact Hibernate was so good that it sparked a complete revisit of the awful EJB approaches so that EJB3 is actually palatable – if you’re starved of any alternative that is. I’m still not convinced that EJB is compatible with an Agile approach to development. I’ve not yet seen an EJB work efficiently in a Jetty environment that neatly supports JUnit integration testing against an in-memory HSQL database. Spring on the other hand manages that feat with ease, particularly in JUnit4.

One good thing about EJB3 is the creation of the JPA and JTA specifications, which formalised what had been going on in Hibernate for ages before. All those lovely annotations were ratified and the persistence layer took a major leap forward. As a developer this puts me in a good position where I can be ambivalent about the implementation technology behind the annotations. Want me to code up your DAOs using JPA and JPQL, that’s fine. Want the same done with Hibernate and HSQL, just gimme a mo to tweak some queries and you’re good to go.

Automated functional testing

I’ve discussed this elsewhere too, but I thought I’d include a mention here for completeness. No development stack is complete without catering for automated testing. Everyone reading this naturally has a unit test for everything – don’t you? You don’t?! You’d better get started then, ‘cos everyone else has. However, once you’ve got your unit tests working nicely with both dependency injection and mocking approaches. Which reminds me, I’d recommend JMockit over JMock these days due to it’s better handling of static methods which are sometimes forced on us by legacy code. Then it’s time to consider the functional testing. For some reason everyone seems to adopt the line of “Oooh, I’d love to do that but I just don’t have the time.”, which in my opinion doesn’t add up. Every developer should strive to make their days easier, and having the safety net of a  comprehensive set of functional tests is essential. Take a look at my detailed article on the most excellent SeleniumRC for more information.

Performance

With the above technologies, I can usually get a full-blown, integration tested RESTful web service off the ground and operational against a database within 2 hours. Most of that time is spent typing and configuring for the particular problem I’m facing. I know other frameworks exist that could speed that up (Appfuse and Spring Roo, er, spring to mind – sorry). And, yes,
I do have templates to solve standard problems. However, within the constraints imposed by my clients who want to use, or be guided into using, a particular technology, this is a pretty good delivery time.

Anybody here ever seen a zombie flick? Well then, we all know what a zombie is, but what is a zombie meme? Read on for more.

What is a meme?

For that we need to refer to Richard Dawkins and his book The Selfish Gene. He makes the parallel between a human gene and a unit of culture, which he termed a meme. I’m using culture here in it’s social sense to mean a collection of ideas that shapes our actions that is common to our society. There are a lot of interesting parallels that cross over between the genome based view of biological systems, and the memetic view of cultural systems.

For example, our genes use us to replicate. Therefore genes compete with other genes (in other humans) to ensure their continued replication. One strategy is to introduce characteristics in their host that promote the end result of successful replication. This is the basis of evolution and the continued survival of the most fit organism for their environment.

Memes compete with each other by taking up space in our culture, which essentially means our awareness. They can be as simple as knowing that an iPod can be used to play music, or as complex as a natural language. To be successful a meme must replicate, and there are many strategies to help it do so. The “put wheels on a sledge to make a wagon” meme was successful in creating the automobile industry because it was immediately obvious to anyone looking at a wagon how to build it. The manifestation of the meme (the wagon) was replicated, and so the meme was able to leverage this to create many more copies of itself than the original item. More people know about wagons than own one.

So a zombie meme is…?

One that no longer serves a useful purpose and is simply getting replicated for the sake of it. People simply adopt the zombie meme as “best practice” without question and mindlessly repeat the instructions regardless of how damaging those instructions are.

Examples of this include wasteful business practices:

“Can you print that email for my records?”
“We only take orders by fax, then we type them into our computer system. Hope you’ve got good handwriting.”
“Please read the 50 page guide before filling out this form.”

or software development methodologies:

“Code around it quick, we’ll get the users to test it.”
“We design, code, test and ship. In that order. Once.”
“We don’t use XP, we use Scrum.”

Hang on – did you just have a go at Scrum?

Indirectly, yes. Scrum is the prevailing Agile implementation and there are a lot of excellent reasons to use it. However, simply using a particular methodology without fully understanding what type of development environment it is targeted at is essentially allowing a zombie meme to infiltrate and replicate.

But everyone does Scrum so it must be the best, I hear you mutter. Not necessarily. Consider the article Flaccid Scrum by Martin Fowler where excited project managers pick up on the Scrum process. All is well for a while then the project slams into a brick wall due to overwhelming technical difficulties caused by a legacy code base. This isn’t Scrum’s fault, you need to have good technical practices in place before Scrum will show it’s many advantages. A better choice in this scenario would be XP with all it’s technical recommendations.

Another scenario is that the nature of the work doesn’t lend itself to a constantly changing work load, such as maintenance programming or rapid response to a fickle client. Some changes take a day, others a week or so. Scrum relies on a clear stretch of time to accomplish the work, whereas Kanban provides a much more flexible approach.

And what if the project involves a dual-shore development team, operating in different time zones? It may be better to adopt the Stride approach (PDF), which is based on Scrum but tailored for separated teams.

As I mentioned earlier, memes replicate, but they also mutate leading to a conflation of ideas that can in turn lead to a better fit. Such is the case with a group of people who are now trying to blend Scrum with Kanban to find an even more efficient and flexible development process.

There are interesting times ahead, and not a zombie in sight.

As a full time developer I spend a lot of time writing code. And I almost never get it right first time. I did once, a long time ago, and since then I’ve not had such luck so I’ve had to rely on a myriad of safety nets, such as unit tests and a keep it simple approach to design. Even then my code tends to grow somewhat organically leading to various code smells that a healthy wave of refactoring wafts away.

I thought I’d take this opportunity to share the ones I use the most often – in fact I couldn’t really live without them these days.

1) Rename and Change Method Signature

My personal favourites. Rename does just what it says on the tin. Initiate the refactoring, tell the IDE to apply it in all the right places and it just does it. When applied to fields in Java beans, the IDE will ask if sir would like those getters/setters renamed as well, and throughout the application, including JSPs. Yes, please, do that for me and save me a whole bunch of time.

Change Method Signature is even better because it includes Rename as a happy side effect. Want to change the order of those parameters and their types but don’t want to screw up all the myriad of calls into those methods? Why, yes, yes I do, Mr IDE. Just click in this box until you’re happy with the new order and let me do all the hard work. Don’t mind if I do (cracks open beer).

2) Extract Method

Especially useful when exploring a new code base. If you want to simplify a method that has clearly gotten out of hand, just select the area you feel should be a method in it’s own right and hit that refactor button. Suddenly all that complex code is whisked away and in it’s place is a simple method call. And if your IDE is really spiffy then you’ll find that all the other copy/pasted versions of that code have also been refactored into the method call. Marvellous.

3) Pull Up and Push Down

Have you suddenly discovered that a method or field is going to be duplicated in a bunch of subclasses? Use Pull Up to move the duplicated method or field into a super class. Done correctly, the IDE will automatically apply the refactoring to all subclasses with the duplication and substitute the super class reference in it’s place.

Push Down, as it’s name suggests, does the opposite. Your wonderful base class is now preventing subclasses from specialising as they should so you push the field or method down so that the subclasses can implement the method or not as they see fit.

4) Extract Base Class

Want to do a Pull Up but don’t have a super class in place? No problem, just use this and select the methods/fields that you want pulled up. Want that base class abstract? Not a problem, just tick the box, sir.

5) Introduce Parameter Object

If there’s one thing I really hate it’s a method with a long parameter list. And by long I mean 5 parameters maximum. Especially if they’re all named string1, string2, stringX and so on because a) it’s mental, 2) it’s confusing and iii) nobody will ever get the parameters in the correct order.

The solution to this debacle is, of course, to introduce a parameter object. A simple Value Object (read Java Bean) that abstracts all the parameters away behind a single reference. Want to add a new parameter but don’t want to have to go and change all the method signatures on your publicly available API? Just make sure they take a parameter object and the API will continue to work as you change the properties on the parameter object. If you were feeling really ambitious you could put validation code into the parameter object to provide a defensive layer, but that’s just silly, isn’t it? Isn’t it?

6) Introduce Constant

We’ve all done it: hard coded a string or number somewhere. Then used it again. And again. Then realised that, really, it should be defined properly somewhere up in the root of the class or ideally injected. And that is exactly what the Introduce Constant refactoring does. With a swift click of the mouse all the duplications are pulled up into a single, well named constant field destined to remain the single point of reference for that datum for all time. Very DRY.

7) Introduce Field

Method signatures getting a bit repetitive with that same reference getting passed around all over the place? It’s about time that parameter was converted into a fully fledged field.

The only problem with refactoring is that after you’ve done it a few times you kind of get addicted to it. Before long you’ll be spotting duplications and excessive parameter lists all over the place and applying refactorings to tidy them all up and what are you left with? Just well designed, clean code that is much easier to maintain.

Nope. Well that was easy. Except you probably do.

According to the Don’t Repeat Yourself (DRY) principle, all code and reference material should have a single point of reference that is both unambiguous and easy to find.

Most major development efforts will be spread over several large projects. As a natural result of this there is going to be some duplication between, at least, the fundamental utility and data access classes. One approach in combating this duplication is to create a project that could absorb a lot of the commonality found in several diverse applications into a single module for inclusion in each.

As most developers know, the Apache Foundation have provided a vast array of common extensions to the Java language – some of which have been included in later versions of the language itself because they were so useful. However, as a taster, there are file utilities, specialised collections, string manipulation methods, date handlers, logging frameworks all of which provide a standardised approach to solving commonly found problems. Rather than roll your own library, why not take the best of breed from Apache and ask new developers coming to the company if they have familiarity with these libraries? The learning curve drops away the more you depend on freely available standard software.

Even after refactoring the existing code to use the various Apache Commons libraries, the size of the common project will increase. It is important to keep it focused so that it does not swell to be an all-encompassing uber-module that all projects must depend on. As the packages contained within the distributable JAR get too numerous, split them out into sub-modules that are highly targeted. A JAR for utilities, another for DAO interfaces, another for web utilities. Different projects can then use these highly specific, and loosely coupled, collections of foundation classes to meet their exact needs without needing a load of extraneous dependencies to also be included.

If you are using a build process like Maven then the issue of dependency management becomes trivial. Developers can make updates to the common modules and release new versions of the JAR into the company repository (managed by Artifactory for example). Other developers are unaffected until they decide to update their project file to include this higher release version. This allows them to continue without the new features saving the upgrade to a later time that suits them.

As these modules begin to grow in size it becomes difficult to keep track of all that useful code that your developers have spent a lot of time creating. It is at  this point you should consider introducing the role of a Code Librarian. This role, ancillary to normal development duties, is responsible for the maintenance of code that is deemed to be useful in more than one project. They can also assist with refactoring the original design of the code so that it fits with a wider audience. Typically the code librarian would sit in on code reviews, or be called over if doubt has arisen. They can then point out where the developer has written code themselves that should have been brought in from a common library, or to find ways of incorporating new code into the library.

The creation of a well-maintained centrally-managed code repository drastically reduces the amount of time and effort it takes to get a new project off the ground and out into production. It also greatly reduces the amount of developer time spent debugging. If the supporting documentation, such as a Wiki, contains good examples and references to other similar classes with slightly different applications then developers will quickly find themselves avoiding the re-invention of the wheel. For an example of this, take a look at the YUI framework documentation.

So you’ve implemented a bunch of unit tests and you’re feeling confident that your system works as expected. Then you realise that you now have to work through all those different use cases one by one every time you make a change just to be sure that you haven’t somehow broken the front end. Surely there must be a way of automating this process? There is, and it’s called Selenium.

In this article I’m going to focus on how to introduce Selenium into your development process assuming that you have developed a reasonably large web application based on Java technology. Selenium will work quite happily with a wide variety of other development languages, but since I’m not so familiar with them I’ll leave that to others to explain.

So, having lost 90% of my readers, I’ll struggle on.

I define a functional test as one that essentially mimics a user working through a particular collection of use cases. In the case of a web application, this means that a functional test needs to remotely control a browser; fill in various forms or click links; and verify that the responses from the server are as they should be. Typically in the Java world some variant of HttpUnit is used for this purpose but this only represents the behaviour of a single browser – the HttpUnit implementation.

As an aside, it should be noted that there is a strong correlation between functional tests and requirements. It is indeed possible to create a functional test matrix that allows requirements to be ticked off as their corresponding functional tests are seen to pass. This follows both the DRY principle and the Agile manifesto maxim of “Working code is the best metric”.

So what does Selenium do? Specifically, in it’s Selenium Remote Control (SeleniumRC) form it acts as a bridge between the browser and your automated build process. You write your tests in JUnit as usual ensuring that your assertions target the responses coming back from a Selenium browser object. For example:

selenium().open("http://localhost:8080/mywebapp/index.html");
assertTrue(selenium().isTextPresent("Hello World");

The selenium() method provides access to the underlying browser. If the assertion fails then the test case fails and the usual reporting mechanism takes over.

OK, so what is going on here? Essentially, selenium() communicates with a SeleniumRC server instance, which is just a single JAR running in the background somewhere, and instructs it to fire up a browser, point it at the given URL, wait for the page to load, then check if the given text is present in the response. Not bad for 2 lines of code. Now here is where SeleniumRC gets very clever: first, you can instruct it to fire up an instance of pretty much any browser you like; second, it will inject some JavaScript into the page which will allow Selenium to perform deep selections within the browser DOM.

That abstraction of the browser, and the corresponding deep inspection, makes Selenium extremely powerful when used in a large scale environment. Since Internet Explorer does not sit well with other versions it is usually necessary to create a collection of virtual machines each with their own version of IE and SeleniumRC installed. Other virtual machines can be created that allow Firefox, Opera, Chrome, Safari etc to be installed in a variety of combinations. However, the same suite of functional tests can be executed against them without change (so long as you use CSS selectors throughout otherwise IE runs like a dog). In a Maven environment this can be implemented as a series of executions each with a few environment parameters that can be passed to the test suite. The environment parameters indicate where the SeleniumRC server is to be found (the VM) and what browser it should target. For example (in build/plugin):

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<!-- Default configuration is to skip -->
<configuration>
 <skip>true</skip>
</configuration>
<executions>
 <execution>
 <!-- Test with Firefox 3 -->
 <id>1_functionalTestWithFirefox</id>
 <phase>verify</phase>
 <goals>
 <goal>test</goal>
 </goals>
 <configuration>
 <skip>${maven.test.skip}</skip>
 <environmentVariables>
 <selenium-rc.server-host>selenium-xp-ie7ff3</selenium-rc.server-host>
 <selenium-rc.server-port>4444</selenium-rc.server-port>
 <selenium-rc.browser-type>*chrome</selenium-rc.browser-type>
 </environmentVariables>
 </configuration>
 </execution>
</executions>
</plugin>

But wait, there’s more.

Since a functional test is considered as a collection of use cases it makes sense to structure the testing environment to follow this pattern. Essentially a series of standard unit tests are created with a FunctionalTest suffix to make their purpose clearly visible. These are usually packaged in line with the various system modules that they test. The functional test simply delegates to the use cases they wrap (passing in a reference to themselves to allow the use case to access test methods). Since there is a lot of common functionality between functional tests (such as the initial creation of the Selenium browser reference) it makes sense to introduce an abstract base class that provides these references. For example:

package org.example.web.functest.security;
@RunWith(BlockJUnit4ClassRunner.class)
public class LoginAdminFunctionalTest() extends AbstractWebTestCase {
  ...
  @Test
  public void loginAdmin() {

    LoginAdminUseCase login = new LoginAdminUseCase(this);
    login.execute();
  }
}

By contrast, the use cases are simply implementations of the Command pattern which do all the work. They have a UseCase suffix to allow for quick identification, and typically extend an abstract base class to provide useful common methods. Extending org.junit.Assert will provide direct access to assertion methods. For example:

package org.example.web.functest.security.usecase;
public class LoginAdminUseCase() extends AbstractUseCase {
  public LoginAdminUseCase(AbstractWebTestCase tester) {
    // The abstract super class provides various selenium methods via tester
    super(tester);
  }

  public void execute() {

    selenium().windowMaximize();
    selenium().deleteAllVisibleCookies();

    // Obviously the URL here would be passed
    // as a parameter by the build process
    selenium().open("http://localhost:8080/mywebapp/login.html");

    // Assumes that AbstractUseCase extends Assert
    assertEquals("My Login Page", selenium().getTitle());

    selenium().type("username", "admin");
    selenium().type("password", "admin");
    selenium().click("submit");

    // Convenience method to introduce standard short delay
    waitForPageToLoad();

    assertEquals("Administration console", selenium().getTitle());

  }
}

Any number of functional tests could re-use the LoginAdminUseCase in any combination. It would be trivial to include a LogoutUseCase that clicked a common logout link. In fact, at the end of every test as part of the tearDown() sequence the LogoutUseCase could be invoked.

I have glossed over several important implementation details in the interests of brevity. For example, the issue of what acts as the container for the web application, and how this can be invoked before testing starts. My favoured approach to this is twofold: one for developers the other for continuous integration servers.

The developer version uses Jetty to act as a lightweight servlet container that can be stopped and started very quickly; is responsive to changes in the web application (e.g. a JSP or CSS adjustment) while running; and allows for programmatic configuration within the functional test module. This allows a developer to work in a much smaller environment with perhaps a single functional test driving their work so there is very little repetitive typing or page navigation going on. Since the environment responds immediately to changes very rapid progress can be made.

By contrast, the continuous integration version relies on Maven to start and stop some larger application container that is representative of the final deployment environment. In the case of Glassfish this is the asadmin plugin, but others exist for different containers (notably the general purpose deployment plugin for Cargo).

In addition to the container, the database must also be regulated so that provides predictable results. Usually, after the application container has been stopped a script is executed against the database to drop all objects and rebuild. Clearly the test database is a minimal subset of the actual production schema. It must be large enough to allow a wide range of use cases to be explored, but small enough so that it doesn’t dominate the build process.

Once the application container is restarted, then the build process can begin the functional test suite with each of a variety of browsers being targeted at the system under test. The individual functional test results are collected and presented as part of the final build result.

In this article I aim to examine whether or not stored procedures should be used instead of Hibernate based queries. I’ll assume that both Java developers and DBAs are likely to be reading this, so here is a quick glossary of common design patterns found in Hibernate applications:

Data Transfer Objects (DTOs) are used to transfer state information between layers in an application. They tend to represent the business domain and it’s relationships quite closely, and contain mapping information so that they can be persisted within the database.

Data Access Objects (DAOs) are used to perform all persistence operations: creating, retrieving, updating and deleting data from the database by means of manipulating DTOs. If a DTO is deleted, this is mapped to a SQL delete operation on the database.

Object/Relational Mapping (ORM) is the process by which application business state is transferred into relational data.

Assumptions

The issue of whether to go wholly for a Hibernate based query approach, or, on the other side of the coin, a wholly stored procedure one is highly contentious with many strong arguments being presented for and against either approach. So let’s start by introducing some important assumptions:

1) both developers and DBAs are working for the same company and so are attempting to achieve the same objective of a simple to use process that promotes good practice;

2) the choice of RDBMS is not going to change in the near future so database independence is not an issue;

3) any move away from an entirely stored procedure approach will remove the burden of data consistency from the DBAs and place it on the application development team (programmers with DBAs acting as advisors);

4) the issue of “dynamic SQL” where SQL statements are created on the fly by, for example, concatenating strings without using parameters is not acceptable under any circumstances (do I hear SQL injection attack?) so all SQL generated within the application can be assumed to be parameterised;

5) application generated SQL is assumed to have come from an O/R mapping solution, such as Hibernate, not directly coded into the application;

6) long running queries acting on millions of rows are expected to be run as stored procedures since the network cost of moving the data over the wire would outweigh any performance improvement available through highly optimized application code.

Overall, there are only 3 choices available: persistence entirely through stored procedures; entirely through Hibernate; or a hybrid approach. Within the hybrid approach there is only the degree to which stored procedures are used within the application.

I have found that here are many myths and arguments associated with the stored procedure vs application SQL debate which collapse under close scrutiny. Here are a few (I’m sure that there are more):

1) Debugging stored procedures is hard

Many developers become used to working within a single IDE, and feel uncomfortable when forced to move outside it. In the case of stored procedures, it is frequently necessary for a Java developer to make use of an application such as Toad or JDeveloper to pick their way through a difficult SQL statement. To use this as an excuse not to use stored procedures is just laziness. The tools are there – learn to use them.

2) Stored procedures improve security

By only permitting data retrieval through stored procedures the impact of inappropriate or erroneous queries against the tables is greatly reduced leading to improved data security. Hmm. The only security that stored procedures provides is through the login credentials of the connecting user. If the application can delete users, then the connecting user can do so, if only by calling the stored procedure instead of typing in the SQL.

3) Stored procedures are faster than application SQL

While stored procedures are often considered to be the equivalent of assembly language for an RDBMS, the perceived performance gain is not always met. Almost all RDBMS implementations provide a caching mechanism such that the first time a query is encountered it incurs a performance penalty while it is parsed and compiled. Subsequent calls can immediately take advantage of the pre-compiled version and it doesn’t matter if this comes from within the RDBMS or the calling application. There are some very clever people behind these compilers and they will create a near optimal SQL statement. I’ll assume that the network lag for actually transmitting the query is assumed to be negligible.

For completeness, the case of stored procedures for update routines should be mentioned. An application can easily generate optimal update SQL for a single field involving a single parameter. In contrast, a stored procedure implementation will normally end up having to provide an “all fields” update mechanism with nullable parameters. This in turn leads to null checking logic within the stored procedure that is technically unnecessary.

Also important is that a stored procedure can take advantage of specialized functions available in the database that a generalized mapping solution, such as Hibernate, cannot emulate. There is, of course, the possibility of hard coded native SQL in the application but this should be frowned upon.

4) Stored procedures provide a single point of update

Given that all retrieval operations come from stored procedures, then any change to the implementation of those stored procedures is conducted behind the scenes and does not affect the applications that depend on it. Thus exclusive use of stored procedures introduces a single point of update should refactoring need to take place.

I agree that this is true up to a point, but consider the situation when an additional column is added to a table that needs to be included in the application. All stored procedures will need updating to include the new field, which will probably involve changing the signature of the stored procedure. This in turn will require that code in the application must be changed to support the new signature (even if the application does not make use of the new field).

Clearly this update process can be optimized, perhaps through versioned stored procedures or overloading with additional parameters, but the application is never completely isolated from the database. Over the course of time a large number of variants on the same theme may appear if multiple applications are served through the same API.

ORM frameworks like Hibernate overcome this by only including the required fields in their queries and relying on nullable fields being present. This helps to decouple them from changes to the underlying resultsets due to additional fields.

5) Data remains consistent because of stored procedures

A common argument for the use of stored procedures maintains that there is a guarantee that the data will always be consistent. This comes about due to a misunderstanding in the use of the term consistency. Consider the case when an application calls a sequence of stored procedures in order to fulfill a business process, but due to a coding error the last one is missed out.

The database is consistent from a mathematical point of view, but from a semantic point of view it is incomplete. If this sequence of calls was entirely within the application then only the developers can be blamed for the semantic error, whereas by forcing stored procedures then it is difficult to apportion blame – the database didn’t complain so it must be right.

In response, the DBA will necessarily have to include business logic in the stored procedures to enforce semantic consistency which negates the reasoning behind using stored procedures in the first place.

6) Stored procedures are application neutral

Once the API is written for the schema then any application can make use of them. Queries buried within JAR files tend to bind an application to a particular object model that may not be suitable.

This can be overcome by ensuring that application queries do not stray outside of the persistence layer, and that the persistence layer is designed against the specific requirements of the application which uses it. Reusable DTOs can then be ported to a common persistence framework but this does lead to a fair amount of complexity within the persistence layer dependency tree. I’d suggest using dependency management frameworks such as Maven or Ivy which can manage this transparently for you.

In order to make this approach work in the pure stored procedure approach, it is necessary to ensure that no knowledge of the database schema is present within the application. Table names should not be available, and columns and their types should be mapped internally to arbitrary names that are presented only through resultsets from the stored procedures. Therefore the application targets an abstract model presented by the stored procedure API which will completely decouple the application from any changes made behind the scenes.

Clearly, the creation of such an abstract model, in addition to that of the schema, is no easy task and the benefits it brings may be questionable. Is it really worth the effort?

7) Stored procedures make it obvious where the SQL is

Stored procedures benefit from easy and obvious navigation – they are in the database API packages. In addition, all databases provide a wealth of tools to both profile and subsequently optimize the queries as part of the development process, rather than as an afterthought when performance degradation is noticed.

In the application, the actual SQL being run is buried in the depths of the mapping framework and usually only appears in the debug logs. The location in the application code where this query is constructed is not always clear – it should be in the DAOs but can be hidden in annotations on DTOs. Consequently, it takes some time and skill to identify where the query is being constructed, and how to alter the structure to make it more efficient. However, by following well known design patterns for entity mapping, these inefficiencies can be ironed out early and permanently.

8) Hibernate is slow because it makes unnecessary calls

Hibernate can be made slow through the inappropriate use of eager fetching strategies where unnecessary associations are explored. This is known as the Dredge anti-pattern. If Hibernate is configured to only use lazy fetching then it can provide the optimal solution for data retrieval, outperforming stored procedures in some cases.

This bold statement needs an example to back it up. Consider the situation of a 3 way relationship between User, Role and Authority. The requirement is to fetch all Users with a first name of “Bob” (a set of 10 Users), along with all their Roles and Authorities. Hibernate can build this object graph in 3 queries: one for each entity followed by application side merging.

In contrast, a pure stored procedure approach will require a routine to filter on the User table to get the ones with the correct first name. Then the Role fetch routine will have to be called to get the Roles for each of the Users, then the Authorities fetch routine will have to be called for each of the Roles. This could lead to a very large number of procedure calls.

Clearly, an alternative would be a single procedure for that specific purpose that returns a union of the tables with inner joins as required, and an assortment of parameters for the filtering. Even then, the resultset will still have to be mapped so the gain from using a stored procedure becomes questionable, especially when the equivalent Hibernate query is 2 lines of obvious code.

9) Hibernate is slow because the SQL generated by the Criteria interface is not consistent

It has been said that Hibernate can also incur a performance hit if all queries are constructed through the Criteria interface instead of directly in HQL. The argument posits that this is because every time the query builder code is executed, say in a DAO, Hibernate will generate new alias names for the tables in the query. In Oracle this means that every time a new Criteria-based query is run, the database must create a QEP – query execution plan – as it is unable to match the SQL it has been given to any in its cache. Creating the QEP can take 30% of the time it takes for Oracle to respond to a SQL statement, so for the second and subsequent executions of the same (but for alias names) SQL statement, Criteria has a built-in overhead that makes it 50% slower than direct HQL.

This is no longer the case with Hibernate 3.3 and above. If it was ever true at all is in doubt since the Hibernate team would certainly aim to create optimal SQL wherever possible. Independent tests demonstrate that the same query is generated by the Criteria interface after repeated calls spanning transactions which is the equivalent of running the application under load. In each case the query remained identical and was therefore able to be cached by Oracle.

There is one grain of truth, though, in that it is necessary to create the query every time using the Criteria interface, whereas using named queries defined in HQL allows for precompilation during application start-up. However, this needs some perspective. The time taken to create a simple “between” query using the Criteria interface is approximately 3ms on an average PC. Embedding HQL within the application is not a good alternative since it does not lead to an intuitive mechanism for maintaining queries with differing fetching strategies, and so the Criteria based approach is deemed the better of the two.

10) Hibernate is difficult to learn

OK, now we’re getting desperate. Constructing a good persistence layer is difficult, regardless of the underlying ORM framework being used. Hibernate provides a wealth of useful tools to reduce the complexity of the resulting layer. In addition there are many examples of how to use Hibernate to solve various persistence problems. Examples include mapping a many to many relationship with attributes; creating an object hierarchy within a single table using discriminators; and managing lazy loading scenarios outside of the persistence layer to name but a few.
Given a sufficient set of examples and a good ER diagram, any reasonable developer should be able to map out DAOs and DTOs for an arbitrary schema in a matter of hours.

Conclusions

Hibernate should be default choice for persistence operations.
When it comes to mapping and then persisting large graphs of DTOs the amount of code that needs to be written is minimal, frequently only requiring a single line of code. Careful construction of DAOs and DTOs using Java generics to enforce type safety further reinforces what is already an excellent persistence framework. The vast majority of work will be conducted by developers in the application domain, using object oriented programming techniques, so it makes sense to provide them with an object persistence model that reduces the complexity of the application.

9 top tips to understand new code

You’ve just started at a new company, or have been introduced to a new system,  and you need to get up to speed quickly. Here are some proven tips and techniques to get you started.

1) Show no fear

Most developers are nervous when they first encounter a new system, but just keep in mind that it’s only code. There is a structure involved, however convoluted and apparently incomprehensible it may appear at the start. Once you start exploring this new realm of code you’ll be able to gain greater understanding, and if you’re the only one who knows it then your position within the company becomes much more valuable.

2) Identify the principal technologies involved

Have a quick scan over a few modules in the system. If you recognise various patterns that indicate classic design approaches (e.g. Spring, EJB, OSGI, Hibernate) then you’re in luck – you can make rapid progress. If not, then don’t despair. It just means you have to do some reading. And if you don’t do the background reading then you’ll be floundering in no time trying to wrap your head around a seemingly incomprehensible approach to coding. However, investing that time in yourself will mean that you’ll recognise patterns and will be able to start making progress.

3) Leave breadcrumbs for yourself

As you step into the code and start exploring the dependencies among the various objects it is important to build up a trail of breadcrumbs to help you find your way back. Just keep a list of class names with a short note of what they apparently do, along with some notes on possible future refactorings . Perhaps some simple diagrams here and there. They don’t have to be a full-on UML representation, just a quick scribble in your notebook is sufficient to understand data flow and immediate class dependencies.

4) Add comments

As you go through the code add comments so that you leave the system in better shape than you found it. The next developer to follow in your footsteps will definitely thank you for it – so long as the comments are accurate and to the point. Don’t comment the obvious: “Setting km to 111″ is redundant – any developer can see that. Instead, reserve comments for the less obvious: “Using an approximate value for kilometres in a degree of latitude – REFACTOR with method?”. This identifies the overall context of the value, and raises the possibility that a method could be employed to return differing values depending on the longitude.

5) Explore the database

Nearly all applications attempt to model their database in some manner, and the database is usually the cleanest part of a system. After all, the database is likely to be reasonably relational in nature and there are a lot of tools available to allow exploration of a schema visually. By exploring the database the principal entities involved in the system can be identified and this in turn provides some insight to starting points within the code. Of course, if the database is a mess, imagine how awful the code that relies on it is likely to be.

6) How does login work?

Almost every system has some kind of login process and since this does not vary an awful lot conceptually it provides a good starting point. In understanding the login process you will have a better idea of how data is taken from the user interface and the database; compared in some kind of business logic and how the result is output to the user interface.

7) Write a test

Once you feel confident navigating around the code base, the next step is to try to add some code. The safest way to do this is to write some code that will never get released to live, but serves a useful purpose: a unit test. Pick a class that has few dependencies and then write the unit test. Do not alter the class under test. Your objective here is not to rewrite the existing code, just to understand how to introduce the unit test framework, or use the existing one.

8) Implement a small change

Pick an innocent looking change from the task list. Ideally, this should be something straightforward like a user interface correction or a small extension to existing code. Then implement it in very small increments, with a unit test covering your efforts as you go. Do not make big sweeping code changes, or refactorings that affect a large number of dependencies. Keep it small. Keep it simple.

9) Break dependencies

Once you’ve got some confidence with the code and you understand (somewhat vaguely) how it all hangs together, then it’s time to identify and break dependencies. This means finding a way to break tightly coupled code into smaller fragments that provide better encapsulation. Usually this means following a dependency injection (or Inversion of Control) design pattern. There are several available refactorings that help with this: Encapsulate Field; Extract Method; Extract Interface; Extract Subclass; Extract Superclass and so on.

How to be agile when all about you are not – part 5

In part 4 of this series, I explained how continuous integration is used to automate the build process. In this article, I shall go on to explain how to further apply the DRY principle with fully automatic standardised build processes that generate code for you.

One of the biggest problems with build scripts is that they are tailored to a specific system and cannot be reused. Furthermore, every developer (and IDE) has their own ideas about how a project should be laid out; where compiled code should be placed and where the dependencies are stored. In some cases there are very good reasons why code is where it is, but more often than not it is because of some kind of workaround to overcome a limitation of the build process.

For a long time, Ant was the king of the Java build process. It is present in just about every mainstream IDE that supports Java and is a solid and reliable workhorse in the development process. However, Ant has many limitations (dependency management and script standardisation being two of the main ones) which is why the developers of Ant moved on to create Maven.

The idea behind Maven is to provide a unified build process that can be applied to just about any project of any size in an intuitive manner. It favours certain conventions that if followed makes your project immediately accessible to anyone else who knows Maven. Learning curves drop considerably for new developers joining the team since they will already know where to find code and how the build process works. Maven is extensible via plugins so that esoteric build requirements can quickly be catered for, and a large number of standard plugins are available for free. These include plugins for creating EAR files, automatically generating code, linking back to Ant scripts and just about every other purpose you could think of.

So how is this in line with the DRY principle? By providing a single point of reference for all project build information that is defined in a consistent format: XML. When I say all project build information, I mean it. Apart from the obvious “project name” and “version” there is a plethora of other information: who the developers are along with contact information; how to run automated tests and where to publish the results; how to generate the javadocs and where to publish them; how to manage versioned releases; how to generate the overall website associated with the project and where to publish it; and, most importantly of all, how to manage the project dependencies.

And in that closing entry lies the distillation of the DRY principle: a single point of reference for all the dependencies of the system, along with their versions so that they do not need to be stored within the project any longer. For many developers this comes as a great relief because there is no need to work out which version of, say, cglib.jar is copmatible with both the Spring 2.5.5 and Hibernate 3.3 frameworks. Instead, all Maven needs to know is what version of Spring you’d like, and which version of Hibernate and it works out the rest.

Maven also provides a very simple but powerful mechanism for avoiding repetition with the project file itself. For example, if the project has an overall dependency on Spring this is actually implemented as a small collection of dependency entries containing the precise modules of Spring that are needed (say core, transactions, web and MVC). If done without thought then there will be 4 places where the version, say 2.5.5, is referenced. However, the designers of Maven have provided a properties mechanism so that this repetition is confined to the name of a property that contains the appropriate version. Thus, to change from 2.5.5 to 2.5.6 all that is required is a simple change in the value of the property and all places where that property is referenced will be automatically updated. In a large build with many modules this saves an enormous amount of work and creates a very orderly build system.

At this point you have a very flexible and easy to maintain build system and a comprehensive set of automated tests that is providing protection against brittle code. So what next? Plenty. You have an agile build process, but you probably don’t have an Agile approach to design and implementation. For that you need to follow the trail “How to be an agile project manager” (coming soon).