A closer look at Kontolino

Two weeks ago I announced the public availability of Kontolino!, our new online service for the accounting needs of small businesses and freelancers. These two weeks have been quite exciting, because this was our first publicly accessible web server project that was done internally from A to Z. We currently welcome a few new test package users every day, and we already receive feedback of all kinds. Not bad for a system that’s aimed at a market that is quite mature and saturated. So things are still exciting and I am quite sure it will remain like that for a while.

Today I want to share with you a little insight of what technologies we use for Kontolino! and try to explain a few decisions we made.

Eating your own Dog Food…

We’ve helped several customers convert their existing Smalltalk applications from Fat Client to Web server architectures over the last 10 or more years. Some of these projects were based on Seaside, some even on older technologies like Smalltalk Servlets on top of VisualAge Web Connection. But we never had to cover the full range of technologies and products from Database to Web Server and Linux administration. There were always people in some department responsible for some things that they’d handle for us or  we could just follow their (seemingly nasty, from time to time) rules to make it right.

Not so this time. We had to go all the way through the stack and find out about Database Backups, Security Models, Firewalls and stuff, Linux distros, blogging software and last but not least all things web and Apache. What a stack. But we made it, we’re live!

From the beginning, our goal was not to use the shiniest, sexiest technologies alive or make it the coolest thing ever. Instead, our accounting system should be easy to use, supporting and intuitive. Stability was ranging high on our priority list. I can imagine many things I’d rather do than learning all about db recovery during a system crash tuesday morning at 3 am. So we had to find a mixture of technologies, tools and frameworks that we could handle, either by learning them or by bringing a backpacker full of experience with us. Even if web apps can be a lot cooler and databases these days are NoSQL for all those cool new reasons. The problem here is: if your new to a lot of things, you have to keep a few of the old ones that still are good enough to do the job, while the cool kids out there use more modern tools.

Use what you know, learn what you need

So we ended up with our technology stack, which, we think, is cool enough to achieve what we need and hot enough to be lots of fun and allow fast evolution on a stable base.

This is why we chose Smalltalk. Smalltalk, as you might know, is almost 40 years old. And still it is one of the most sophisticated development environments that I know of. The nature of live objects that you can watch and modify as they perform their task, the level of interactivity the Smalltalk debugger provides are still unmatched, even in multi-billion products like eclipse. If you think Firebug is a cool way of debugging JavaScript, you probably missed a lot: you should definitely take a look at a modern Smalltalk implementation and think again. Of course we also chose Smalltalk because it is a development that’s fun for us and makes us feel like we can dance around much larger teams building functionality while they work on making their compilers happy.

In our case, we had tons of experience with VA Smalltalk, one of the environments that was very late to jump on the Seaside train, but had been very early on the web technologies hill with a rock-solid Server Smalltalk framework that carries quite some Web Services and HTTP processing on its shoulders. Seaside still is not fully supported on VA, because its VM is not capable of supporting Continuations. But it turned out you can live quite well without them, even if you need to sail around some of the niceties of Seaside. On the other side, VA Smalltalk’s versioning and configuration component named Envy/Developer is a real killer. I miss it whenever I work on anything else. Combined with a few home-brew build automation tools and with the help of Jenkins, you can do wonderful things to build and deploy instantaneously or at least extremely often. I must admit it took quite some work to get there, but we had been doing this for several customer projects over the last few years.

Believe it or not: Seaside, the web framework we use in Smalltalk, is about to turn 10 next year. And we’ve used it quite a lot since about 2006. So we were good enough in it do build a web based system while we had to learn a lot about the underlying infrastructure, like Apache and firewalls and such. Seaside shows its age when it comes to deep integration with client-side JavaScript code, but being written in Smalltalk, we found ways to tightly couple some jQuery-plugins with Ajax-Callbacks on the server. We feel our GUI could still be better, but hey, would we be good developers if we were happy?

Let’s now look at our database. Nowadays it seems like a web based system has to use one of the cool new NoSQL databases, because they scale endlessly, are brand new and come with so many cool new features. Sorry guys, we didn’t need most of that. In Accounting, you end up with an object model where everything is linked to everything else.  The model doesn’t lend itself very well to the notion of documents. You have to make cuts through your object model to define document borders and save references to other documents. You are pretty much on your own to do this if you choose a document database like couch or Riak. So we tried to make a few comparisons of how we’d have to manage our data in a document store compared to a relational db. We found many possibilities that would be interesting for our project in couch db, but in the end we decided to stay with relational for two main reasons: existing know-how and some open issues about partitioning of our data.

We’d love to use PostgreSQL as our database, as it is very stable and reliable in our experience, and it is free and open. Unfortunately, VA Smalltalk doesn’t ship with supported PoestgreSQL-drivers and we were somewhat in doubt about using ODBC on Linux to connect to PostgreSQL. So we were somewhat limited in choices. VAST supports Oracle and DB2, so we had to choose between these two. We decided to use DB2, because we can start with the free DB2 Express-C edition, which is free for commercial use. If we need more steam, we can always move to another DB2 edition which is not free any more, but comes with rock-solid features and a lot of performance tricks as well as scalability options.

The logical choice for an O/R mapper was GLORP. It is more or less the gold standard of O/R mappers on all Smalltalk dialects. It is based on a design that was directly derived from TOPLink, the once most prominent O/R mapper in the Smalltalk world. So far our experiences with GLORP were very good. It’s quite performant and most of the bugs we found in it could either be worked around or even fixed quite easily. We also encountered a few interesting effects of O/R mapping in general and we had to learn a lot about the proper use of relational databases with objects. The possibility of migrating data and searching for problems using SQL has saved us lots of effort many times.
But let’s face it: Glorp is almost undocumented. We had been working with O/R mappers like Unity; VASER or POLAR in Smalltalk or hibernate in Java in the past, so most concepts of O/R mappers were well-understood. The Glorp discussion group on Google Groups was also a very helpful resource. I guess we can also claim to have helped stabilize the Glorp port on VA Smalltalk by contributing a little code and bug reports. You can also read some Glorp related posts on this blog.

The infrastructure

We’re hosting an a virtualized Linux Server that is currently only running with 2 CPU cores and 2 GB of RAM. But this server is very flexible and we can increase all parameters from cores to RAM and disk space. There are much cheaper options to host a web application out there, but we tried a few hosters and several models from VServers to root servers and found a lot of different problems. The worst problems were when we tried to install DB2 on a virtual server and failed, and even IB; support couldn’t help us, not to speak of the hoster…

So we are happy to pay a bit more and get the service and stability we need. If it were only for me, I’d switch to something cheap, but we’re handling customer data and need to be available as much as ever possible.

Our web server is Apache. Well-documented, scalable, can do a lot more than we need, hard to learn sometimes, but it does its job very well so far. One of the main arguments for Apache was that we’ll hopefully have to scale up any time in the future and we can use Apache httpd to introduce load balancing with sticky sessions, which is needed if you work with a stateful session manager like Seaside.

Does Kontolino! Scale to the billions?

You know, I somewhat dream of the day when we need to worry about this question.
So far, our testing shows we can handle quite a few users with one single Smalltalk image, because our architecture allows us to put some of the most heavy lifting to the database. Accounting is about reducing a list of accounting list to the relevant entries and calculate a little bit. So we do a lot of searching, grouping, summing up and such. That’s what SQL databases are good at. So we can move workload out of Smalltalk for some tasks and use these CPU circles for something else. The SST framework is quite effective and does a good job of kicking off Smalltalk green threads for answering http requests.

Our plan is to add Smalltalk images and work with Load Balancing to make things scale. And I hope we’ll have to address this problem soon. For now, the feedback we get is that Kontolino! is very snappy and comfortable to use. We’ve also identified a few spots where we need to work on optimizations, but luckily these areas are in corners where you don’t work often. But we’re working on that.

So what is this post good for?

I don’t know, to be honest. It addresses a few of the questions that I usually ask myself when I hear from Smalltalk colleagues who put up a web service. Feel free to ask questions in the comments if I missed something. If I don’t want to talk about some specific questions, I’ll let you know 😉