Glorp Wisdom (pt. 313): Never make two collections exclusive that can share objects

(Please note: the problems described here are not limited to Glorp. This is an issue to keep in mind in all O/R mapping technologies, not only in Smalltalk. What I’m talking about here is also relevant in other languages and frameworks. It just fits nicely into the Glorp Wisdom series)

Some days are even worse than you might think (see the last installment in this series), because they make you shake your head and pray that at least this last stupid error was not yours.

In this case, it was. As I’ve mentioned before, marking a OneToManyRelationship in Glorp to #beExclusive can cause problems if you want to move the object from one such collection to any other collection or even a 1:1 relation.

Looking a bit deeper, this error is quite pardonable compared to an even dumber one:

Never mark two (or more) OneToManyRelationships as exclusive if they could ever share objects.

Why, you ask? Glad you do, because it makes me feel a bit better. Because as I said in pt. 312, teh object will be relentlessly deleted if it is removed from one of the two collections. No matter if you intended it to remain present in the other one, Glorp will be ruthless (because you told it to be, not because it is a bad guy) and kill it. Forever.

Okay, you think that’s obvious and clear and not worth mentioning, and this would never happen to you. I bet you are wrong!

Because object models evolve. What once was clearly a case of “if it’s not referenced by this object, there’s no use in its existence, ever” may one day look completely different. You may add some 1:m relationship for performance reasons, or you may simply look at another aspect of your business model for a new feature and have the same idea: “well, if it’s removed from here, than it cannot be anywhere else”. Unless it can for reasons that are long forgotten or just not in your focus today.

After the experiences I’ve had today, I’d tend to say #beExclusive should be avoided as long as possible. It does make sense in some cases, but only if an object in an exclusive relationship really is onle related to its containing object and cannot be referenced from anywhere else. And even if you think that is clearly the case in your project, be careful!

Why do I say such stupid things? Because if you have a few thousand unreferenced objects in your database after a while of operation, you can always delete them or simply let them linger a little longer. Storage is cheap. Maybe you’ll finde a bug one day and be glad there are ways to recover their references and build some highly complex SQL code to get them back. An unreferenced object in the DB will not disturb your application’s operations and not get in your way.

If Glorp, however, deleted too many objects because of your mapping error, you may have trouble recovering any data. And it can take months and years to find this problem.

I’m in the flow today, so I’ll give you another word of wisdom for free:

Always act as if you had no relationship mappings, foreign keys or contsraints like ON DELETE CASCADE and clean up your backpointers and stuff in the application instead of relying on some “obscure” database functionality. Clean up on the object side and make sure objects that shouldn’t reference an object, really don’t, rather than assuming the database will do it anyways.

If you follow this rule, there are multiple benefits:

• Your objects in the image reflect the situation in the database much better. The objects in the image may look good before and after a commit, but once you read them back in another session, the situation may be completely different
• Hunting for issues becomes much easier because you can watch what is being de/referenced in the debugger. There is no magic hidden in an ORM layer or even the Database (which is not even visible in an SQL log…)
• Your application is better suited for moving it to other storage techniques, be it simple serialization, NoSQL databases or an OODB system, because these all have no equivalent for constraints like #beExclusive

 

Glorp Wisdom (pt. 312): Don’t reuse objects in exclusive relationships!

There are days when you hunt for a bug and almost are close to giving up. On some of these, it’s best to go home, play with your kids or watch a good movie and start again the next day.

On some, however, you finally find out teh whole problem was just a case of programmer’s stupidity at work.

I won’t tell you what kind of day today is for me, but I’ll share some words of wisdom with you:

Don’t use objects in an exclusive OneToManyRelationship and move them into another collection. You won’t like the results.

Want more details? Okay, my friend, let’s sit down and I’ll tell you all about it.

In Glorp, you can define a 1:m relationship to #beExclusive. This makes a lot of sense (that’s why many other ORM Frameworks have the same ability, maybe named a bit differently). What it means is that if an object that holds on to an exclusive relationship (or better: a collection of related objects) and gets deleted, all these objects will be deleted as well. Exclusive in that context means: the objects in this collection cannot exist withou the objext that holds them.

There is, however another way to look at it: it also means that if the programmer removes an object from this collection, it is not going to exist on its own any more. At least not after a commitTransaction.

So far, all of this makes sense.

Enter Joachim (you may guess right now which kind of day I had today).

Because all of this also means that there is aboslutely no sense in moving this object into another exclusive collection. Because you said it’s doomed. It’s gonna be doomed, no matter what. The object may be reachable from other collections or stuff, but an exclusive collection says: if you leave me, you die. It really is that simple. Glorp is gonna delete it, and never going to update it to reflect its membership of another collection. That would make sense if the object wasn’t coming from an exclusive collection, but, hey, it does!

So if you use exclusive collections (and I do because it makes sense in my business case), always take care what you do with your business objects. Moving it to another object’s collections won’t be reflected by updating it’s foreign keys, even if you do all backpointering and stuff correctly. Glorp’s answer will ALWAYS be a DELETE statement.

So moving an object from one exclusive collection to another always has to be done by removing the original object from the first collection and creating a new one for the other. You may copy references and attributes, even non-exclusive OneToMany collections to this new instance, but the object itself has to be a new one.

Issuing a REORG TABLE command to DB2 from VA Smalltalk (and Glorp)

You may have realized already that I misuse my blog and therewith you, my valued reader, as a swap space for small and maybe not so small little tricks I find in my day job from time to time.

And here is one little thing I just learned about how to invoke commands in DB2 that are not SQL statements form VA Smalltalk, in this specific case I mean REORG TABLE, but there are many other commands for which this may be useful.

Let me give you a little bit of context on why on earth I’d need that. Real men and even more so real DBA’s would simply fire up their DB2 command prompt and solve the problem at hand like a man. A simple table reorg would never stop a real man from saving the world in a day…

But here’s my problem. I am working on a Seaside Application in VA Smalltalk that is to be deployed to a Linux server. This application uses GLORP for persistence and I added some code that changes the database tables on server startup whenever I deploy a new image version. So if I add a new attribute to some persistent class, I have to add a new column to the underlying table(s). Some changes to the object model or some optimizations also need changes to foreign keys, indexes or even primary keys. These changes often cause DB2 to stop doing anything before I Reorg the modified table (Hint to IBM: Maybe that could be automated. The error message already tells me I need to Reorg, so why doesn’t it just do it for me???). The Error after such a change looks like this:

[SQLSTATE=57016 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0668N  Operation not allowed for reason code "7" on 
table "MYTABLE".  SQLSTATE=57016
 [Native Error=-668]]

Yes, you’re guessing right: this error has made my life harder than I wanted it to be more than once in the past.

So one step in these schema migrations often is to change tables and then move data around, add foreign keys and stuff. In theory, that’s not too hard (I’ve learned a whole lot about this stuff from one of my friends and customers, hi Peter!). Unless DB2 gets into my way and tells me now that I’ve changed the primary key, I need to reorg the table first before I can change data.

Unfortunately, REORG TABLE is not a normal SQL statement. It is not intended to be used by normal SQL users and therefor cannot be issued just like a  normal SQL statement. Here’s what you get from DB2 if you inspect myGlorpSession accessor executeSQLString: 'REORG TABLE schema.tablename':

AbtError:  rc=-1 for '42601' in an AbtIbmCliCSDatabaseConnection at (24.04.2013 15:50:41)  '[SQLSTATE=42601 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0104N  An unexpected token "TABLE" was found following "REORG ".  Expected tokens may include:  "JOIN <joined_table>".  SQLSTATE=42601
 [Native Error=-104]]

So this meant whenever I wanted to change a table’s primary key or add an index, I had to start the server once, stop it, use db2 to REORG tables by hand and start the server again. The steps of modifying the indexes and the following steps had to be separate migrations, each of them in their own transaction. Quite annoying.

But I found a solution to this problem. Real men, of course, knew it already. You can use a normal CLI client and issue admin commands by wrapping them into this:

myGlorpSession accessor executeSQLString:  'CALL ADMIN_CMD (''REORG TABLE schema.tablename'')' .

With this I can reorg a table first and then modify data in one single step during my server startup.

Doesn’t sound like a big deal? You’re wrong. This was a real hurdle to simple deployment. Just imagine you have to redo the same stuff on a development machine, a test server and a production server over and over again. And you must remember when to do what in the right order. Especially on a production server which shouldn’t be offline for too long, this is very important.

Before you ask: I am aware of the AUTO_REORG parameter, but as far as I know, Table reorganization can only run in an offline window. That’s not exactly what I am looking for. I need to do it in the middle of a migration script, not somewhen later tonight …

Pharo 2.0 ist offiziell verfügbar

Das Pharo-Projekt, das einst mit dem simplen Anspruch startete, die beste Smalltalk-Umgebung zu entwickeln und als open-source kostenfrei zur Verfügung zu stellen, hat auf diesem Weg einen neuen Pflock eingeschlagen. Die Version 2.0 wurde gestern freigegeben. Es wurde viel aufgeräumt, vereinheitlicht und verbessert, aber vor allem wurden weit über 1000 Bugs gefixt:

We are proud to announce the release of Pharo 2.0!

You can find information about Pharo at:
http://www.pharo-project.org

About this release
——————
All in all, there were over 1600 issues treated in the issue tracker
and 1350 improvements integrated into 2.0.

Pharo ist die Heimat diverser Projekte, die in der Smalltalk-Welt und darüber hinaus für Aufsehen gesorgt haben. Als erstes fallen mir da natürlich Seaside und einige darauf aufbauende Tools (Pier, Magritte etc.) ein, sowie Moose mit seinen Teilprojekten. Nicht zu vergessen DBXTalk, das eine komplette Infrastruktur rund um GLORP und openDBX für Pharo aufbaut. Viele davon sind auf anderen Umgebungen ebenfalls verfügbar, aber werden in Pharo gepflegt.

Ein Ansatz der Pharo-Gemeinde, der von Anfang an im Mittelpunkt stand, war, dass man sich von anderen Plattformen abspaltet, und seinen eigenen Weg geht. Das hat eine ganze Menge Vorteile, etwa eine sehr hohe Schlagzahl an Innovationen, aber eben auch Schattenseiten. Es gab und gibt immer wieder Maintainer von Projekten, die sich entschieden haben, Pharo als Plattform nicht mehr primär zu bedienen, sondern dazu aufriefen, es möge sich jemand als Portierer bzw. Maintainer eines Ports nach Pharo bereit erklären. Letztes prominentes Beispiel ist Chris Muller mit seiner OO-Datenbank Magma. Das kann aber im Endeffekt natürlich bedeuten, dass, wenn sich ein Porter findet, mehr Zeit für das eigene Projekt bleibt, und somit kann auch dies eine gute Nachricht sein. Und wer weiss, vielleicht wird aus dem Porter eines Tages ein Hauptentwickler, und so wächst das Projekt noch schneller…

Pharo Smalltalk ist  bei weitem die Smalltalk-Umgebung mit der aktivsten open source – Gemeinde und legt ein enormes Tempo bei der Integration von Neuerungen hin. Es lohnt sich also ganz sicher, sich mit Pharo Smalltalk zu beschäftigen, wenn man sich für Smalltalk interessiert.

Pharo Smalltalk läuft auf Windows, Linux, Mac OS X, Android und iOS und deckt damit neben Squeak die meisten Plattformen ab. Der Code ist auf allen Plattformen unverändert lauffähig (man kann einfach das Image auf einen anderen Rechner transportieren und dort starten, ohne jegliche Anpassungen), sodass ein Portierungsaufwand nicht anfällt, solange man keine Plattformspezifischen APIs nutzt (was vor allem auf iOS und Android natürlich schwierig sein kann, vor allem wegen des nativen Look&Feels).

Wer also von Smalltalk schon gehört hat, und sich das irgendwie mal gerne anschauen würde, sollte sich unbedingt Pharo 2.0 herunterladen und loslegen. Auf den Projektseiten von Pharo sind viele Links zu Einsteiger- und weiterführendem Material.

Wir haben übrigens erst vor ein paar Wochen ein Interview mit Stéphane Ducasse auf unserem Podcast “Smalltalk Inspect” geführt, in dem wir über die Entwicklung und Beweggründe hinter Pharo gesprochen haben. Ausserdem haben wir viel über die Bemühungen des Projekts in Richtung Finanzierung und Weiterentwicklung von Pharo gelernt. Es lohnt sich also, auch da mal reinzuhören.

GLORP and “Conflicting values in rows”

Note to self: if GLORP keeps on insisting that your UPDATEs to a certain object really can’t be accepted due to ”Conflicting values in rows”, please start by looking into the following thing first:

Maybe you just changed the :1 part of a 1:n relationship without removing the changed object from the :n part of the relationship.

If that is not enough of an explanation, take this example: If you have an Order object with OrderItems, always make sure you not only change the #order instance variable of an item when you move it from one order to another, but also remove it from the Order’s #orderItems before you commit your transaction. GLORP will be a little upset otherwise.

Only after you’ve made sure that this is not the problem (I mean, *absolutely* sure), you can start developing another completely unorganized and fruitless plan for guessing about the causes

…and now for my Glorp on VAST error handling rant.

I’ve written in my previous post about a debugging session that took longer than necessary for two reasons: my stupidity and a combination of several unfriendly factors. So let’s continue our journey up through my walkback of today’s image crash.

Once I figured there was no Seaside problem in the crash, I scrolled further up through the walkback and what I found there really made me upset about the error handling in Glorp (I was also upset about myself and the fact that I had just spent an hour for nothing). Just about 1500 lines towards the top of the walkback was this:

ExceptionalEvent>>#signalWith:
  receiver = Exception: Database error
  arg1 = AbtError:  rc=-1 for '57016' in an AbtIbmCliCSDatabaseConnection at (12.12.2012 11:25:20)  '[SQ
LSTATE=57016 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0668N  Operation not allowed for reason code "7" on 
table "MYTABLE".  SQLSTATE=57016
 [Native Error=-668]]
'
[] in VADatabaseAccessor>>#loginIfError:
  receiver = a VADatabaseAccessor
  blockarg1 = AbtError:  rc=-1 for '57016' in an AbtIbmCliCSDatabaseConnection at (12.12.2012 11:25:20)  '[SQLSTATE=57016 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0668N  Operation not allowed for reason code "7" on table "MYTABLE".  SQLSTATE=57016
 [Native Error=-668]]
'

Holy Moly! So this was the cause of my problem. I had altered MYTABLE in some schema migration routine and the table needed reorg. This is easy to handle once you know it needs to be done: fire up some DB2 client and do a REORG TABLE MYTABLE.

But what is Joachim complaining about? It’s his job to know what he did to the database, not Glorp’s…

I am complainning about the fact that Glorp simply obfuscates a very clear error message. Remember what the walkback started with? If you forgot, here are the first lines of the walkback:

Walkback at 11:25:20 on 12.12.2012
Database error: 
    [] in AbtHeadlessRuntimeStartUp class>>#outputWalkback:process:  
    ...etc...

That’s what I am complaining about. I could have fixed the problem in just three minutes if the walkback told me what’s wrong.
DB2 tries hard to tell the stupid programmer what’s the problem. AbtIbmCliDatabaseConnection>>#fetchNextRowFromCursor:ifError: hands this exception on to its caller, and Glorp (or better, the VAST port of it) simply takes that message and puts it into the dustbin or keeps it as a secret. Eat this, programmer!

This is due to some strange error handling code that comes from the original Glorp implementation. It has to be adopted to VAST, because the adaptation that was done in the original Glorp code is defunct. I tried to fix that on my more than once, but only found another problem in VAST that completely befuddled me so I gave up. Whatever I tried, something didn’t work. That problem can be fixed very easily, and I think it will be integrated into VAST soon, but it isn’t yet.

So this is my rant about the current VA ST Glorp port and its error handling mess. Other than that, I must say Glorp is stable and works very well, even if the VAST version is a few versions behind.

Ah, before I forget it: After a reorg of the table it seems my application runs fine and I am in the middle of testing and fixing.

How to waste less time debugging a Seaside/Glorp application

Yesterday I packaged my Seaside Application for the first time on VA Smalltalk 8.5.2 and deployed it to a staging server.

And promptly – as expected, I got some errors: The first few were easy to find. One of them being a missing rule in AbtXDSingleImagePackagingRule (or some superclass) to include the new EsTimeZone code. That could be fixed by hand.

But this morning I spent quite some time searching for a problem in the walkback.log that didn’t exist. And this post is mostly intended for myself to remember next time. But it might also save you some time. The second purpose of this post (or, to be exact, the next one) is to underline why I think the VAST port of Glorp has a lousy adaption of error handling.

But let’s start at the beginning: My image crashed as soon as a web user logged on to the web application. The image exited with Error code 60 and wrote a walkback log. So far, so good. The first thing this tells me is that my error handling code is not perfect yet, because I should have seen an error page instead of an HTTP-503.

So I started reading the walkback. My usual way of doing so is to start with the beginning of the walkback:

Walkback at 11:25:20 on 12.12.2012
Database error: 
[] in AbtHeadlessRuntimeStartUp class>>#outputWalkback:process:  
    receiver = AbtHeadlessRuntimeStartUp  
    arg1 = 'Database error: '  
    arg2 = Process:Dispatch worker: 8{running,3}  
    temp1 = 'walkback.log'  
    temp2 = a CfsWriteFileStream
BlockContextTemplate(Block)>>#valueWithErrorHandler:oldHandler:onReturnDo:  
    receiver = [] in AbtHeadlessRuntimeStartUp class>>#outputWalkback:process:
   ... and so on

So one thing was for sure: this is another one of those useless error messages that come from Glorp. Unfortunately, I decided to skip step two of my usual

Glorp and 1:n relationships: a word of advice

Regular readers may remember a post of mine about an issue with Glorp’s commitUnitOfWorkAndContinue method. When you delete objects that refer other objects in an exclusive 1:n relationship, Glorp currently re-inserts those deleted objects in a later transaction. In the process of building a small test application that makes the error transparent, I stumbled upon some strange problems that seemed to be caused by problem with auto-increment (or sequence) columns.

But let’s start at the beginning. The idea was that I’d build a tiny application with one model class called TreeNode. Such a Node can have a parent and zero or more children, besides a title. Should be easy, right?

Wrong.

First things first:

VA Smalltalk 8.5.2

Instantiations has just released VA Smalltalk Version 8.5.2. As the version number indicates, this is more or less a maintenanc release with a number of bug fixes and enhancements.

But there are also new features in this version, the most important from my perspective are:

• Code Assistance now available in Inspectors:
  The addition of Code Assist was a productivity booster for me. I got so used to this little friend that going back to older versions of VAST felt somewhat awkward. I was constantly waiting for the suggestions to pop up. Getting available variables and methods in the Inspector makes life even easier now.
• Monticello Importer added
  This could be the beginning of a little wave of new open source contributions to VASTGoodies.com. Now that it’s become a little easier to load Smalltalk code from Monticello repositories, at least the lowest hurdle for porting code from Squeak/Pharo to VA Smalltalk is gone. Instantiations made the Monticello Importer available as a beta/preview feature some months ago, but now it is part of the product. If you wonder what the process of porting code to VAST using the Monticello Importer, you could go back to this post of yours sincerely and start porting code from squeaksource or, better, squeaksource3, right away, thus making our lives with VA ST a little more fun.
  I really hope Instantiations will also consider building a tool for the opposite way: we need to move closer to the rest of the Smalltalk world and we need the ability to contribute fixes to open source projects and should also be able to release VAST goodies for other dialects. You may not have realized yet, but there are quite a few nice goodies coming from the VA Smalltalk community that might be of interest to Squeakers or Pharoers.
• Support for Time zones introduced
  Oh, the joys of handling Dates, Times, Daylight Saving Time and Time differences. What else should I say?
• System>>#getProcessId added to retrieve the pid of the running process
  This one goes back to a question I asked on the VA Smalltalk support group. It can be very handy to know the PID of a running Smalltalk image, especially on a web server that needs to take control over what images are running and/or need to be stopped or restarted.
• GLORP updated to version 0.4.190
  Unfortunately, I have no idea what this version number means. To my knowledge, Glorp has the versioning scheme of the VisualWorks version it ships with. So this might still be some quite old Glorp version. I had to fix a number of issues in the Glorp version that shipped with VA Smalltalk over the last few months, but still I find Glorp to be a very powerful and nice framework, even though I sometimes find the code hard to understand (so I still have unresolved issues due to my inability to fix them). But it is good to see that Instantiations is updating Glorp and thus offers a current and powerful relational database framework for VA Smalltalk. Glorp sure is the current gold standard here in the Smalltalk world, since it is now supported on all major ST dialects

Where to get it?

VA Smalltalk 8.5.2 is available immediately and can be downloaded from Instantiations’ website. If you are a registered and/or paying user, you can simply download it using your registration details. If you’d like to purchase licences, you can either contact Instantiations or one of their resellers ( as it happens, my company is a reseller for Germany, Austria, Switzerland and Benelux countries) for more details.

Use VA Smalltalk for free?

Did you know VA Smalltalk is free to use for personal / evaluation purposes? Simply register with Instantiations for download credentials. Instantiations also gives away licences with email support for educational institutions and for open source committers. Visit their website for more details.

Glorp and #commitUnitOfWorkAndContinue

[Please note: This post has been updated on Sept. 13th, 2012. The solution that I suggested in the original version of this post completely messes following commits for objects that got read from the DB before this commit and are changed later. Consecutive commits simply ignore changes to such objects. You can visit the discussion thread on Google Groups to see why and what all of this evolves to. The problem, however, is not trivial in situations where objects get deleted and should cascade their deletion to dependent objects. I do suggest a change to Glorp in that thread and am currently waiting for Alan to comment on it. ]

I just spent a bit over a day hunting some strange problem in a Seaside-based Web Application that uses VA Smalltalk and Glorp. The problem was that it was almost impossible to do a clean delete of an object that resides in two OneToManyMappings.
First, let me mention that our application was prototyped on top of an object database and it ran just fine, so it was not an application issue.

The object has some removal code that removes the object from the collections in the objects on the Many-side of the relationship. We considered this business code, and therefor just left it in place and added a

self session delete: theObject.

to the deletion method.

We thought all was running well for quite a while, until we realized we have lots of undeleted objects in the DB tables which have a foreign key of NULL. So the application was not really affected by the problem, but still we’d like to not collect too much rubbish in our DB. Once we’re live, we want to keep things small and manageable, and data garbage is not a good thing to start with.

So what exactly happened? We had no idea until we tried deleting objects and fired off our debuggers. At first, all seemed fine: a Transaction would be started, the objects deleted and then the transaction would be committed. But still, after a while, we’d find new corpses in the table (in fact we had more tables that were affected).

So what the heck was going on? By accident I saw that teh very next transaction after the delete would re-INSERT the objects with foreign Keys set to NULL. The following transaction didn’t really have to deal with the kind of Object we had deleted, it could be completely unrelated, just run in the same GlorpSession. Uhh!

Then followed a day of playing around, debugging and wondering.
We tried leaving our deletion-business code out of the game,