Marten shares a source snippet on how to create a zip archive with VA Smalltalk:
Today I was in the need to create a zip archive in VASmalltalk and zip-code had been introduced with the new Monticello Importer in version 8.5.2 (application: MZZipUnzipApp)
Today I was on the hunt. And I finally shot the beast that killed hundreds of good men over the last decades or so.
Well, at least it seems it’s dead, and I don’t really know why. I know it’s dead and I was hunting for it, but I am not sure I pulled the trigger.
But let’s start at the beginning. Yesterday I had a shiny new version of my Seaside Application and I had cross packaged it for Linux and tested it on two different Linux machines. And it crashed. It crashed without much comment, the Server just didn’t respond any more, the process was dead and all it left for me was a vmtrap.log and an vmtrap.img file in the image directory. Funnily, it crashed very reliably, within only a few Web page requests, but it died in many different places. The vmtrap.log almost always indicated the image had gotten to some method that was not the same of any of the previous crashes.
The funny thing was that the packaged image of the day before ran nicely, without any crashes. It ran using the very same VM and all external libs and stuff. The only difference was that other image. The code changes were only in application code, no external calls, strange extensions to UndefinedObject or any evil class initialization stuff. Just a few bug fixes and stuff.
So to sum the situatuon up: the dev image on Windows ran, a dev image on Linux ran as well, the packaged image of the previous day ran – even on the same machines using the same path and vm and libs and ini and whatever. The VM crashed in various places, and I could not find any pattern why it may have crashed.
I even created two completely fresh XD images and repackaged and went through the .es files and whatnot. I also repackaged the image of the previous day and all was the same again: old one was good, new one was broken.
So I was somewhat out of luck. I asked the wizard of G and found a very old post that seemed to talk about the very same problem. It came up with the (crude?) theory that someone must have screwed bytecodes in the library (Maybe I should mention I use a Linux XD image for packaging while I develop on Windows, so it’s not related to any image problems. I throw images away very often to avoid such problems right from the start) and recompiling methods helped their problem.
So I thought there’s not much to loose. I fired up a Config Map Browser, browsed the Changes between previous day and current (incl. required maps) and simply changed all changed methods by entering a blank or newline somewhere in the Methods. Luckily, I hadn’t changed that much code yesterday, maybe 50 methods. Have I ever mentioned I really love envy?
And what can I say? I fired up the Linux VM, cross-packaged the new (unchanged, but recompiled) code and the problem seems to be solved. The packaged image is running and seems to be stable.
So what could have gone wrong? I am packaging once or twice a week and haven’t seen this ever before.
Ah, and before you ask: None of the methods I had changed were ever mentioned in the vmtrap.log files…
Very strange…
But here’s my theory (please take it with a grain of salt):
I remember two things that happened to me yesterday. One was that I added an instance variable with accessors (using RB) named ‘class’ which wasn’t a good idea, so I later renamed the variable (but not the Accessors) to ‘cssClass’. I know I later couldn’t version/release that class because it was “inconsistent withe the edition in the Library”, an issue that is easy to solve most of the times.
I remember I once had lots of variables and accessors to rename in a very poor code base and I’ve had similar problems with these accessor methods that obviously didn’t work right. The source code said it would be doing one thing, but the results of executing them was different… Back then, changing the accessor methods by hand and saving /compiling new editions of them solved the issue.
I am not sure this theory is any good and am not sure how I could ever build a test for this stuff or write a somewhat useful support case. I only hope I remember this post if I ever encounter such a strange problem again.
So what do we learn from this?
Es hat sich sicher schon herumgesprochen, dass die Entwicklungs- und Supportmannschaft von GemStone/S und GLASS sowie das darauf basierende MagLev für Ruby nun in einer neuen Firma zuhause sind. Das neue Unternehmen heisst GemTalk Systems und besteht aus der kompletten Mannschaft, die bisher an GemStone gearbeitet hat. Für das Produkt sind das sehr gute Nachrichten. GemStone/S war bei vmware irgendwie ein Stiefkind. Man musste auf der vmware-Website lange suchen, bis man GemStone/S fand. Selbst auf der zentralen Seite aller Produkte waren diverse Produkte aufgelistet, die ursprünglich von GemStone stammten (z.B. GemFire, SQLFire), nicht jedoch das eigentliche Flaggschiff.
Aus den Pressemitteilungen und Blog-Einträgen ist nicht zu entnehmen, wer die Investoren sind, die GemStone von vmware übernommen und GemTalk daraus gemacht haben. Auch auf der Website sind keine Informationen dazu zu finden. Dafür aber, dass vmware sich auch von den anderen Produkten wieder getrennt hat, wegen derer sie GemStone vor drei Jahren gekauft hatte. Diese sind ein paar Wochen zuvor in die Firma Pivotal ausgelagert worden und werden dort weitergeführt. Was das ganze für vmware für einen Sinn hatte, erschliesst sich mir nicht. Aber wer GemStone ein bisschen kennt, weiss, dass es schon in den neunziger Jahren mal ganz ähnlich zuging: der Börsenkracher Brokat aus Böblingen/Stuttgart hatte GemStone gekauft, um sein Portfolio mit GemStone/J (das es meines Wissens inzwischen nicht mehr gibt) aufzupeppeln. Glücklicher Weise ging Brokat erst vor sdie Hunde, nachdem GemStone mit viel Verlust wieder verkauft worden war. Auch bei Brokat hatte man den eigentlichen Edelstein von GemStone nie verstanden. Martin McClure hat auf der letzten Smalltalks 2012 einen sehr interessanten Vortrag zur Geschichte seines Arbeitgebers gehalten (hier gibt es das Video dazu). Um einen running gag aus diesem Vortrag fortzuführen: GemTalk Systems bleibt im selben Bürogebäude und zieht dort einen Stock höher 
Nun könnte man meinen, sowohl Brokat als auch vmware haben sich an GemStone verschluckt, weil Smalltalk einfach nicht mehr zeitgerecht ist und niemand Smalltalk nutzt, zumindest niemand, der dafür Geld ausgeben möchte. Also war es nur logisch, das Zeug wieder abzustossen, und die Perlen des Unternehmens zu behalten.
Schaut man genauer hin, ergibt sich dann aber doch ein ganz anderes Bild: Die Kundenliste von GemTalk Systems ist sehr beeindruckend. Nicht, weil die Namen darauf so bekannt sind und Eindruck machen. Vielmehr, weil dahinter Unternehmen mit sehr komplexen, datenintensiven und äusserst flexiblen, veränderlichen Abläufen stecken.
Wie sonst, ausser durch seine besonderen Eigenschaften gerade für sehr komplexe Abläufe und hohe Datenvolumina liesse sich erklären, dass es Investoren gibt, die eine Ausgründung von GemTalk Systems ermöglichen? Warum sollte jemand Geld in eine Firma stecken, deren Geschäft eine heisse Kartoffel ist, die vmware einfach fallen lassen möchte?
GemTalk hat eine beeindruckende Kundenliste, die – so wird gemunkelt – für einen attraktiven Cash-Flow sorgt. GemStone ist ein profitables Produkt, und das schon seit Jahrzehnten. GemStone war für Brokat eine Cash Cow und bei VMware war es nicht anders. Wie sonst liesse sich erklären, dass GemStone weiter existierte und das Produkt weiter entwickelt wurde? Wie sonst liesse sich erklären, dass vmware die kostenlose Verwendung von GLASS auch für kommerzielle Zwecke weiter gestatttete und sogar den Umfang der kostenfreien Lizenz erweiterte, das kommerzielle Produkt aber weiter führte? Wie sonst wäre zu erklären, dass erst unter VMware so recht Schwung in die Öffnung von GemStone hin zu open source-Projekten kam, sich Dale Henrichs zunehmend auch Themen wie Metacello und der Kompatibiltät von GemStone mit Pharo/Squeak widmen konnte? GemStone macht gerade einen sehr wichtigen Schritt: Weg von GemBuilder und der Notwendigkeit einer zweiten Smalltalk-Tools als Entwicklungs-Frontend hin zu einem kompletten Werkzeug mit einer eigenen (Webbasierten) IDE. Hier wird sehr viel mit Amber und Pharo gearbeitet und es gibt Initiativen hin zu einer Versionsverwaltung in Tools wie git oder subversion.
Der Mond über GemStone ist also ganz sicher nicht erblasst durch diese Ausgründung. Im Gegenteil, wir werden sicher noch viel Ineteressantes über GemStone hören in den nächsten Monaten. Die erste Etappe hierzu wird sicher die im Juni stattfindende STIC 2013 in den USA sein, auf der GemTalk dabei ist. Auch MagLev wird GemTalk weiterführen, und was uns Smalltalkern sicher am wichtigsten sein wird: es wird weiterhin ein kostenloses GLASS geben.
Die Ausgründung von GemTalk Systems steht nicht alleine da. Erst vor drei Jahren passierte bei Instantiations, dem Anbieter von VA Smalltalk, etwas ganz ähnliches: die Firma stiess ihr gesamtes Java-Produktportfolio samt Entwicklungsmannschaft an Google ab, und führte ihre Geschäfte als reine Smalltalk-Firma weiter (Joachims Small World berichtete). Seither hat sich auch an dem Produkt mehr getan, als in den Jahren zuvor.
Smalltalk scheint also nach wie vor eine Umgebung zu sein, in der es sich lohnt, zu investieren. Alle drei wichtigen kommerziellen Anbieter (Cincom, GemStone und Instantiations) scheinen profitabel zu sein und arbeiten fleissig an ihren Produkten. Auch im open source-Umfeld tut sich eine ganze Menge, angefangen bei Pharo, über Amber bis hin zu Squeak und GNU Smalltalk.
Um einen der abgeklopfteren Sprüche berühmter Persönlichkeiten zu bemühen: Die Gerüchte über das Ableben von Smalltalk sind noch immer völlig übertrieben….
Zum Abschluss noch ein paar weiterführende Links:
(Please note: the problems described here are not limited to Glorp. This is an issue to keep in mind in all O/R mapping technologies, not only in Smalltalk. What I’m talking about here is also relevant in other languages and frameworks. It just fits nicely into the Glorp Wisdom series)
Some days are even worse than you might think (see the last installment in this series), because they make you shake your head and pray that at least this last stupid error was not yours.
In this case, it was. As I’ve mentioned before, marking a OneToManyRelationship in Glorp to #beExclusive can cause problems if you want to move the object from one such collection to any other collection or even a 1:1 relation.
Looking a bit deeper, this error is quite pardonable compared to an even dumber one:
Never mark two (or more) OneToManyRelationships as exclusive if they could ever share objects.
Why, you ask? Glad you do, because it makes me feel a bit better. Because as I said in pt. 312, teh object will be relentlessly deleted if it is removed from one of the two collections. No matter if you intended it to remain present in the other one, Glorp will be ruthless (because you told it to be, not because it is a bad guy) and kill it. Forever.
Okay, you think that’s obvious and clear and not worth mentioning, and this would never happen to you. I bet you are wrong!
Because object models evolve. What once was clearly a case of “if it’s not referenced by this object, there’s no use in its existence, ever” may one day look completely different. You may add some 1:m relationship for performance reasons, or you may simply look at another aspect of your business model for a new feature and have the same idea: “well, if it’s removed from here, than it cannot be anywhere else”. Unless it can for reasons that are long forgotten or just not in your focus today.
After the experiences I’ve had today, I’d tend to say #beExclusive should be avoided as long as possible. It does make sense in some cases, but only if an object in an exclusive relationship really is onle related to its containing object and cannot be referenced from anywhere else. And even if you think that is clearly the case in your project, be careful!
Why do I say such stupid things? Because if you have a few thousand unreferenced objects in your database after a while of operation, you can always delete them or simply let them linger a little longer. Storage is cheap. Maybe you’ll finde a bug one day and be glad there are ways to recover their references and build some highly complex SQL code to get them back. An unreferenced object in the DB will not disturb your application’s operations and not get in your way.
If Glorp, however, deleted too many objects because of your mapping error, you may have trouble recovering any data. And it can take months and years to find this problem.
I’m in the flow today, so I’ll give you another word of wisdom for free:
Always act as if you had no relationship mappings, foreign keys or contsraints like ON DELETE CASCADE and clean up your backpointers and stuff in the application instead of relying on some “obscure” database functionality. Clean up on the object side and make sure objects that shouldn’t reference an object, really don’t, rather than assuming the database will do it anyways.
If you follow this rule, there are multiple benefits:
There are days when you hunt for a bug and almost are close to giving up. On some of these, it’s best to go home, play with your kids or watch a good movie and start again the next day.
On some, however, you finally find out teh whole problem was just a case of programmer’s stupidity at work.
I won’t tell you what kind of day today is for me, but I’ll share some words of wisdom with you:
Don’t use objects in an exclusive OneToManyRelationship and move them into another collection. You won’t like the results.
Want more details? Okay, my friend, let’s sit down and I’ll tell you all about it.
In Glorp, you can define a 1:m relationship to #beExclusive. This makes a lot of sense (that’s why many other ORM Frameworks have the same ability, maybe named a bit differently). What it means is that if an object that holds on to an exclusive relationship (or better: a collection of related objects) and gets deleted, all these objects will be deleted as well. Exclusive in that context means: the objects in this collection cannot exist withou the objext that holds them.
There is, however another way to look at it: it also means that if the programmer removes an object from this collection, it is not going to exist on its own any more. At least not after a commitTransaction.
So far, all of this makes sense.
Enter Joachim (you may guess right now which kind of day I had today).
Because all of this also means that there is aboslutely no sense in moving this object into another exclusive collection. Because you said it’s doomed. It’s gonna be doomed, no matter what. The object may be reachable from other collections or stuff, but an exclusive collection says: if you leave me, you die. It really is that simple. Glorp is gonna delete it, and never going to update it to reflect its membership of another collection. That would make sense if the object wasn’t coming from an exclusive collection, but, hey, it does!
So if you use exclusive collections (and I do because it makes sense in my business case), always take care what you do with your business objects. Moving it to another object’s collections won’t be reflected by updating it’s foreign keys, even if you do all backpointering and stuff correctly. Glorp’s answer will ALWAYS be a DELETE statement.
So moving an object from one exclusive collection to another always has to be done by removing the original object from the first collection and creating a new one for the other. You may copy references and attributes, even non-exclusive OneToMany collections to this new instance, but the object itself has to be a new one.
Well, a developer’s day is full of victories and waterloos. So I’ve had my developer’s day today.
After I had discussed my Server startup problems related to ports already in use, Marten came up with an explanation for VAST’s behavior on the VAST Support Group:
The reason for this behaviour is located in the method: SstTcpTransport>>#basicOpenListeningSocket and it seems to be correctly handled by VASmalltalk.
In this method a socket option (“SOREUSEADDR”) is set to true (if the configuration of the tcp network allows it, which is normally the case of VASmalltalk).
Hmm. Even if it’s not incorrect, I still don’t like it.
So my first waterloo was that Marten proved me wrong: this is not a bug or wrong behavior – from a very technical standpoint. I still thinnk the server pretends to be ready when it is not, and I don’t like it if my software tells me lies right into my face, especially if that even costs me lots of time…
The thread at the discussion group then continues with suggestions like changing the setting Marten mentioned in some very low level method for binding to a Socket. That is of course very deep down and you never know what you break if you open ALL TCP/IP Sockets as exclusive listeners.
So I decided to stick with my workaround that I described in my initial post on the subject (the reworked one, of course). It ran great – on Windows.
So here was my second waterloo. When I had XD packaged this and tried to start the same image twice on the Linux test machine, the Adaptor never left the #isStarting phase. The log contained lots of entries telling me we’re waiting for the adaptor to finish starting:
'2013-04-24 17:32:48,917: [INFO] Waiting for the Adapter to finish starting' '2013-04-24 17:32:49,519: [INFO] Waiting for the Adapter to finish starting' '2013-04-24 17:32:50,121: [INFO] Waiting for the Adapter to finish starting' '2013-04-24 17:32:50,723: [INFO] Waiting for the Adapter to finish starting' '2013-04-24 17:32:51,325: [INFO] Waiting for the Adapter to finish starting'
.. and so on. So the Socket behavior differs between Windows and Linux. On Windows, the retry loop was never run, the Adapter was not running immediately.
So I decided to extend my ugly workaround a little more with a retry count. I hate the code as it looks now, because it reminds me of really dark days in my programmer career. I’ll show it to you nevertheless, because it may be useful for others as well (I know, it’s not rocket science, but copying it from here is faster than thinking about it – you’re welcome!):
startServerAdaptor "self startServerAdaptor" |adaptor maxRetries retries| maxRetries := 5. retries := 1. [ (adaptor := WASstServerAdaptor port: self port) start. [adaptor isStarting and: [retries <= maxRetries]] whileTrue: [ EsLogManager info: ('Waiting for the Adapter to finish starting, retry count = %1' bindWith: retries asString). (Delay forSeconds: 0.5) wait. retries := retries +1. ]. adaptor isRunning ifFalse: [Error signal: ('Adaptor is not Running - maybe port %1 is already in use?' bindWith: self port asString)]. EsLogManager info: 'WASStServerAdaptor started on port: ' , port asString] on: Error do: [:ex | EsLogManager error: 'Seaside Adaptor couldn''t start due to: ' , ex description. ex pass]
This is not a victory to be proud of, but one that will help me waste far less time lookinkg for phantom bugs. Here’s what happens in the Server ssh session when I start the image a second time:
UIProcess reportError: Adaptor is not Running - maybe port xxxxx is already in use? Dumping walkback to file: walkback.log
And the server process is immediately exiting. When I look into the application log I can see the server didn’t start because the port was in use. Great! Why not be happy about a small victory that may lay the ground for faster progress. It’s not much ado about nothing, and II’m pretty sure it pays back one day.
The European Smalltalk User Group just issued its call for presentations for its 21st annual conference that will be held in Annecy, France from 9 to 13 September 2013:
For the past 21 years, the European Smalltalk User Group (ESUG) has organised the International Smalltalk Conference, a lively forum on cutting edge software technologies that attract people from both academia and industry for a whole week. The attendees are both engineers using Smalltalk in business and students and teachers using Smalltalk both for research and didactic purposes.
As every year, this year’s edition of the largest European Smalltalk event will include the regular Smalltalk developers conference with renowned invited speakers, a Smalltalk camp that proves fruitful for interactions and discussions. This year will be held the 9th edition of the Innovation Technology Awards where prizes will be awarded to authors of best pieces of Smalltalk-related projects and an international workshop on Smalltalk and dynamic languages
You can read the full call for presentations and learn about deadlines and such here.
You may have realized already that I misuse my blog and therewith you, my valued reader, as a swap space for small and maybe not so small little tricks I find in my day job from time to time.
And here is one little thing I just learned about how to invoke commands in DB2 that are not SQL statements form VA Smalltalk, in this specific case I mean REORG TABLE, but there are many other commands for which this may be useful.
Let me give you a little bit of context on why on earth I’d need that. Real men and even more so real DBA’s would simply fire up their DB2 command prompt and solve the problem at hand like a man. A simple table reorg would never stop a real man from saving the world in a day…
But here’s my problem. I am working on a Seaside Application in VA Smalltalk that is to be deployed to a Linux server. This application uses GLORP for persistence and I added some code that changes the database tables on server startup whenever I deploy a new image version. So if I add a new attribute to some persistent class, I have to add a new column to the underlying table(s). Some changes to the object model or some optimizations also need changes to foreign keys, indexes or even primary keys. These changes often cause DB2 to stop doing anything before I Reorg the modified table (Hint to IBM: Maybe that could be automated. The error message already tells me I need to Reorg, so why doesn’t it just do it for me???). The Error after such a change looks like this:
[SQLSTATE=57016 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0668N Operation not allowed for reason code "7" on table "MYTABLE". SQLSTATE=57016 [Native Error=-668]]
Yes, you’re guessing right: this error has made my life harder than I wanted it to be more than once in the past.
So one step in these schema migrations often is to change tables and then move data around, add foreign keys and stuff. In theory, that’s not too hard (I’ve learned a whole lot about this stuff from one of my friends and customers, hi Peter!). Unless DB2 gets into my way and tells me now that I’ve changed the primary key, I need to reorg the table first before I can change data.
Unfortunately, REORG TABLE is not a normal SQL statement. It is not intended to be used by normal SQL users and therefor cannot be issued just like a normal SQL statement. Here’s what you get from DB2 if you inspect myGlorpSession accessor executeSQLString: 'REORG TABLE schema.tablename':
AbtError: rc=-1 for '42601' in an AbtIbmCliCSDatabaseConnection at (24.04.2013 15:50:41) '[SQLSTATE=42601 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0104N An unexpected token "TABLE" was found following "REORG ". Expected tokens may include: "JOIN <joined_table>". SQLSTATE=42601 [Native Error=-104]]
So this meant whenever I wanted to change a table’s primary key or add an index, I had to start the server once, stop it, use db2 to REORG tables by hand and start the server again. The steps of modifying the indexes and the following steps had to be separate migrations, each of them in their own transaction. Quite annoying.
But I found a solution to this problem. Real men, of course, knew it already. You can use a normal CLI client and issue admin commands by wrapping them into this:
myGlorpSession accessor executeSQLString: 'CALL ADMIN_CMD (''REORG TABLE schema.tablename'')' .
With this I can reorg a table first and then modify data in one single step during my server startup.
Doesn’t sound like a big deal? You’re wrong. This was a real hurdle to simple deployment. Just imagine you have to redo the same stuff on a development machine, a test server and a production server over and over again. And you must remember when to do what in the right order. Especially on a production server which shouldn’t be offline for too long, this is very important.
Before you ask: I am aware of the AUTO_REORG parameter, but as far as I know, Table reorganization can only run in an offline window. That’s not exactly what I am looking for. I need to do it in the middle of a migration script, not somewhen later tonight …
There are probably a few questions that you may ask about this problem I just posted about. And I try to answer them here so you may understand better why I ask for an exception (or at least a Warning):
Why don’t you simply make sure the image is only running once on the server?
Because that will hopefully not be the case for long. I hope for hundreds, better thousands of customers for my service, and scaling requires running multiple images behind a load balancer. Restricting myself to just one esnx process or only one image of a certain name seems like a bad idea here. Maybe the same server will once have to run another Seaside Server Application in addition to this one, and then the whole solution falls like a house of cards. BTW: You’ve heard about the possible ill effects of the Singleton pattern, haven’t you? This is one example that supports the theorem.
Why don’t you simply see if the port is taken before you start the Server Adaptor?
I don’t want to. It’s not smalltalkish to check before hand. We have great exception handling capabilities in Smalltalk and this is a great use case for them. Look at the code before and after the change in my last post. The first version is clean and easy to understand and completely sufficient to handle the situation the Smalltalk way. The final version is littered with stupid error condition checks and even polling. The only thing worse I can think would be some is...Error checks. Ah, no there is an even worse one: an integer return value that needs to be interpreted. That was old school back in 1983. C’mon!
You don’t like this answer? I’ll give you another one. I don’t know how to ask the OS whether a port is in use or not. Especially not if I have to keep my code portable between Windows and Linux. I am using a high-level language that aims to keep me abstracted from this OS Level stuff whenever it can, but still allows me to take the stairs down the basement and dig in the dirt there.
Why don’t you just check for instances of the Adaptor and stop the running one?
At least you’re still reading. Thanks for that, you’re a true friend.
As I’ve mentioned before, this already running instance is in another image, meaning in another OS Process and another address space. I cannot access it without using some sophisticated Inter Process Communication feature. Just for checking if I can use a port, that is a bit much. At least in my opinion.
There’s so much you can do with PID files and such on Linux. Why don’t you automate your deployment and add some fancy PID / Port lock files and stuff so that step 4 is never forgotten any more?
First, this is a great idea. And I am working on getting better in that area. Automation is a hobby of mine and I’ve helped quite a few customers automate their Packaging and Deployment in VAST projects.
Still, this would still be an incomplete solution. What if I decide to use another copy of my image (e.g. when I need to scale) and to use a port that is used by one of the daemons on the machine (I know, you can always check open ports, there are tons of lists on the web where you can find out what ports are usually used by the most-popular 1000000 applications)? What if that other application simply doesn’t use my mechanism, even though I’ve spent years making it perfect?
[Update: Please be sure tor read this post as well if you're interested in more details on this subject and this one if you want to see what my solution finally looks like]
This is a little lesson I learned with my VAST-based Seaside Application on a headless Linux Server that is probably not really a gem in every Seasider’s toolbox, but the initial problem cost me quite a few nerves.
Let me start by explaining my workflow for a new Image Version:
Continue reading
