Marten shares a source snippet on how to create a zip archive with VA Smalltalk:
Today I was in the need to create a zip archive in VASmalltalk and zip-code had been introduced with the new Monticello Importer in version 8.5.2 (application: MZZipUnzipApp)
(Please note: the problems described here are not limited to Glorp. This is an issue to keep in mind in all O/R mapping technologies, not only in Smalltalk. What I’m talking about here is also relevant in other languages and frameworks. It just fits nicely into the Glorp Wisdom series)
Some days are even worse than you might think (see the last installment in this series), because they make you shake your head and pray that at least this last stupid error was not yours.
In this case, it was. As I’ve mentioned before, marking a OneToManyRelationship in Glorp to #beExclusive can cause problems if you want to move the object from one such collection to any other collection or even a 1:1 relation.
Looking a bit deeper, this error is quite pardonable compared to an even dumber one:
Never mark two (or more) OneToManyRelationships as exclusive if they could ever share objects.
Why, you ask? Glad you do, because it makes me feel a bit better. Because as I said in pt. 312, teh object will be relentlessly deleted if it is removed from one of the two collections. No matter if you intended it to remain present in the other one, Glorp will be ruthless (because you told it to be, not because it is a bad guy) and kill it. Forever.
Okay, you think that’s obvious and clear and not worth mentioning, and this would never happen to you. I bet you are wrong!
Because object models evolve. What once was clearly a case of “if it’s not referenced by this object, there’s no use in its existence, ever” may one day look completely different. You may add some 1:m relationship for performance reasons, or you may simply look at another aspect of your business model for a new feature and have the same idea: “well, if it’s removed from here, than it cannot be anywhere else”. Unless it can for reasons that are long forgotten or just not in your focus today.
After the experiences I’ve had today, I’d tend to say #beExclusive should be avoided as long as possible. It does make sense in some cases, but only if an object in an exclusive relationship really is onle related to its containing object and cannot be referenced from anywhere else. And even if you think that is clearly the case in your project, be careful!
Why do I say such stupid things? Because if you have a few thousand unreferenced objects in your database after a while of operation, you can always delete them or simply let them linger a little longer. Storage is cheap. Maybe you’ll finde a bug one day and be glad there are ways to recover their references and build some highly complex SQL code to get them back. An unreferenced object in the DB will not disturb your application’s operations and not get in your way.
If Glorp, however, deleted too many objects because of your mapping error, you may have trouble recovering any data. And it can take months and years to find this problem.
I’m in the flow today, so I’ll give you another word of wisdom for free:
Always act as if you had no relationship mappings, foreign keys or contsraints like ON DELETE CASCADE and clean up your backpointers and stuff in the application instead of relying on some “obscure” database functionality. Clean up on the object side and make sure objects that shouldn’t reference an object, really don’t, rather than assuming the database will do it anyways.
If you follow this rule, there are multiple benefits:
There are days when you hunt for a bug and almost are close to giving up. On some of these, it’s best to go home, play with your kids or watch a good movie and start again the next day.
On some, however, you finally find out teh whole problem was just a case of programmer’s stupidity at work.
I won’t tell you what kind of day today is for me, but I’ll share some words of wisdom with you:
Don’t use objects in an exclusive OneToManyRelationship and move them into another collection. You won’t like the results.
Want more details? Okay, my friend, let’s sit down and I’ll tell you all about it.
In Glorp, you can define a 1:m relationship to #beExclusive. This makes a lot of sense (that’s why many other ORM Frameworks have the same ability, maybe named a bit differently). What it means is that if an object that holds on to an exclusive relationship (or better: a collection of related objects) and gets deleted, all these objects will be deleted as well. Exclusive in that context means: the objects in this collection cannot exist withou the objext that holds them.
There is, however another way to look at it: it also means that if the programmer removes an object from this collection, it is not going to exist on its own any more. At least not after a commitTransaction.
So far, all of this makes sense.
Enter Joachim (you may guess right now which kind of day I had today).
Because all of this also means that there is aboslutely no sense in moving this object into another exclusive collection. Because you said it’s doomed. It’s gonna be doomed, no matter what. The object may be reachable from other collections or stuff, but an exclusive collection says: if you leave me, you die. It really is that simple. Glorp is gonna delete it, and never going to update it to reflect its membership of another collection. That would make sense if the object wasn’t coming from an exclusive collection, but, hey, it does!
So if you use exclusive collections (and I do because it makes sense in my business case), always take care what you do with your business objects. Moving it to another object’s collections won’t be reflected by updating it’s foreign keys, even if you do all backpointering and stuff correctly. Glorp’s answer will ALWAYS be a DELETE statement.
So moving an object from one exclusive collection to another always has to be done by removing the original object from the first collection and creating a new one for the other. You may copy references and attributes, even non-exclusive OneToMany collections to this new instance, but the object itself has to be a new one.
Well, a developer’s day is full of victories and waterloos. So I’ve had my developer’s day today.
After I had discussed my Server startup problems related to ports already in use, Marten came up with an explanation for VAST’s behavior on the VAST Support Group:
The reason for this behaviour is located in the method: SstTcpTransport>>#basicOpenListeningSocket and it seems to be correctly handled by VASmalltalk.
In this method a socket option (“SOREUSEADDR”) is set to true (if the configuration of the tcp network allows it, which is normally the case of VASmalltalk).
Hmm. Even if it’s not incorrect, I still don’t like it.
So my first waterloo was that Marten proved me wrong: this is not a bug or wrong behavior – from a very technical standpoint. I still thinnk the server pretends to be ready when it is not, and I don’t like it if my software tells me lies right into my face, especially if that even costs me lots of time…
The thread at the discussion group then continues with suggestions like changing the setting Marten mentioned in some very low level method for binding to a Socket. That is of course very deep down and you never know what you break if you open ALL TCP/IP Sockets as exclusive listeners.
So I decided to stick with my workaround that I described in my initial post on the subject (the reworked one, of course). It ran great – on Windows.
So here was my second waterloo. When I had XD packaged this and tried to start the same image twice on the Linux test machine, the Adaptor never left the #isStarting phase. The log contained lots of entries telling me we’re waiting for the adaptor to finish starting:
'2013-04-24 17:32:48,917: [INFO] Waiting for the Adapter to finish starting' '2013-04-24 17:32:49,519: [INFO] Waiting for the Adapter to finish starting' '2013-04-24 17:32:50,121: [INFO] Waiting for the Adapter to finish starting' '2013-04-24 17:32:50,723: [INFO] Waiting for the Adapter to finish starting' '2013-04-24 17:32:51,325: [INFO] Waiting for the Adapter to finish starting'
.. and so on. So the Socket behavior differs between Windows and Linux. On Windows, the retry loop was never run, the Adapter was not running immediately.
So I decided to extend my ugly workaround a little more with a retry count. I hate the code as it looks now, because it reminds me of really dark days in my programmer career. I’ll show it to you nevertheless, because it may be useful for others as well (I know, it’s not rocket science, but copying it from here is faster than thinking about it – you’re welcome!):
startServerAdaptor "self startServerAdaptor" |adaptor maxRetries retries| maxRetries := 5. retries := 1. [ (adaptor := WASstServerAdaptor port: self port) start. [adaptor isStarting and: [retries <= maxRetries]] whileTrue: [ EsLogManager info: ('Waiting for the Adapter to finish starting, retry count = %1' bindWith: retries asString). (Delay forSeconds: 0.5) wait. retries := retries +1. ]. adaptor isRunning ifFalse: [Error signal: ('Adaptor is not Running - maybe port %1 is already in use?' bindWith: self port asString)]. EsLogManager info: 'WASStServerAdaptor started on port: ' , port asString] on: Error do: [:ex | EsLogManager error: 'Seaside Adaptor couldn''t start due to: ' , ex description. ex pass]
This is not a victory to be proud of, but one that will help me waste far less time lookinkg for phantom bugs. Here’s what happens in the Server ssh session when I start the image a second time:
UIProcess reportError: Adaptor is not Running - maybe port xxxxx is already in use? Dumping walkback to file: walkback.log
And the server process is immediately exiting. When I look into the application log I can see the server didn’t start because the port was in use. Great! Why not be happy about a small victory that may lay the ground for faster progress. It’s not much ado about nothing, and II’m pretty sure it pays back one day.
You may have realized already that I misuse my blog and therewith you, my valued reader, as a swap space for small and maybe not so small little tricks I find in my day job from time to time.
And here is one little thing I just learned about how to invoke commands in DB2 that are not SQL statements form VA Smalltalk, in this specific case I mean REORG TABLE, but there are many other commands for which this may be useful.
Let me give you a little bit of context on why on earth I’d need that. Real men and even more so real DBA’s would simply fire up their DB2 command prompt and solve the problem at hand like a man. A simple table reorg would never stop a real man from saving the world in a day…
But here’s my problem. I am working on a Seaside Application in VA Smalltalk that is to be deployed to a Linux server. This application uses GLORP for persistence and I added some code that changes the database tables on server startup whenever I deploy a new image version. So if I add a new attribute to some persistent class, I have to add a new column to the underlying table(s). Some changes to the object model or some optimizations also need changes to foreign keys, indexes or even primary keys. These changes often cause DB2 to stop doing anything before I Reorg the modified table (Hint to IBM: Maybe that could be automated. The error message already tells me I need to Reorg, so why doesn’t it just do it for me???). The Error after such a change looks like this:
[SQLSTATE=57016 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0668N Operation not allowed for reason code "7" on table "MYTABLE". SQLSTATE=57016 [Native Error=-668]]
Yes, you’re guessing right: this error has made my life harder than I wanted it to be more than once in the past.
So one step in these schema migrations often is to change tables and then move data around, add foreign keys and stuff. In theory, that’s not too hard (I’ve learned a whole lot about this stuff from one of my friends and customers, hi Peter!). Unless DB2 gets into my way and tells me now that I’ve changed the primary key, I need to reorg the table first before I can change data.
Unfortunately, REORG TABLE is not a normal SQL statement. It is not intended to be used by normal SQL users and therefor cannot be issued just like a normal SQL statement. Here’s what you get from DB2 if you inspect myGlorpSession accessor executeSQLString: 'REORG TABLE schema.tablename':
AbtError: rc=-1 for '42601' in an AbtIbmCliCSDatabaseConnection at (24.04.2013 15:50:41) '[SQLSTATE=42601 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0104N An unexpected token "TABLE" was found following "REORG ". Expected tokens may include: "JOIN <joined_table>". SQLSTATE=42601 [Native Error=-104]]
So this meant whenever I wanted to change a table’s primary key or add an index, I had to start the server once, stop it, use db2 to REORG tables by hand and start the server again. The steps of modifying the indexes and the following steps had to be separate migrations, each of them in their own transaction. Quite annoying.
But I found a solution to this problem. Real men, of course, knew it already. You can use a normal CLI client and issue admin commands by wrapping them into this:
myGlorpSession accessor executeSQLString: 'CALL ADMIN_CMD (''REORG TABLE schema.tablename'')' .
With this I can reorg a table first and then modify data in one single step during my server startup.
Doesn’t sound like a big deal? You’re wrong. This was a real hurdle to simple deployment. Just imagine you have to redo the same stuff on a development machine, a test server and a production server over and over again. And you must remember when to do what in the right order. Especially on a production server which shouldn’t be offline for too long, this is very important.
Before you ask: I am aware of the AUTO_REORG parameter, but as far as I know, Table reorganization can only run in an offline window. That’s not exactly what I am looking for. I need to do it in the middle of a migration script, not somewhen later tonight …
There are probably a few questions that you may ask about this problem I just posted about. And I try to answer them here so you may understand better why I ask for an exception (or at least a Warning):
Why don’t you simply make sure the image is only running once on the server?
Because that will hopefully not be the case for long. I hope for hundreds, better thousands of customers for my service, and scaling requires running multiple images behind a load balancer. Restricting myself to just one esnx process or only one image of a certain name seems like a bad idea here. Maybe the same server will once have to run another Seaside Server Application in addition to this one, and then the whole solution falls like a house of cards. BTW: You’ve heard about the possible ill effects of the Singleton pattern, haven’t you? This is one example that supports the theorem.
Why don’t you simply see if the port is taken before you start the Server Adaptor?
I don’t want to. It’s not smalltalkish to check before hand. We have great exception handling capabilities in Smalltalk and this is a great use case for them. Look at the code before and after the change in my last post. The first version is clean and easy to understand and completely sufficient to handle the situation the Smalltalk way. The final version is littered with stupid error condition checks and even polling. The only thing worse I can think would be some is...Error checks. Ah, no there is an even worse one: an integer return value that needs to be interpreted. That was old school back in 1983. C’mon!
You don’t like this answer? I’ll give you another one. I don’t know how to ask the OS whether a port is in use or not. Especially not if I have to keep my code portable between Windows and Linux. I am using a high-level language that aims to keep me abstracted from this OS Level stuff whenever it can, but still allows me to take the stairs down the basement and dig in the dirt there.
Why don’t you just check for instances of the Adaptor and stop the running one?
At least you’re still reading. Thanks for that, you’re a true friend.
As I’ve mentioned before, this already running instance is in another image, meaning in another OS Process and another address space. I cannot access it without using some sophisticated Inter Process Communication feature. Just for checking if I can use a port, that is a bit much. At least in my opinion.
There’s so much you can do with PID files and such on Linux. Why don’t you automate your deployment and add some fancy PID / Port lock files and stuff so that step 4 is never forgotten any more?
First, this is a great idea. And I am working on getting better in that area. Automation is a hobby of mine and I’ve helped quite a few customers automate their Packaging and Deployment in VAST projects.
Still, this would still be an incomplete solution. What if I decide to use another copy of my image (e.g. when I need to scale) and to use a port that is used by one of the daemons on the machine (I know, you can always check open ports, there are tons of lists on the web where you can find out what ports are usually used by the most-popular 1000000 applications)? What if that other application simply doesn’t use my mechanism, even though I’ve spent years making it perfect?
[Update: Please be sure tor read this post as well if you're interested in more details on this subject and this one if you want to see what my solution finally looks like]
This is a little lesson I learned with my VAST-based Seaside Application on a headless Linux Server that is probably not really a gem in every Seasider’s toolbox, but the initial problem cost me quite a few nerves.
Let me start by explaining my workflow for a new Image Version:
Continue reading
Erst gestern schrieb ich im Zusammenhang mit dem Google Summer Of Code und der Programmiersprache Smalltalk:
Smalltalk macht enormen Spass. Die sehr dynamische Umgebung ist sehr motivierend, weil man nicht toten Quelltext pflegt, sondern sich stets direkt im lauffähigen Objektsystem aufhält. Man muss es ausprobiert haben, um das zu verstehen…
und just einen Tag später lese ich auf Torsten’s blog über ein exzellentes Beweisvideo von Esteban Lorenzano, das dies sehr schön belegt:
Interactive Smalltalk
A really nice video from Esteban that demonstrates the interactive nature of Smalltalk development and scripting facilities - here with Pharo. The demo is using Storm (a small game engine).
I guess having learned something new is what makes a day a good one. So I’ve had a good day today.
In my Seaside Application, I try to add a few fancy ajax gadgets that use Ajax and do something before and after the actual Ajax call is performed. Especially in the case where you need to do something only after the ajax call is finished, you need to always keep in mind that the Ajax call is asynchronous, that means if you do an $.ajax, the following javascript statements will be run after the ajax call is issued (in other words: immediately), not after the call is finished.
I thought I knew all about this.
Until today.
Because I had something that looked like this:
$.ajax({
"url": options.url,
"data": [
options.queryFields,
options.serializeCallback+"="+self.val(),
].join("&"),
"complete": drawTarget(),
});
Don’t worry about the data parameter, that’s not important for the moment. Just imagine I send out a Seaside Callback’s identification as part of the options to my plugin and that is what gets handed back to the server in this call.
The point of my post is about the complete callback.
It turns out that I had some strange effect in that the callbacks (the one from this ajax and one in the drawTarget() function) came into my Seaside image in the wrong order, but only after the ajax was done. So the complete callback worked to some extent, but not quite right.
It turns out the use of complete, success and so one as parameters to $.ajax has long been deprecated:
So after I changed my ajax code to
$.ajax({
"url": options.url,
"data": [
options.queryFields,
options.serializeCallback+"="+self.val(),
].join("&"),
}).always(drawTarget());
The callbacks come in in the correct order and my application works as it should. This is just another proof of the “once you do it right, it simply works ” theorem, I guess
And, it also is another hint that reading documentation can sometimes be a good idea.
Nun also ist mein MB Pro 4,1 (early 2008) seit ein paar Tagen mit einer Samsung SSD ausgerüstet (Dieses Blog berichtete) und die Re-Installation sowie Einrichtung meiner kuscheligen Arbeitsumgebung schreitet voran. Man sieht ab und an mal, dass es mit der Intuitivität (heisst das so???) beim Mac OS nicht immer allzu weit her ist.
War es ein Erfolg, das Aufrüsten? Das ist sicher die Frage, die den geneigten Leser so weit getrieben hat, hier noch immer zu lesen. Die kurze Antwort: Aber Hallo!
Die Startzeiten werden gerne als Referenzwert hergenommen, um zu beweisen, wie toll schnell der Rechner nun ist. Nun hat dessen Praxisrelevanz einen überschaubaren Umfang. Nur, weil meine SSD darin nun schneller ist, boote ich den Rechner freiwillig nicht ein einiziges mal mehr am Tage… Genau hier übrigens fand ich den Effekt nicht besonders aufregend. Bis der Kaffee geholt war, war auch früher schon alles so weit
Das wirkliche Boah! war eigentlich erst beim Start von VMWare, LibreOffice, Illustrator oder Coda zu spüren. Und das ist schon sehr gut spürbar. Erst mit einer SSD merkt man so richtig, das CPU-Power beileibe nicht alles ist. Mein Laptop kann so sicher noch zwei, drei Jahre seinen Dienst tun, ohne, dass ich ständig neidisch auf schicke Unibody-Macs mit verklebtem Akku, patentierten Schrauben ohne jegliche Unebenheiten, Kohlefaserversenkte Speicherbänke und so weiter schielen muss. Das gute Gefühl, ein solides Stück Hardware noch eine Weile länger zu nutzen, und nur eine Festplatte dem Ökosystem zuzumuten anstatt ein komplettes, ansonsten funktionstüchtigen Laptops, kann ich nun erhobenen Hauptes geniessen und dennoch meine Arbeit verrichten. Das Budget freut sich auch, denn diese Umrüstung hat unter 10% eines neuen Laptops gekostet.
Zum Abschluss dieses ja eigentlich langweiligen Themas noch ein paar Praxistipps:
Eine vielleicht sehr wichtige Info, die ich eher durch Zufall gefunden habe: Samsung bietet ein Tool zum Update der Firmware der 840er SSDs für den Mac an. Im Web findet man fast nur (veraltete) Forumsdiskussionen, in denen genau das bemängelt wird: Dass nämlich ein Firmware-Update nur durch Ausbau der Platte und Anschluss an einen PC möglich sei. Dem ist nicht so, es gibt ein Tool namens “Samsung SSD Firmware Updates for Mac Users“. Dieses Tool arbeitet genau so, wie im zugehörigen manual beschrieben, allerdings war meine Platte bereits auf dem neuesten Stand, also kann ich nicht sagen, ob ein FW-Update funktioniert. Da die Platte läuft und gute Dienste zu tun scheint, ist das aber auch in Ordnung, ich werde vermutlich nie ein weiteres Firmware-Update durchführen, wenn nicht irgendwas schlimmes passiert.
Der Trim Enabler ist ein Tool, das eigentlich eine Frechheit ist. Halt, nicht falsch verstehen: das Tool ist genial. Genial einfach und genial wichtig. Aber es ist eine Frechheit von Apple, das Anschalten von TRIM für fremd-Platten einfach nicht selbst anzubieten. Umso mehr ist es angebracht, dem Entwickler des Trim Enablers eine Freude zu machen, und den Donate-Button auf seiner Seite zu klicken. Das Ding ist super simpel zu bedienen (Schieber von Off auf On und den Rechner neu starten) und tut seinen Dienst absolut problemlos (Über diesen Mac berichtet, TRIM sei aktiviert). Das mit der Spende hat allerdings einen Haken: die verlinkte Paypal-Seite zum Spenden ist auf finnisch, und ich hatte bisher keine Ahnung, dass “Deutschland” auf finnisch “Tyskland” heisst. Ich hätte aber eine Nachbarin fragen können, wenn ich es nicht erraten hätte…
