The WASstServerAdaptor story and its (preliminary) end

Well, a developer’s day is full of victories and waterloos. So I’ve had my developer’s day today.

After I had discussed my Server startup problems related to ports already in use, Marten came up with an explanation for VAST’s behavior on the VAST Support Group:

The reason for this behaviour is located in the method: SstTcpTransport>>#basicOpenListeningSocket and it seems to be correctly handled by VASmalltalk.

In this method a socket option (“SOREUSEADDR”) is set to true (if the configuration of the tcp network allows it, which is normally the case of VASmalltalk).

Hmm. Even if it’s not incorrect, I still don’t like it.
So my first waterloo was that Marten proved me wrong: this is not a bug or wrong behavior – from a very technical standpoint. I still thinnk the server pretends to be ready when it is not, and I don’t like it if my software tells me lies right into my face, especially if that even costs me lots of time…

The thread at the discussion group then continues with suggestions like changing the setting Marten mentioned in some very low level method for binding to a Socket. That is of course very deep down and you never know what you break if you open ALL TCP/IP Sockets as exclusive listeners.

So I decided to stick with my workaround that I described in my initial post on the subject (the reworked one, of course). It ran great – on Windows.

So here was my second waterloo. When I had XD packaged this and tried to start the same image twice on the Linux test machine, the Adaptor never left the #isStarting phase. The log contained lots of entries telling me we’re waiting for the adaptor to finish starting:

'2013-04-24 17:32:48,917: [INFO] Waiting for the Adapter to finish starting'
'2013-04-24 17:32:49,519: [INFO] Waiting for the Adapter to finish starting'
'2013-04-24 17:32:50,121: [INFO] Waiting for the Adapter to finish starting'
'2013-04-24 17:32:50,723: [INFO] Waiting for the Adapter to finish starting'
'2013-04-24 17:32:51,325: [INFO] Waiting for the Adapter to finish starting'

.. and so on. So the Socket behavior differs between Windows and Linux. On Windows, the retry loop was never run, the Adapter was not running immediately.

So I decided to extend my ugly workaround a little more with a retry count. I hate the code as it looks now, because it reminds me of really dark days in my programmer career. I’ll show it to you nevertheless, because it may be useful for others as well (I know, it’s not rocket science, but copying it from here is faster than thinking about it – you’re welcome!):

startServerAdaptor

	"self startServerAdaptor"

	|adaptor maxRetries retries|

	maxRetries := 5.
	retries := 1.

	[
		(adaptor := WASstServerAdaptor port: self port) start.
		[adaptor isStarting and: [retries <= maxRetries]] whileTrue: [
			EsLogManager info: ('Waiting for the Adapter to finish starting, retry count = %1' bindWith: retries asString).
			(Delay forSeconds: 0.5) wait.
			retries := retries +1.
			].
		adaptor isRunning ifFalse: [Error signal: ('Adaptor is not Running - maybe port %1 is already in use?' bindWith: self port asString)].
		EsLogManager info: 'WASStServerAdaptor started on port: ' , port asString]
			on: Error
			do: [:ex |
				EsLogManager
					error: 'Seaside Adaptor couldn''t start due to: ' , ex description.
				ex pass]

This is not a victory to be proud of, but one that will help me waste far less time lookinkg for phantom bugs. Here’s what happens in the Server ssh session when I start the image a second time:

UIProcess reportError: Adaptor is not Running - maybe port xxxxx is already in use?
 Dumping walkback to file: walkback.log

And the server process is immediately exiting. When I look into the application log I can see the server didn’t start because the port was in use. Great! Why not be happy about a small victory that may lay the ground for faster progress. It’s not much ado about nothing, and II’m pretty sure it pays back one day.

The European Smalltalk User Group needs your presentations for ESUG 2013

The European Smalltalk User Group just issued its call for presentations for its 21st annual conference that will be held in Annecy, France  from 9 to 13 September 2013:

For the past 21 years, the European Smalltalk User Group (ESUG) has organised the International Smalltalk Conference, a lively forum on cutting edge software technologies that attract people from both academia and industry for a whole week. The attendees are both engineers using Smalltalk in business and students and teachers using Smalltalk both for research and didactic purposes.

As every year, this year’s edition of the largest European Smalltalk event will include the regular Smalltalk developers conference with renowned invited speakers, a Smalltalk camp that proves fruitful for interactions and discussions. This year will be held the 9th edition of the Innovation Technology Awards where prizes will be awarded to authors of best pieces of Smalltalk-related projects and an international workshop on Smalltalk and dynamic languages

You can read the full call for presentations and learn about deadlines and such here.

Issuing a REORG TABLE command to DB2 from VA Smalltalk (and Glorp)

You may have realized already that I misuse my blog and therewith you, my valued reader, as a swap space for small and maybe not so small little tricks I find in my day job from time to time.

And here is one little thing I just learned about how to invoke commands in DB2 that are not SQL statements form VA Smalltalk, in this specific case I mean REORG TABLE, but there are many other commands for which this may be useful.

Let me give you a little bit of context on why on earth I’d need that. Real men and even more so real DBA’s would simply fire up their DB2 command prompt and solve the problem at hand like a man. A simple table reorg would never stop a real man from saving the world in a day…

But here’s my problem. I am working on a Seaside Application in VA Smalltalk that is to be deployed to a Linux server. This application uses GLORP for persistence and I added some code that changes the database tables on server startup whenever I deploy a new image version. So if I add a new attribute to some persistent class, I have to add a new column to the underlying table(s). Some changes to the object model or some optimizations also need changes to foreign keys, indexes or even primary keys. These changes often cause DB2 to stop doing anything before I Reorg the modified table (Hint to IBM: Maybe that could be automated. The error message already tells me I need to Reorg, so why doesn’t it just do it for me???). The Error after such a change looks like this:

[SQLSTATE=57016 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0668N  Operation not allowed for reason code "7" on 
table "MYTABLE".  SQLSTATE=57016
 [Native Error=-668]]

Yes, you’re guessing right: this error has made my life harder than I wanted it to be more than once in the past.

So one step in these schema migrations often is to change tables and then move data around, add foreign keys and stuff. In theory, that’s not too hard (I’ve learned a whole lot about this stuff from one of my friends and customers, hi Peter!). Unless DB2 gets into my way and tells me now that I’ve changed the primary key, I need to reorg the table first before I can change data.

Unfortunately, REORG TABLE is not a normal SQL statement. It is not intended to be used by normal SQL users and therefor cannot be issued just like a  normal SQL statement. Here’s what you get from DB2 if you inspect myGlorpSession accessor executeSQLString: 'REORG TABLE schema.tablename':

AbtError:  rc=-1 for '42601' in an AbtIbmCliCSDatabaseConnection at (24.04.2013 15:50:41)  '[SQLSTATE=42601 - [IBM][CLI Driver][DB2/LINUXX8664] SQL0104N  An unexpected token "TABLE" was found following "REORG ".  Expected tokens may include:  "JOIN <joined_table>".  SQLSTATE=42601
 [Native Error=-104]]

So this meant whenever I wanted to change a table’s primary key or add an index, I had to start the server once, stop it, use db2 to REORG tables by hand and start the server again. The steps of modifying the indexes and the following steps had to be separate migrations, each of them in their own transaction. Quite annoying.

But I found a solution to this problem. Real men, of course, knew it already. You can use a normal CLI client and issue admin commands by wrapping them into this:

myGlorpSession accessor executeSQLString:  'CALL ADMIN_CMD (''REORG TABLE schema.tablename'')' .

With this I can reorg a table first and then modify data in one single step during my server startup.

Doesn’t sound like a big deal? You’re wrong. This was a real hurdle to simple deployment. Just imagine you have to redo the same stuff on a development machine, a test server and a production server over and over again. And you must remember when to do what in the right order. Especially on a production server which shouldn’t be offline for too long, this is very important.

Before you ask: I am aware of the AUTO_REORG parameter, but as far as I know, Table reorganization can only run in an offline window. That’s not exactly what I am looking for. I need to do it in the middle of a migration script, not somewhen later tonight …

More on WASstServerAdaptor and used ports

There are probably a few  questions that you may ask about this problem I just posted about. And I try to answer them here so you may understand better why I ask for an exception (or at least a Warning):

Why don’t you simply make sure the image is only running once on the server?

Because that will hopefully not be the case for long. I hope for hundreds, better thousands of customers for my service, and scaling requires running multiple images behind a load balancer. Restricting myself to just one esnx process or only one image of a certain name seems like a bad idea here. Maybe the same server will once have to run another Seaside Server Application in addition to this one, and then the whole solution falls like a house of cards. BTW: You’ve heard about the possible ill effects of the Singleton pattern, haven’t you? This is one example that supports the theorem.

Why don’t you simply see if the port is taken before you start the Server Adaptor?

I don’t want to. It’s not smalltalkish to check before hand. We have great exception handling capabilities in Smalltalk and this is a great use case for them. Look at the code before and after the change in my last post. The first version is clean and easy to understand and completely sufficient to handle the situation the Smalltalk way. The final version is littered with stupid error condition checks and even polling. The only thing worse I can think would be some is...Error checks. Ah, no there is an even worse one: an integer return value that needs to be interpreted. That was old school back in 1983. C’mon!

You don’t like this answer? I’ll give you another one. I don’t know how to ask the OS whether a port is in use or not. Especially not if I have to keep my code portable between Windows and Linux. I am using a high-level language that aims to keep me abstracted from this OS Level stuff whenever it can, but still allows me to take the stairs down the basement and dig in the dirt there.

Why don’t you just check for instances of the Adaptor and stop the running one?

At least you’re still reading. Thanks for that, you’re a true friend.
As I’ve mentioned before, this already running instance is in another image, meaning in another OS Process and another address space. I cannot access it without using some sophisticated Inter Process Communication feature. Just for checking if I can use a port, that is a bit much. At least in my opinion.

There’s so much you can do with PID files and such on Linux. Why don’t you automate your deployment and add some fancy PID / Port lock files and stuff so that step 4 is never forgotten any more?

First, this is a great idea. And I am working on getting better in that area. Automation is a hobby of mine and I’ve helped quite a few customers automate their Packaging and Deployment in VAST projects.
Still, this would still be an incomplete solution. What if I decide to use another copy of my image (e.g. when I need to scale) and to use a port that is used by one of the daemons on the machine (I know, you can always check open ports, there are tons of lists on the web where you can find out what ports are usually used by the most-popular 1000000 applications)? What if that other application simply doesn’t use my mechanism, even though I’ve spent years making it perfect?

Starting WASstServerAdaptor and handling used ports

[Update: Please be sure tor read this post as well if you’re interested in more details on this subject and this one if you want to see what my solution finally looks like]

This is a little lesson I learned with my VAST-based Seaside Application on a headless Linux Server that is probably not really a gem in every Seasider’s toolbox, but the initial problem cost me quite a few nerves.

Let me start by explaining my workflow for a new Image Version:

More

Die Interaktivität der Smalltalk-Entwicklung im Video

Erst gestern schrieb ich im Zusammenhang mit dem Google Summer Of Code und der Programmiersprache Smalltalk:

Smalltalk macht enormen Spass. Die sehr dynamische Umgebung ist sehr motivierend, weil man nicht toten Quelltext pflegt, sondern sich stets direkt im lauffähigen Objektsystem aufhält. Man muss es ausprobiert haben, um das zu verstehen…

und just einen Tag später lese ich auf Torsten’s blog über ein exzellentes Beweisvideo von Esteban Lorenzano, das dies sehr schön belegt:

Interactive Smalltalk

Hier kann man live zusehen, wie man sich als Smalltalk-Entwickler ein Objekt in einem Inspector greift, und direkt während des Programmablaufs Änderungen daran durchführt. Im verlinkten Video werden nur Variablen von Objekten zur Laufzeit geändert und damit der Verlauf des Programms beeinflusst.
Aber das ist erst der Anfang: Nutzt man den Smalltalk-Debugger, kann man direkt während des Programmlaufs den Code verändern und das Programm direkt weiter laufen lassen. Da es in Smalltalk keinen Compiler/Link-Zyklus gibt, ist das Ergebnis sofort verfügbar, man sieht die Auswirkungen seiner Änderungen sofort – und kann sie eventuell auch sofort noch feintunen. Leider ist der Debugger nicht im Video zu sehen.
Und das gezeigte funktioniert nicht nur mit virtuellen Bällen und Körben, sondern auch mit Flugbuchungen, SEPA-Lastschriftaufträgen, Wertpapier-Orders oder Buchungssätzen. Oder auch mit komplexen Berechnungen und so weiter. Ein Programm ist so seine eigene Simulation. Das meine ich mit motivierend und Dynamik.
Danke Torsten, für die Argumentationshilfe zum perfekten Zeitpunkt 😉
Und natürlich Danke an Esteban für das tolle Video.

Die European Smalltalk User’s Group ist Mentoring Organization für Googles Summer Of Code 2013

Auch dieses Jahr hat Google die ESUG (European Smalltalk User’s Group) als Mentoring Organization für die Teilnahme am Summer Of Code ausgewählt. Das sind für die Smalltalk-Welt sehr gute Nachrichten.

Was ist der Google Summer Of Code?

Die Kurzform: Google bezahlt einen oder mehrere Studenten dafür, dass sie diesen Sommer an einem open source – Projekt zusammen mit einer Mentoring Organization arbeitet. Mitmachen können alle Studenten einer Bildungseinrichtung wie z.B. Universitäten, Fachhochschulen, duale Hochschulen etc. Jeder (zugelassene) Student, der zum Abschluss des Summer Of Code seine Ziele erreicht, erhält von Google ein Stipendium in Höhe von 4500 US Dollar. Ja, richtig gelesen: eine schöne Summe fürs Programmieren an einem open source – Projekt.

Die Mentoring Organization, also in diesem Fall die ESUG bekommt von Google für jeden betreuten Studenten eine Aufwandsentschädigung in Höhe von 500 USD. Auch das ein sehr willkommener Beitrag für eine Organisation, die open source – Projekte durchführt.

Die Langform und wirklich alle wichtigen Informationen finden sich auf der Homepage des Google Summer Of Code 2013. Dort findet sich auch eine sehr umfangreiche Sammlung von FAQs.

Wie kann man mitmachen?

Es gibt die Möglichkeit, als Mentor für ein Projekt mitzuhelfen. Für Studenten heisst es nun, sich bis 22. April schlau zu machen, was für Projektideen die ESUG vorgeschlagen hat, und woran man Spass hätte, mit zu arbeiten. Die Projektideen der ESUG finden sich hier. Als Teilnehmer wendet man sich zunächst an die Mentoring Organization, die dann anhand der (noch nicht festgelegten) Anzahl gesponserter Projekte die Studenten auswählt. Die Bewerbungsfrist für Studenten endet am 3. Mai.

Warum sollte man sich ausgerechnet für ein Smalltalk-Projekt im Google Summer Of Code interessieren?

Es gibt einige gute Gründe:

  1. Smalltalk ist die Mutter der OO-Sprachen. So ziemlich alle Konzepte rund um die objektorientierte Programmierung haben ihren Ursprung in Smalltalk. Selbst wenn man nicht vorhat, später in Smalltalk zu arbeiten, lernt man hier eine ganze Menge über gutes Design. Smalltalk ist wie Latein: Spanisch, Französisch und andere Sprachen lassen sich mit fundiertem Latein leichter erlernen.
  2. Im Smalltalk-Umfeld passiert sehr viel mehr, als man vielleicht denkt. Gerade in Europa: Pharo Smalltalk zum Beispiel ist ein Projekt, das hauptsächlich in Frankreich, der Schweiz und Deutschland von zahlreichen Leuten unterstützt wird. Daneben ist Smalltalk auch stark in Südamerika vertreten (wie wäre es mit einem Job in Buenos Aires???).
  3. Smalltalk macht enormen Spass. Die sehr dynamische Umgebung ist sehr motivierend, weil man nicht toten Quelltext pflegt, sondern sich stets direkt im lauffähigen Objektsystem aufhält. Man muss es ausprobiert haben, um das zu verstehen…
  4. Die Smalltalk-Gemeinde, vor allem im open-source-Umfeld, ist seit Jahren am Wachsen. Auch, wenn es auf den ersten Blick so aussieht, als wenn Smalltalk seine beste Zeit gehabt hätte, macht Smalltalk sich seinen Weg langsam aber sicher zurück in den Markt. Ähnlich wie es Ruby, PHP und andere Technologien vorgemacht haben, bewegt sich Smalltalk nicht mehr, wie das in den neunziger Jahren noch üblich war, durch große Strategie-Entscheidungen in Organisationen hinein, sondern durch Projekte einzelner, die einfach eine Lösung für existierende Probleme implementieren. So setzen aktuell Unternehmen Smalltalk ein, in denen “das Management” gar nichts davon weiss, und leistet gute Dienste.
  5. Die ESUG hat einige Erfahrung mit der Betreuung von Studenten im Rahmen des GSoC. 2013 wird das sechste Jahr sein, in dem die ESUG teilnimmt. Einige der Mentoren, die bisher Projekte vorgeschlagen haben, sind schon mehrfach dabei gewesen und haben mit ihren Studenten wertvolle Tools und Frameworks in die Smalltalk-Welt eingebracht. Manche Mentoren für 2013 waren selbst in vorangegangenen GSoC’s als Student dabei. Einen Überblick dazu kann man sich hier verschaffen.
  6. Die Smalltalk-Welt ist überschaubar: Wir sind eine relativ kleine Familie (verglichen mit Communities wie JavaScript oder PHP), in der man mit einem einzelnen Projekt durchaus etwas bewegen kann, und nicht eines von vielen Projekten ist, das vielleicht niemals Beachtung finden wird. Es kann also durchaus sehr befriedigend sein, hier mit zu arbeiten. Und vielleicht gibt es dann auch noch ein kleines Stückchen Ruhm zu erlangen 😉

Also: Gründe gibt es genug, und wir freuen uns über jeden, der Mitmachen möchte. Es gibt viel zu tun und hohe Ziele zu erreichen, aber vor allem viel zu lernen und dabei eine Menge Spass zu haben!

Informier Dich hier! 

 

Getting jQuery ajax and callbacks to Seaside into the right order

I guess having learned something new is what makes a day a good one. So I’ve had a good day today.

In my Seaside Application, I try to add a few fancy ajax gadgets that use Ajax and do something before and after the actual Ajax call is performed. Especially in the case where you need to do something only after the ajax call is finished, you need to always keep in mind that the Ajax call is asynchronous, that means if you do an $.ajax, the following javascript statements will be run after the ajax call is issued (in other words:  immediately), not after the call is finished.

I thought I knew all about this.

Until today.

Because I had something that looked like this:

$.ajax({
  "url": options.url,
  "data": [
    options.queryFields,
    options.serializeCallback+"="+self.val(),
  ].join("&"),
  "complete": drawTarget(),
});

Don’t worry about the data parameter, that’s not important for the moment. Just imagine I send out a Seaside Callback’s identification as part of the options to my plugin and that is what gets handed back to the server in this call.

The point of my post is about the complete callback.

It turns out that I had some strange effect in that the callbacks (the one from this ajax and one in the drawTarget() function) came into my Seaside image in the wrong order, but only after the ajax was done. So the complete callback worked to some extent, but not quite right.
It turns out the use of complete, success and so one as parameters to $.ajax has long been deprecated:

Deprecation Notice: The jqXHR.success(), jqXHR.error(), and jqXHR.complete() callbacks are deprecated as of jQuery 1.8. To prepare your code for their eventual removal, use jqXHR.done(), jqXHR.fail(), and jqXHR.always() instead.

So after I changed my ajax code to

$.ajax({
  "url": options.url,
  "data": [
    options.queryFields,
    options.serializeCallback+"="+self.val(),
  ].join("&"),
}).always(drawTarget());

The callbacks come in in the correct order and my application works as it should. This is just another proof of the “once you do it right, it simply works ” theorem, I guess 😉

And, it also is another hint that reading documentation can sometimes be a good idea.