GSoC Progress: DBusKit

Posted on 19 July 2010 by Niels Grewe

As David pointed out earlier, both Eric and I got the opportunity to participate in Google's Summer of Code programme with projects for GNUstep. Since GSoC "midterms" are just over, I'd like to talk a bit about the progress I've been making.

D-Bus and Distributed Objects

My project for this summer is to bring D-Bus support to GNUstep. D-Bus is, as you might know, an IPC mechanism that has been widely adopted on the *nix desktop: If you are sitting in front of a Gnome or KDE desktop, you most certainly also have D-Bus installed. GNUstep has traditionally had a different, but very powerful means of doing IPC that is called "Distributed Objects" (DO, which in fact dates back to the OpenStep spec). It allows you to refer to a distant object (running in a different process) just like you would to a local object. You can just get a proxy to the remote object by calling the appropriate method on NSConnection and just send messages to it as usual:

id thermometer = [NSConnection rootProxyForConnectionWithRegisteredName: @"TemperatureServer"
                                                                       host: @""];
temp = [thermometer temperature];

That's just it. You can even use every return value from the root proxy just as you would normally. Actually the system is very intelligent about that: If process A accesses an object from process B and that object returns a remote object it got from A, DO will be smart enough not to do a round trip A→B→A, but will return the local object instead. Thus, for native IPC needs, GNUstep already has everything one could wish for.

But D-Bus support is still a useful thing for GNUstep because many services on a modern *nix desktop are exposed via D-Bus: If you re-configure your WiFi card on the fly, it's probably through NetworkManager, and if your media player inhibits the power saving functions of your laptop when watching a movie, this is probably done through upower (or HAL). All theses services use D-Bus to provide their functions to other applications.

Using D-Bus from Objective-C

The goal of my project is to make accessing and providing D-Bus services from within an Objective-C application as easy as using Distributed Objects. So you would just do

id thermometer = [NSConnection rootProxyForDBusConnectionWithRegisteredName: @"org.foo.temperature"
                                                                            bus: DKDBusSystemBus];

And while I'm not quite there yet, the DBusKit framework that provides D-Bus support is starting to get useful. It already manages basic method invocations on D-Bus object, albeit that support is at present limited to the object at the root of an object graph. E.g. you can get a list with all names registered on D-Bus with the following lines of code:

NSConnection *conn = [NSConnection connectionWithReceivePort: [DKPort port]
                                                    sendPort: [[DKPort alloc] initWithRemote: @"org.freedesktop.DBus"]];
id dbus = [conn rootProxy];
NSArray *names = [dbus ListNames];

The present implementation is very flexible in the way it interacts with D-Bus. For example, all the following method declarations are valid ways to invoke the D-Bus method NameHasOwner() on the org.freedesktop.DBus object:

- (NSNumber*)NameHasOwner: (NSString*)name;
- (NSNumber*)NameHasOwner: (char*)name;
- (BOOL)NameHasOwner: (NSString*)name;
- (BOOL)NameHasOwner: (char*)name;

So you are free to choose whether you want plain C types or Objective-C objects (though this isn't possible for collection D-Bus values yet: these will always be returned as objects). Of course you have to provide the method signatures for the D-Bus methods you want to call (e.g. in a protocol declaration), but in the future I will provide a little tool that will do that for you.

Outlook

So what remains to do for the second half of GSoC? Apart from the better integration, I will tackle the task of exposing Objective-C objects on D-Bus. This will involve designing a tool that allows you to generate annotated D-Bus introspection data from Objective-C interface or protocol declarations so that you can easily customise what methods an object will expose via D-Bus. Also D-Bus supports not only method calls, but also signals and properties, both of which can easily be mapped to notifications and properties on the Objective-C side. So stay tuned for updates on D-Bus support in GNUstep.

Further reading

If you want to play around with the code, you can find it in the modules/dev-libs/dbuskit directory of the GNUstep SVN tree this also includes an example that exposes the Apertium machine-translation system (which has a D-Bus interface) as a GNUstep service. This is accessible from every GNUstep application and allows you to translate selected text on the fly. Information about the Distributed Objects system can be found in the GNUstep Base Programming Manual, the Apple documentation or an tutorial by Nicola Pero. If you want to learn more about D-Bus, Dan Williams has an excellent introductory post about it.

More Optimization

Posted on 28 April 2010 by David Chisnall

One of the things that is traditionally very slow in Objective-C is sending messages to classes. When you do something like:

[NSMutableArray new];

The compiler expands it to something roughly like this:

Class receiver = objc_lookup_class("NSMutableArray");
SEL new = @selector(new);
IMP method = objc_msg_lookup(receiver, new);
method(receiver, new);

There are several causes of overhead here. The first is the class lookup. In the new runtime, the class table is implemented as a hopscotch hash, which is relatively fast, but a lookup still requires hashing the string, and looking it up in the table. This accounts for the majority of the cost of a class message send.

The second bit of overhead is the class lookup. The new runtime uses the objc_msg_lookup_sender() function, which has a slightly different signature. As I wrote yesterday, you can cache the return value from this call, so we can save a lot of the overhead involved.

The final part is the overhead involved in constructing the call frame for the method and jumping there. This is present even when calling C functions. Overall, this adds up. Sending a million class messages took 56 seconds of CPU time on my machine.

With the new ABI, classes are exported as a public symbol. This means that we can save the cost of the class lookup, as long as that symbol is available. One of the optimizations I committed to the libobjc2 tree today substitutes that symbol for the call to objc_lookup_class(). With that, the cost of a million class messages drops to a bit over 10 seconds. Not bad.

Yesterday, I talked about caching the message lookup for message sends in loops. Class messages always have the same receiver, so they're also a good choice for caching. Another pass that I added today caches all class message sends. Now we're down to only a bit over 4 seconds for a million message sends.

What about the cost of the function? In C, for a small function, we could inline it, but this isn't an option for Objective-C because of the dynamic dispatch. Or is it? The final pass that I added does speculative inlining. This means that it inlines the function that it guesses will be called, and wraps it in a test. If the (cached) lookup returns the function that we are expecting, we go down the inlined path. If not, we call the returned function pointer. The current pass always inlines class methods if possible, but I'll change that soon so that it only inlines them if it's also sensible.

With speculative inlining, we're now down to 2 seconds for a million class messages. For comparison, a million C function calls took 3 seconds on the same machine.

That's the sort of performance I'm aiming for. And, because these optimizations are all done at the LLVM layer, they will work with both Objective-C and Smalltalk. They depend on libobjc2, although it should be possible to implement something similar for Apple's runtime (but not for the old GNU runtime).

Google Summer of Code

Posted on 28 April 2010 by David Chisnall

To start with, the bad news (which is not really news anymore) is that Étoilé was, once again, not accepted for the GSoC this year and, unfortunately, neither was GNUstep. The good news is that the GNU Project was, and GNUstep was allowed to participate under this umbrella.

This is really great for Étoilé: both of the students accepted to work on GNUstep this summer are active Étoilé contributors.

Niels is going to be mentored by Fred Kiefer, the GNUstep AppKit maintainer, and will be working on the DBUS to Distributed Objects bridge. For those unfamiliar with DBUS, it's an ugly, badly designed, copy of Distributed Objects, created about a decade and a half later. At the protocol level, it contains roughly the same information as Distributed Objects. With a working bridge, we'll be able to expose our objects as DBUS services and use DBUS services as objects, without any extra code.

The other project is the one that I get to mentor. Eric is continuing his excellent work on Opal, which is the CoreGraphics implementation that sat unloved in Étoilé svn for a year or two before being moved to GNUstep. Eric's already implemented shadows and layers, and the next step is improving the text rendering support. Shadows are not essential, although they do make lots of things look nicer.

Layers are essential if we want to implement CoreAnimation, which (as well as making things look pretty) has a rendering model designed to be efficient on modern hardware. NeXT-era graphics hardware had very little RAM and caching rendered images was not really feasible. Now, even a handheld has more video RAM than a NeXT workstation had total RAM (and a much smaller screen), but redrawing things consumes CPU or GPU power and shortens the battery life, so it makes more sense to keep as much cached as possible. The layer model in CoreGraphics and CoreAnimation lets each view (potentially) have its own buffer, so redrawing just means compositing that layer onto the window; something a relatively recent GPU can do with no effort.

While Eric's working on CoreGraphics, I plan on tidying up some of the typography code that I've been playing with over the last few years and working on a CoreText implementation on top of Opal.

These two projects have a lot of potential to improve the foundations that Étoilé is built upon.

Smalltalk and Objective-C Performance

Posted on 27 April 2010 by David Chisnall

If you find yourself optimizing your code, then it means that the author of your compiler has failed. In Étoilé, we use two languages; Smalltalk and Objective-C. In theory, we can also use EScript (a dialect of ECMAScript using LanguageKit, but I don't think anyone ever has aside from a couple of examples).

Clang, which compiles Objective-C, and LanguageKit, which compiles Smalltalk, both use LLVM for code generation. This means that they both produce an intermediate representation in the same form. LLVM provides a lot of infrastructure for transforming this intermediate representation, which is how you implement compiler optimizations.

As part of the libobjc2 project, I've been writing a few of these that speed up code targeting the new runtime (which both Clang and LanguageKit do). The first of these passes is very simple. The new runtime adds support for non-fragile instance variables. With older Objective-C implementations, instance variables were accessed via a fixed offset. This is nice and fast, but it means that, if you modify one class's instance variable layout (including just adding an ivar), then you must recompile all subclasses.

With non-fragile ivars, you access all instance variables via an indirection variable. This records the offset of the ivar, and is fixed up by the runtime when the class is loaded. This means that you always get the right offset, even if other ivars are rearranged. This is great, but sometimes you don't actually need it. If your class inherits directly from NSObject, for example, or from intermediate classes declared in the same library (which, it turns out, includes about 90% of classes), this extra overhead is for no benefit, because you will be recompiling the subclasses when you recompile the class anyway.

The ivar lowering pass reverses this. It introduces hard-coded ivar offsets if it is safe to do so without increasing the fragility of the library. This means that you only get the performance penalty from non-fragile ivars if you actually need them, for example when you subclass something from a third-party library.

This isn't especially interesting. The extra cost of indirect ivar accesses is really small. You're only likely to notice it if you are doing a huge number of ivar accesses in different objects and your cache is full.

Message sends, on the other hand, have a big impact on performance. Every time you send a message in Objective-C or Smalltalk, you need to do a dynamic lookup to find the method to call, then you call the method in the same way that you call a C function. Both of these have some cost.

One way you can work around this in code is IMP caching, where you perform the lookup yourself, store the result, then call it as a function pointer. The compiler couldn't do this itself, because caching the IMP (instance method pointer - the function pointer for the method) broke some of the dynamic features of Objective-C. If you changed the selector to method mapping (either explicitly via runtime functions or by loading a category) then the cache becomes invalid and there was no mechanism for the runtime to invalidate it.

This changed with the new runtime. Now, the method lookup function returns a slot, which can be safely cached and invalidated later. The new pass makes use of this to automatically cache slots for message sends that happen in loops. To test it, I wrote a simple benchmark program that sends a message 1,000,000,000 times. The method does nothing, it just returns immediately, so all of the time is spent in the message lookup and sending.

Unoptimized, this takes 10 seconds. With the auto-caching pass, it takes around 4.6 seconds. Adding in the normal set of optimizations, these times drop to 8 and 3 seconds, respectively. For reference, replacing the message send with a C function call makes the time 3.5 seconds, so Objective-C is very, very close to raw C performance in this case. Note that if the function is in a different library, as it often is, you have to go via a relocation table, which brings the speeds much closer, and can even make the cached Objective-C version faster.

This isn't the end, of course. One thing that you can do easily in C/C++, but not Objective-C, is inline a function. This involves replacing a call to a function with a copy of the function body. This eliminates the call overhead and lets you do some other optimizations easily, like constant and subexpression propagation between functions.

We can do inlining with Objective-C too, in theory, but it has to be speculative. We can inline methods, then wrap the inlined version in a test that checks that it really is the correct one. That's next on the list.

We probably can't get Smalltalk quite as fast as C, but if it's within 10-20%, there's very little reason not to use it.

EtoileText, LaTeX, and HTML

Posted on 16 April 2010 by David Chisnall

In a dramatic break with tradition, I've recently been working on some things in Étoilé that are actually useful. When I'm not hacking on Étoilé, the thing I'm actually paid for is writing. I've had two books published, and the third one is currently undergoing technical review.

When I write, I like to use a custom form of semantic markup, using LaTeX syntax. LaTeX, for those who have not encountered it, is the abomination that caused me to lose all respect for Donald Knuth. There is simply no excuse for anyone to design a programming language that does not include concepts like scoping, or any support for structured programming. LaTeX is a programming language that looks like someone (Knuth) thought a Turing Machine was actually a sane programing model, rather than a useful theoretical tool.

So why do I use LaTeX? Two reasons. One is that TeX-style markup is easy to type. Entering XML markup is too distracting, but TeX is very simple and quick to enter, so doesn't interrupt my flow. The other is that the output is really beautiful.

I don't really use LaTeX though, I use a set of custom macros built on top of it, just as LaTeX itself is built on top of TeX. For example, if I write \keyword{Smalltalk}, then the resulting file will have Smalltalk written in the keyword style and also added to the index. If I write \code{NSObject} then NSObject emitted as syntax-highlighted Objective-C code.

I have a few hundred lines of LaTeX code that does this translation. It's the only way to use LaTeX and remain (moderately) sane. Like Lisp, LaTeX is a language for writing languages, rather than a language for using directly.

For my next book, however, the rise of the ePub format (another horrible format, but that's another issue) means that the publisher wants an HTML version of the text.

There are some good tools for mapping LaTeX to XHTML. The best of these is tex4ht, which runs a full TeX virtual machine and then transforms the output into HTML. The problem with this is that it loses all of my nice semantic markup, and styling it is a problem.

I want to be able to define a mapping from my TeX-style semantic markup to XHTML semantic markup. For example, I want \code{NSObject} to become <span class="code">NSObject<span>.

This is where EtoileText fits in. EtoileText is the framework that I've been working on a bit over the last couple of months for editing structured, semantically tagged, rich text. It maintains something conceptually similar to a DOM tree. Text is only stored in leaf nodes of the tree. Each parent node may contain a semantic type and custom presentation attributes.

You can define translators that map the semantic types to AppKit presentation attributes, and plug it directly into NSTextView. Alternatively, you can use the visitor API to generate some other output. The latter is exactly what I've been doing.

I've added a simple (and incomplete) parser for TeX-style markup, which constructs an EtoileText tree from my LaTeX sources, preserving all of my semantic tagging. Then, a visitor walks this tree and emits semantic XHTML. Add to this about half a dozen lines of CSS, and you end up with an XHTML version of the LaTeX sources that looks correct: Not exactly the same as the PDF, but how the markup would look if it were typeset for a browser window instead of a printed page. This uses the ETXMLWriter class from

The TeX parsing classes in EtoileText are not meant to be a general TeX parser (this is actually a non-computable problem; reason #135 why I hate LaTeX). They are intended as a set of tools for building special-purpose parsers for semantic markup languages implemented in TeX. The ones that start with the ETTeX prefix correspond to standard LaTeX commands, the ones with the TRTeX prefix correspond to my own set of macros.

The code is very rough around the edges at the moment. When it's a little more complete, I'll split the TR* classes out into a separate example program.

All of these classes are designed to be usable easily from Smalltalk, which means that you can use Smalltalk to extend it, just as you can extend LaTeX with macros.

Over the summer, I hope to spend some time working on a CoreText implementation and improvements to the GNUstep text system, so hopefully we'll be able to produce typeset PDFs from EtoileText trees that look as beautiful as anything that LaTeX produces by the end of the year. Nicolas keeps promising to work on a structured text editing component for Étoilé too, so hopefully I'll be able to work entirely inside Étoilé soon...

LanguageKit SmallInt Improvements

Posted on 26 January 2010 by David Chisnall

When you use a selector name or instance variable that was declared in Objective-C from LanguageKit, you inherit its types. This is very important for interoperability. If you wrote an object pointer into an instance variable declared as int, or wrote a small integer value into an instance variable declared as an id, then the next time Objective-C code tried to access this value you'd get a crash.

When I started working on LanguageKit, I followed the Objective-C rules that everything was an id until proven otherwise. SmallInts were stored on the stack, but never passed as arguments to methods, never returned from methods, and never stored in instance variables.

This condition is now relaxed somewhat. Instance variables declared in LanguageKit code are now allowed to contain SmallInts. Methods that are not present in Objective-C may take SmallInts as arguments and may return them. This means that LanguageKit is now doing a lot less pointless BigInt creation.

What does this mean in terms of performance? Let's return to the trusty Fibonacci benchmark to find out. This calculates the 30th Fibonacci number, 100 times in a loop. I've implemented this in C, Objective-C, and Smalltalk. Now, the Smalltalk version comes in two flavours. The first always returns an int, so values that won't fit in an int will be truncated (as with Objective-C). The second returns an LKObject, so small integers will be hidden inside a pointer, big integers will be returned in BigInt instances. The performance numbers are:

C fibonacci execution took 2.351562 seconds.  
ObjC fibonacci execution took 6.601562 seconds.  
Ratio: 2.807309 (to C)
Smalltalk fibonacci execution took 8.750000 seconds.  
Ratio: 1.325444 (to Objective-C)
Smalltalk fibonacci SmallInt version execution took 5.687500 seconds.  
Ratio: 0.861538 (to Objective-C)

Note that these were done in a VM, and the timing results are slightly wonky. On subsequent runs, the ratios remained roughly constant, but the absolute times varied by up to 50% in either direction. Although the last line looks like the Smalltalk code is faster than Objective-C (which would be very nice), it's quite unlikely that this is really the case. Please don't start citing this blog as claiming that Smalltalk is faster than Objective-C.

The take home message, however, is that using LanguageKit SmallInts is sufficiently close in terms of performance to using C ints that it's difficult to accurately measure. Using SmallInts also makes you safe from overflow. The Smalltalk version will just get really slow when you can no longer fit the value in a SmallInt. The C and Objective-C versions will start giving you the wrong answer.

Perhaps more interesting is the fact that the Smalltalk version that was shoehorned into the Objective-C type system was slower. The extra overhead of converting to and from ints was noticeable.

Running it several times, I got one result where the SmallInt version took 20% longer than the Objective-C version. This was the worst case result for LanguageKit, but is probably the most representative of real-world performance. If you can afford using 20% more cycles, and aren't doing anything floating-point intensive (LanguageKit's floating point performance still sucks), then there isn't much reason to choose Objective-C over Smalltalk.

It's worth noting that the C version was much faster than either. This kind of recursive call is the absolute best case for polymorphic inline caching. The new runtime supports this optimisation, but I haven't added it to the compiler yet. I'm planning on doing it via an LLVM optimisation pass, so both Objective-C (compiled with clang) and Smalltalk will benefit. This should remove most of the cost of the method lookup, and even allow speculative inlining of calls. I expect that we can get Smalltalk performance closer to 5 seconds for this benchmark.

As always, this benchmark needs to come with a reminder that using a less stupid algorithm is orders of magnitude faster and a sensible Fibonacci implementation is faster even in the LanguageKit interpreter than a stupid algorithm in C. Good algorithms with bad compilers always beat good compilers with bad algorithms.

OMeta and Benchmarks

Posted on 9 January 2010 by David Chisnall

This week, Günther committed his implementation of OMeta. OMeta is a shiny way of writing parsers from the Viewpoints Research Institute. It's a really simple way of writing domain specific languages and it's great to have an implementation in Étoilé. It will be fun to see what people will use it for. Eric is planning on writing a Smalltalk parser in OMeta, so we can have a completely self-hosting parser.

Glancing over the code, I noticed that Günther had implemented quite a few things in a category on BigInt. This is how you add operations to integers with LanguageKit, and all of the ones he'd added looked useful. I moved this into the SmallInt and BigInt implementations supplied with the compiler, so now they execute in (very fast) inlined C functions if the receiver is a small integer, rather than in (slower) Smalltalk.

While I was hacking on BigInt, I also added an implementation of a class that boxes floating point values and Eric added support in the parser for floating point literals. You can now use floating point values in Smalltalk, although they are quite slow. I'll probably work on optimising this a bit later, but I can't really be bothered now, because you can just write performance-critical floating point code in an Objective-C method if it's not fast enough in Smalltalk.

Since it's a fairly complex piece of code, it seemed like a good thing to use for some real-world benchmarking of LanguageKit. Unfortunately, this is where I started to hit problems. It ran fine with the JIT compiler, but not the JTL or interpreter.

I spent quite a while hunting bugs in the interpreter - you can see the svn log for details if you care about them - and then moved on to the JTL. It turned out that the problem with the JTL was an LLVM bug, not a LanguageKit problem. Linking together LLVM bitcode modules containing global aliases that pointed to bitcasts was broken. I've now fixed that, so it's worth upgrading LLVM to r93052 or later if you want LanguageKit to work properly.

After that, I could run Günther's OMeta tests. You can see a summary of the results in this table:

The wall clock time and user time are reported by the time utility when running edlc. The Smalltalk time is the time reported by edlc as the time taken for the SmalltalkTool class's -run method to complete. This is not reported by the JTL because the code is run by the bundle loader in LanguageKit and can't be separated out by the tool.

As you can see, JIT-compiled code is about 41 times faster than interpreted code. We can probably make the interpreter a bit faster, because it's currently quite a naïve implementation, but given that we can get a big performance benefit just from using the JIT, it might not be worth it.

The JIT has a long start-up time. Most of this time is spent by LLVM optimisation passes. Note the difference between the debug and release LLVM builds. Disabling all of the assert() statements in LLVM and enabling compiler optimisations when building the JIT makes a huge difference to the performance; going from 16 to two seconds of CPU time. I didn't do this when I originally posted this entry because I always forget that I'm using a debug build (I hack on LLVM as well as Étoilé). With a release build, it's significantly faster. Slightly slower than the interpreter overall, but for longer-running programs this will go away.

One of the things we can do now is use the interpreter by default and then compile and install methods in the background, and only once they have been run a few times.

Günther left a comment saying that it might be worth rewriting the OMeta stuff in Objective-C, but given that the tests take a fraction of a second to run I don't think that's particularly worthwhile.

Smalltalk Workspace and other LanguageKit News

Posted on 4 January 2010 by David Chisnall

It's been a while since I talked about LanguageKit, so I'd like to take this opportunity to introduce a few new features. One is an improvement to error reporting. Previously, warnings were sent to the standard error, while errors were reported via an exception.

These were fine for the command-line compiler tool, but not very flexible. One thing that I've been meaning to do for ages, and which I finally did just before Christmas, was to factor this out into a delegate. All errors and warnings are now sent to the delegate. This lets you do some quite nice things, including repair the AST without reparsing. More on this later.

The other nice new feature is something that I did almost none of the work for: an interpreter. LanguageKit includes JIT and static compilation already, but interpreting has a few advantages over these approaches. If you only run a statement once, a slow interpreter is still faster than compiling and optimising the statement. The interpreter is very useful for development, because you can trivially change the code in an interpreter, for example. Eric wrote most of the interpreter code. Unlike the compiler, the interpreter runs on Cocoa, as well as GNUstep.

In future, I plan on incorporating profiling into the interpreter and JIT compiler. We can then transparently recompile code based on profiling information. This includes things like speculative inlining, which the new runtime makes possible.

Eric's combined these two features to create something new; a Smalltalk Workspace. This is a feature that most Smalltalk implementations have. You can type Smalltalk statements into a window, select them, and run them. You can now do that in Étoilé. Currently, this uses the interpreter. In future, things like blocks will automatically be JIT compiled if they are executed more than a few times.

If you want to play with it, you'll need the latest LanguageKit svn and the SmalltalkWorkspace from Developer/Services in svn. If you use ETTranscript for output, as most of the example scripts do, then you'll notice that it works correctly with the workspace, redirecting the output to the window, rather than sending it to the terminal. The -log method on objects still goes to the standard error.

Gratuitous Book Plug!

Posted on 19 December 2009 by David Chisnall

Today's blog post isn't really about Étoilé, so feel free not to read it. Instead, I'm going to talk about my new book: Cocoa Programming Developer's Handbook.

Étoilé is based on GNUstep, which implements the same set of core APIs as Apple's Cocoa. Both implement the OpenStep specification and both extend it in various ways. GNUstep tries to follow Cocoa's extensions and a number of GNUstep's extensions are available on OS X via the GNUstep Additions framework.

Apart from one appendix, covering porting Cocoa applications, the book does not contain much specific to GNUstep (although a few of the examples are based on Étoilé code). That doesn't mean that it's completely useless to GNUstep and Étoilé developers, however. Most chapters contain something relevant to people on other platforms:

1 Cocoa and Mac OS X talks about the history of OpenStep and Cocoa and where it fits into OS X.

2 Cocoa Language Options introduces Objective-C and discusses the choice of compilers. Most of this is applicable to GNUstep. I don't think there are Python bindings for GNUstep, but there are Java and Ruby bridges. For Étoilé, of course, you can use LanguageKit and write Smalltalk code.

3 Using Apple's Developer Tools introduces XCode and Interface Builder, which are specific to OS X. The introduction to Objective-C and the conventions in Cocoa, which account for most of this chapter, are relevant to GNUstep developers.

4 Foundation: The Objective-C Standard Library introduces the Foundation framework. Aside from the section on Core Foundation, everything here should be relevant to GNUstep - I finished the NSCache implementation before the book was published - and support for Core Foundation APIs in GNUstep is under development.

5 Application Concepts introduces the core concepts in the AppKit. Everything here should apply to GNUstep-gui.

6 Creating Graphical User Interfaces covers more of AppKit. I've not tried using Cocoa Bindings, which this chapter explains, with GNUstep. The code looks as if it would work, but there is no interface in GORM for connecting them so you'd need to do that in code (or improve GORM).

7 Windows and Menus also mainly covers things that will work in GNUstep. Note that GNUstep doesn't support sheets (yet). The same APIs work, but you get window-modal dialog boxes instead.

8 Text in Cocoa talks about the Cocoa text system. Most of this is unchanged from OpenStep and should work with GNUstep. There are a few bits that won't. Rather embarrassingly, one of the examples didn't compile with GNUstep because of a typo in a GNUstep header (fixed now). Fred is doing some great work on the text system at the moment, so if any of it doesn't work then it probably will soon.

9 Creating Document-Driven Applications introduces the NSDocument family, which is very well supported by GNUstep. You can take the Apple developer examples that explore this part of the system and compile them with pbxbuild without any problems.

10 Core Data introduces the Core Data framework. GNUstep provides the gscoredata framework, which is intended to be a drop-in replacement. I've not used this (Étoilé's CoreObject is nicer), but apparently it works.

11 Working with Structured Data describes the more complex view classes in AppKit. NSCollectionView doesn't yet exist in GNUstep, but EtoileUI provides a nicer way of producing this kind of layout (and works on both GNUstep and Cocoa).

12 Dynamic Views talks about modifying the view hierarchy at run time. All of this is standard OpenStep stuff with the exception of the part talking about full-screen applications. This uses some low-level Quartz calls, which won't work with GNUstep.

13 Custom Views teaches you how to write your own views. The CoreGraphics stuff won't work with GNUstep yet, although we have a partial implementation of CoreGraphics in Étoilé svn and are trying to get it moved into GNUstep. The section on creating Interface Builder palettes may also not be directly applicable to GNUstep. Gorm supports palettes in a similar way, but I don't know how similar the interfaces are. This is only important if you are packaging a framework with new view objects for third-party developers and want to add a little polish.

14 Sound and Video contains a lot that is specific to OS X. None of the QuickTime stuff will work with GNUstep, although Étoilé's MediaKit fills the same rôle (and might work on OS X). I committed support for speech synthesis to GNUstep a while ago, but speech recognition is still missing. If anyone wants to wrap (Pocket)Sphinx and implement NSSpeechRecognizer before I get around to it, patches are most welcome!

15 Advanced Visual Effects covers a lot of stuff that is only available on OS X. CoreAnimation, CoreImage, and Quartz Composer only work on OS X. OpenGL works on any platform and there have been some patches added to GNUstep's NSOpenGLView recently by people porting Cocoa apps or writing OpenGL apps from scratch with GNUstep.

16 Supporting PDF and HTML covers PDFKit and WebKit. Gregory is currently working on porting WebKit to GNUstep, so these examples should work in the next few months. PopplerKit in Étoilé svn provides a lot of the same features as PDFKit, but is GPL'd. Writing a new PDF framework is on my TODO list, but it's a very long way down so don't hold your breath...

17 Searching and Filtering covers SearchKit and Spotlight, which are not available off OS X. Étoilé's LuceneKit, based on Apache Lucene, is very similar to SearchKit. The NSPredicate stuff in this chapter is also relevant to GNUstep, and Étoilé uses predicates in a few places.

18 Contacts, Calendars, and Secrets covers the Address Book, Calendar Store and Keychain APIs. We have an implementation of the Address Book APIs in svn, which we will be replacing soon with one built on top of CoreObject. The Calendar Store API is quite simple, but hasn't yet been reimplemented. I have an implementation of the subset of the Keychain API that I talk about in this chapter somewhere, which I'll commit once I've finished tidying it (it's difficult to do securely without Mach ports).

19 Pasteboards covers the pasteboard APIs. These are the same on GNUstep, although the 10.6 APIs (which are much, much, cleaner) aren't implemented yet. I like these APIs a lot, so I'll probably implement them soon.

20 Services covers the system services mechanism that Apple inherited from NeXT. This is one of my favourite features of GNUstep and works well. We have a few things that use it in svn.

21 Adding Scripting talks about AppleScript. If you read and understand these APIs, you'll know why no one has bothered reimplementing them. This chapter is around 30 pages. You can describe how to do the same thing with Étoilé's ScriptKit in about half a page.

22 Networking talks about sockets, NSStream, the URL loading system, distributed objects, and Bonjour. All of these should work with GNUstep, although I've not tested NSNetService.

23 Concurrency talks about various ways of writing parallel code. The NSOperation implementation in GNUstep is quite immature, so probably won't work. Grand Central works on FreeBSD, but not yet anywhere else.

24 Portable Cocoa is obviously relevant to GNUstep.

25 Advanced Tricks covers some tricks with the runtime system. Most of these work as-is on non-Apple platforms, and a few have example code for the GNU runtime.

A lot of the examples from the book use declared properties, so if you want to try them with GNUstep you'll need to use clang (and either libobjc2 or link Étoilé's Objective-C 2 framework). I found declared properties very useful in writing concise code for the examples and this provided me with the motivation to get them working on GNUstep (they do now).

InformIT sells both the printed copy and the (DRM-free PDF) eBook edition, while Amazon has the dead-tree edition for a bit less.

Compilers, Runtimes, and Web Apps

Posted on 19 November 2009 by David Chisnall

As regular readers will know, as well as hacking on Étoilé, I also maintain the GNUstep Objective-C runtime and associated support in clang. Recently, I started working on something that we discussed at the hackathon: web app integration.

Étoilé has lots of features that would make it convenient for writing web applications. The EtoileUI framework, for example, lets you specify user interfaces in quite an abstract way. We would like, for example, to be able to take the EtoileUI tree and turn this into a tree of web-based views, just as we currently turn it into a tree of AppKit views. This would make it trivial for all Étoilé user interfaces to have an 'export to web' menu button, so when you are away from your computer you can press this button and access your Étoilé programs from a remote browser.

With this support, it might also be nice to be able to write stand-alone web applications with Étoilé. That's more or less what I've been working on for the last week or so. I now have a set of classes that talk FastCGI to a web server and handle events and session management. This will be used to construct a set of MVC views that can be used from EtoileUI or directly.

While I was working on it, I got a bit bored writing accessor methods, so I decided to use clang, which supports declared properties. I've also been using the GNUstep runtime on my own machine for a while. When using clang, it made sense to use the new non-fragile ABI supported by this runtime.

Of course, doing so immediately broke everything. After a bit of bug fixing, in both clang and the runtime, I got the code working with the new ABI. This properly supports properties, non-fragile instance variables, and fast forwarding, among other improvements.

Clang is now able to compile most of Étoilé with the new ABI. This is backwards compatible with the old one, so you can still link against libraries (like GNUstep, which uses @defs and so doesn't support the non-fragile ABI) compiled with the GCC ABI. There are a few more bugs to fix, but then we should be able to fully support clang as a compiler for Étoilé. In the meantime, the GNUstep runtime is working as a drop-in replacement for the GCC runtime.

CoreObject over XMPP

Posted on 15 September 2009 by David Chisnall

One of the best things about hippyware development is when someone else implements something you'd planned to do. If you've been following this blog for a few years, you'll remember that one of the ideas I had very early on when I started working on EtoileSerialize (the low-level part of CoreObject) was to be able to send objects remotely for automatic whiteboarding of arbitrary objects. Today, in SILC, Niels announced:

Niels Grewe: I just send the first object over XMPP :-D

For the past few weeks, Niels has been working on tidying up some of the rough edges of XMPPKit and EtoileSerialize and joining them together. He has rewritten the XML serializer and added a deserializer.

At the hackathon this year, I added an ETXMLWriter class and protocol, which implement a similar set of APIs to the ETXMLParser delegate (this is great for testing, because you can plug the two together and just check that the input matches the output). This is now used by XMPPKit and is also used by Niels' new code in EtoileSerialise. This makes it relatively easy to join the two together, so you can embed serialized objects in an XMPP stream and send them to other users.

Although the code is still quite messy, and Niels is tidying it before committing, it has a lot of potential. Trivial whiteboarding from every Étoilé application is the most obvious use; we can send CoreObject invocations between users so they can both edit the same documents easily. Another exciting possibility is live backups. While you are editing a document, all of the changes you make are recorded to disk by CodeObject, so a power failure won't lose you anything except, maybe, the couple of edits if your OS doesn't commit them to disk fast enough. But what happens if the disk dies, or you drop your laptop? With CoreObject streaming the changes over XMPP (which doesn't take a huge amount of bandwidth), you can start working again exactly where you left off, as soon as you find a new computer to work on.

I hope to put together some demos of this over the next few months. It probably won't make it into 0.4.2, but it should be in 0.5.

The Étoilé Runtime is dead, long live the GNUstep runtime!

Posted on 10 September 2009 by David Chisnall

A few years ago, I wrote a new Objective-C runtime, which had lots of new and exciting features, but one big disadvantage; it was not backwards compatible. Since then, I've been heavily involved in clang and have implemented support there for the GNU runtime. I added preliminary support for the new runtime, but it bitrotted before I finished it.

After looking at the GNUstep code, I came to the conclusion that updating it to support a new runtime would be very difficult. In particular, changing the type of IMP would have broken a lot of existing code.

Since then, I've also been working on ObjectiveC2.framework in Étoilé svn. This provides an implementation of Apple's new runtime API on top of the GNU runtime. This makes it easier to port code from OS X to GNUstep and also implements a few of the newer functions that clang is now adding calls to for Objective-C 2 support. These include bits of the language like fast enumeration (for...in loops), @synchronize(), and declared properties.

About a week ago, I started a new project. This has now been committed to the GNUstep repository as libobjc2. The new runtime starts as a fork of the GNU runtime, incorporates the improvements from the ObjectiveC2 framework, a number of ideas from the Étoilé runtime, and a little bit of new stuff.

From the GNU runtime we get a fairly reasonable implementation of the core Objective-C language. From the ObjectiveC2 framework we get most of the things required for Objective-C 2. From the Étoilé runtime we get a safe way of caching method pointers (so the compiler can now safely insert IMP caching and you probably never need to do it in code ever again) and a means of modifying the receiver of messages (so we can now implement incredibly fast proxies for things like CoreObject). The new stuff includes built-in support for Object Planes, support for non-fragile instance variables, introspection on declared properties and optional methods in protocols, and a few other things. The performance characteristics of the GNUstep runtime are similar to those of the Étoilé runtime; slightly slower than the GNU runtime for basic message sends, faster for cached message sends, and much faster for proxy messages.

I finished the compiler support for all of the new features last night and will be committing it to clang once it's undergone code review. In the meantime, I've been updating GNUstep so that it now supports all of the new magic on NSObject added with 10.5 and 10.6. I've also added type introspection on blocks, so we can use them with ETPrototype safely, just as we use Smalltalk blocks now.

The Étoilé runtime is now official deprecated. It should be regarded as a research prototype. Almost all of the ideas from the paper have now been merged into the GNUstep runtime. The - horribly buggy, bloated, and untested - threading code has now all gone. The current version of the runtime does not support type-dependent dispatch, but it does return the method types in the slot, so they can be tested in LanguageKit to avoid problems with polymorphic selectors. One feature that was dropped was the offset field in slots. After profiling, it was determined that the test for whether this field was present on every message send was costing more than it was saving in the cases where it existed. We can get the same benefit in a more general way by speculatively inlining the accessor methods.

The new runtime is now fully supported by GNUstep, bringing support for all of the 10.5 and 10.6 methods that were added to NSObject. The patches for support in clang are pending review, but should be merged soon. Once this has been done, I will also add LLVM passes that automatically cache the lookup. This optimization code can then be shared between clang and LanguageKit, because it will work on the LLVM IR generated by either.

It's a Bitter Sweet Compiler that's Just Too Late...

Posted on 23 July 2009 by David Chisnall

LanguageKit has supported just-in-time (JIT) compiling for a while, but in my last commit I added support for just-too-late (JTL) compiling too. The idea behind JIT compilation is that you only generate the executable code just before it's called. With JTL compilation, we generate it afterwards.

On the face of it, this isn't very useful. In practice, applications run more than once, and the second time is now a fair bit faster. If you've run Melodie, you'll have noticed that it takes a while longer to start than most GNUstep / Étoilé applications. This is because it is parsing all of the Smalltalk code, compiling it to LLVM bitcode, optimising this, and then starting.

If you use the latest LanguageKit svn (note: it needs a very recent LLVM svn build), you will notice that the second time you load it, it starts a lot faster. This is because it's loading a shared object (.so) file that was generated in the background the first time the program ran. It's currently quite conservative about regenerating this; if any framework or source file is modified then it should fall back to the JIT and run the JTL compiler in the background to refresh the cache.

So, as I write this, Melodie is playing my test album (Bitter Sweet Symphony EP, courtesy of iTunes Plus), without having done any compilation. It doesn't even load the code generation bundles, because the code generator is never instantiated (the bundle with the code generation support is loaded on-demand).

The performance is not yet ideal. In future, the JIT compiler will do some run-time profiling and the JTL compiler will then use the profiled version of the library. This means that every Étoilé application written in a LanguageKit language (e.g. Pragmatic Smalltalk or EScript) will benefit from profiling-driven optimisations, and may even out-perform statically-compiled Objective-C (although profiled Objective-C is likely to still be faster).

There are currently two places where LanguageKit will store the caches; in the source bundle or in the user's home directory. This means that you can benefit from static compilation even when you don't have write access to the bundle (e.g. it's an application installed by root).

This is something I've been meaning to do with LanguageKit for a long time, so it's nice to see it working properly. Note that this works with all LanguageKit bundles, not just applications. If you put a plugin in your LKPlugins directory, for example, it will benefit from the same kind of caching.

Summer Update - Video, Reflection, Model Description

Posted on 11 July 2009 by Eric Wasylishen

My summer break is about half over, and as the Étoilé "summer student" developer, I thought I should give everyone an update on what I've been working on. The past two months have been a bit unfocused:

• Inspired by Nicolas' patch to make NSSplitView resizing "live" in GNUstep, I wrote a similar patch for NSTableView. Now, when resizing or moving table columns, the table is updated in real-time. In my opinion, these small changes really make GNUstep feel more modern.
• I wrote a prototype of a movie view for MediaKit. There are two versions: one is software-only (FFMPEG handles decoding, colorspace conversion, scaling, and the resulting bitmap data is painted on an NSView), the other uses an NSOpenGLView to scale the video in hardware. This was really quite easy (a few hundred lines of code): the FFMPEG libraries do all of the hard work. This isn't in MediaKit yet - we need to improve the threading architecture a bit to handle video - but it shouldn't be too difficult to finish.
• Since last year, I've wanted to write a reflection API for Etoile based on mirrors (see Mirrors: Design Principles for Meta-level Facilities of Object-Oriented Programming Languages). I finally wrote a basic implementation: (Header, Source, Test), written using the OS X Objective-C 2.0 Runtime API - so it works on both OS X and GNUstep (thanks to David's ObjectiveC2 framework). My implementation is read-only and only reflects the runtime state of the program. The great thing about mirrors is the idea that you should be able to use the same API to examine and manipulate running code as you use to manipulate source code. In the future I would like to extend it with write access (so we can add and remove classes and methods), sub-method reflection integrated with LanguageKit (specific to each language provided by LanguageKit), and the ability to mirror source code.

Here is a simple class browser written in Smalltalk:

• At the end of the hackathon, Quentin explained the idea of model description to me, and I thought it was really cool. Similar to how mirrors separate meta-information about an object's structure (class, instance variables) in to a separate object, a model description object separates meta-information about a model object's role in the overall model. For example, the model description could list the properties of the model object, which are read-only or read-write, the type of value permitted for the property (using UTI's), whether the property is multivalued, etc. I'm working on something similar to FAME, and hope to commit a basic version of it in the next few days.

Lastly, I've been working on polishing the website. There is still more to do, but you'll notice there is now more content, and hopefully it is easier to navigate.

As for the rest of the summer, there are several projects I would like to work on:

• UI design. From my perspective, we have a consensus on the goals of Etoile, and the UI of some parts of the system. As we're getting closer to implementing more of it, I think it would be useful to continue design discussions and plan some of the smaller details.
• CoreObject development. I think the model description framework will help modularize and simplify CoreObject, and I want to investigate serialization of objects using the model description (which FAME does)
• Work with EtoileUI more; prototype better UI's for Melodie and the photo manager
• Start the messaging framework (chat, email, etc.)

Higher-order messaging in EtoileFoundation

Posted on 2 July 2009 by Niels Grewe

Over the past weeks, there have been some interesting additions to the EtoileFoundation framework to provide higher-order messaging facilities. Higher-order messaging means having (or at least: having the illusion of having) messages that take messages as their arguments. This can turn out to be quite handy at times. For example, if you are using any class that descends from NSObject, thanks to Quentin you now have a -ifResponds method at your disposal, which frees you from the fear that you might send a message to an object that it doesn't understood.

Borrowing an example from Quentin's documentation of that method, suppose you have cats and dogs in your code zoo where your Dog class implements a -bark method, but the Cat class doesn't. The following naïve approach will then cause an exception to be thrown, because cat does not respond to -bark:

[dog bark];
[cat bark];

If you send the message to -ifResponds, you can make sure that this doesn't happen:

[[dog ifResponds] bark];
[[cat ifRepsonds] bark];

This is neat, especially when you're sending a message to an object that might or might not implement the method but don't care about the case that the message is not understood. Internally you are, of course, not sending a message to a message. -ifResponds rather returns a proxy object that, upon a message send, will forward the message to the original object if it responds to it. Otherwise it will simply return nil.

Another example of higher-order messaging at its best is the -inNewThread method from EtoileThread, which has been around for a while. Any message you send to this will be executed in a new thread, meaning that your programme will only block when you try to access the return value. Sometimes it can be really nice if you don't need to wait for a lengthy operation to complete, especially if you are a hungry cat:

[[dog inNewThread] takeForALongWalk];
[cat feed];

Higher-Order Messaging for Collections

The other addition, which I managed to squeeze into EtoileFoundation with much help and support from David and Quentin, is higher-order messaging related to collections. These higher-order messaging methods make it really easy to manipulate the objects in a collection. One of them is -mappedCollection, which allows you to send messages to every object in the collection, giving you a new collection with the manipulated objects. Since EtoileFoundation provides the useful ETCollection protocol, which aggregates the collection classes from the GNUstep/Cocoa Foundation framework, this works indiscriminately with NSArray, NSDictionary, NSSet, NSIndexSet and their subclasses.

Even if you subclassed one of those yourself and have special needs on how to handle the elements in your custom collection you can still use the existing mechanism by implementing the two methods beginning with -placeObject:. This is already used by some classes, as you can see in the ETCollection+HOM.m source file.

Consider the following example: You have an NSArray named fruitBasket which holds several instances of a class named Fruit which implements a -peeledFruit method to return an instance of its PeeledFruit subclass. How would one solve the problem of peeling all the fruit in the basket? The conventional way would be to enumerate all the objects in the array, sending the -peeledFruit message to each and placing the object returned in a new array, yielding roughly between 5 and 15 lines of code, depending on whether you (can) use fast enumeration and how paranoid you are about sending the message to a wrong object. -mappedCollection helps you to reduce that code to just one line:

NSArray *basketOfPeeledFruit = (NSArray*)[[fruitBasket mappedCollection] peeledFruit];

If fruitBasket had been a mutable collection, you could also have used -map, which modifies the collection directly. There is also another useful method using the same pattern: The -filter method allows you to send a message returning BOOL to each element in the collection to determine whether it should remain in the collection or be removed. This would, for example, allow me to make sure that there are no foul fruit in my basket prior to peeling them. Unfortunately, -filter is only available in a variant that modifies the collection directly, so I have no way to keep the original basket and return a new collection. Why?

The problem with returning a new collection after filtering is, that, again, the whole higher-order messaging stuff is strictly speaking a con. -mappedCollection, -map and -filter (as well as -leftFold and -rightFold, see below) in truth return proxy objects that forward the messages you send to them to each object in the original collection. There is no such thing as higher-order messaging from the compiler's perspective. If you want to return a new collection, everything will work well as long as you only send messages to the proxy that return pointers to objects.

In the above case, the compiler would look at the return value of the method -peeledFruit and allocate enough space to hold a pointer to a PeeledFruit instance. But the proxy object does in fact return an NSArray. This is why you need to cast the return value appropriately, but that is no problem since both are pointers to some area in memory and thus are of the same size. This is not the case if you send a message to a filter proxy. It will return BOOL and the compiler will only allocate enough space for a BOOL return value. If the filter-proxy did return a pointer to the filtered collection, it would not fit into the BOOL-sized portion of memory reserved by the compiler. Terrible things could ensue, and that's why it's better not to do it.

So let's assume our fruitBasket is in fact mutable and the “Fruit”-class implements -isNotFoul (which returns a BOOL-value), we can remove the bad fruit from the basket and peel them afterwards by saying:

[[fruitBasket filter] isNotFoul];
[[fruitBasket map] peeledFruit];

The map operation can also serve as a substitute for the KVC method -valueForKey:. Imagine that every month my phone company will send an array of records for the calls I've placed and received (named callLog). Each record is an NSDictionary containing information about a single call: The names and numbers of caller and callee, duration of the call etc. If I want to get a list of the people I called, I have the following two options, one using higher-order messaging, the other using key-value coding:

NSArray *calledPersons = (NSArray*)[[callLog mappedCollection] objectforKey: @"calleeName"];
NSArray *calledPersons = [callLog valueForKey: @"calleeName"];

Of course, the KVC-variant is arguably more concise, but I'd like to demonstrate how higher-order messaging can help me to find out how much time I spent on the phone this month. I simply extract the “duration”-field from each record:

NSArray *callDurations = (NSArray*)[[callLog mappedCollection] objectForKey: @"duration"];

(Again -valueForKey: could serve the same purpose.) All I need to do now is compute the total of all objects stored in callDurations; a task that can be immensely simplified by two other new methods: -leftFold and -rightFold, which repeatedly invoke a method with an accumulator and each element in a collection, building up the return value in the accumulator. By “left” or “right” you can indicate in what order the elements shall be processed. Since my Duration-class implements -durationByAddingDuration: I can easily find the total this way:

Duration *total = [[callDuration leftFold] durationByAddingDuration: [Duration nullDuration]];

The argument [Duration nullDuration] is used to set up an initial value for the accumulator. I'm setting it to an empty value, since I have no particular use for it. Of course, since addition is a commutative operation, I could have used -rightFold as well. It might also be worth noting that you shouldn't use nil as the initial value of the accumulator, at least for left folds. In left folds the accumulator will be the receiver of the message you send to the proxy object, and if it's set up to nil initially, no message can change that and nil will also be the return value of the operation.

This is nice and convenient, but might not be particularly speedy depending on your use case. Since a message sent to the proxy object goes through the second-chance dispatch mechanism of the Objective-C runtime before it is forwarded to elements of the collection, there is some penalty associated with this initial message send. The more objects there are in your collection, the less likely it is that you will notice it, but in performance critical code you might want to check whether that penalty is still significant.

Using Blocks

But there is also a second area where higher-order messaging might be found wanting. If you are often doing various complex modifications of objects in collections, and want to use -map and friends, you might find yourself adding quite a few methods like -doComplexStuff:withFoo:andBar:andEvenMoreArguments: to your classes, even if you only use them once or twice. If that is the case, and you are lucky enough to have compiled EtoileFoundation with clang (passing the -fblocks switch and linking to the ObjectiveC2 framework), you can also use the new blocks feature of Objective-C 2.0 to achieve something quite similar to higher-order messaging. EtoileFoundation provides analogues to all collection higher-order messaging methods for use with blocks. So we could rewrite the fruit peeling example as follows:

[fruitBasket mapWithBlock: ^(id x){ return [x peeledFruit]; }];

Which, for this simple case, might seem to increase complexity, but is a lot more expressive since the block can include an arbitrary amount of code (and it can even reference variables outside the scope of the block). Additionally, since there is no proxy object needed to which arbitrary messages are sent, there is even a -filteredCollectionWithBlock: method, which is know to return a pointer to an object and won't cause any trouble.

You can get an overview of the implemented methods from the ETCollection+HOM.h header file. I hope that they are going to be useful and that you will have as much fun using them as I had writing them. If you run into any problems please drop me a line. For those interested in more information on higher-order messaging etc. both Marcel Weiher's and Stéphane Ducasse's paper on the topic (PDF) and David's article on Advanced Flow Control for Objective-C are highly recommended.

VirtualBox Dev Image

Posted on 21 June 2009 by Eric Wasylishen

A few weeks ago, I put together a VirtualBox image which has a snapshot of Etoile trunk installed. If you've been wanting to play with Smalltalk or EtoileUI but had trouble compiling everything, hopefully this will be useful! Available here.

Dealing with Documents

Posted on 3 June 2009 by Jesse Ross

The Étoilé team has a pretty strong opinion about how users should be able to work with documents on their computers. It seems as if we're not alone, though. Lukas Mathis has written an article lamenting the state of document creation on modern systems, and then takes a look in the history books to show that the way we do it now isn't the only way, and, most importantly, is probably not the best way. Lukas offers up a good solution for approximating the ideal using OS X and a template-based system. Our proposed solution is closer to OpenDoc, but if you'd like to understand why we feel so strongly about this (as well as read about some of the systems that have influenced us), give the article a read.

Hackathon Roundup

Posted on 24 May 2009 by David Chisnall

Last week saw the second Étoilé hackathon, hosted once again by the Swansea University Computer Science Department. They gave us a nice room overlooking the bay to hack in, and the use of their Internet connection for a bit. The hackathon officially started on Wednesday, but Eric arrived a day early:

Eric: After a few days of being a tourist in London and Bristol, I took the train to Swansea Monday evening. I neglected to print out a map which included both David's house and the train station, so I soon got lost and walked up Town Hill. I eventually found myself on the map, and arrived at David's much later than I should have - we were late for trivia at a local pub, so we headed there right away. I enjoyed meeting Davids' friends, answered (maybe) one trivia question, and tried some local beer. Afterwards, David showed me the results of his initial work on ProjectManager.

We spent some time on Tuesday hacking on LanguageKit on Eric's iBook. A few bits of LanguageKit are specific to the host platform and this was the first time anyone had tested it on PowerPC Linux. By Tuesday evening, LanguageKit (Smalltalk) was passing all of the tests on PowerPC Linux that it passes on other supported platforms. I pointed Eric at the right bits of code for implementing the PowerPC/Linux ABI, but he did the real work and improved the test suite a bit. I hacked on the Project Manager some more.

Nicolas had a traditional experience with British trains, and arrived later than anticipate, while Quentin had the opposite experience, resulting in their meeting at the station.

Quentin: On May 12, I was on the plane from Paris to Bristol to head to the new Étoilé hackathon edition, with my main luggage largely filled with cheese and saucisson. Seven hours later I was in Swansea one hour earlier than expected, so I decided to wait Nicolas a bit since our arrival were roughly in the same time slot. After a short taxi ride, we met Eric and David which were both actively working on getting LanguageKit working on Eric's iBook (a ppc machine).

Nicolas: We mostly spent this first evening discussing about various technical topics and which things we wanted to work on during the rest of the hackathon.

The evening wasn't just spent on technical discussions. We opened a couple of bottles, and each of us took it in turns to do some short demos of stuff we'd been working on.

Quentin: We celebrated Étoilé and the beginning of the new 2009 hackathon organized by David with plenty of good stuff: red wine, bread, saucisson and cheese. Nicolas was pretty quick to suggest the opening of the saucisson and I convinced him we couldn't eat it without some bread :-)

I had a cold the hackathon week, so was falling asleep shortly after midnight most days. The others stayed up a lot later, with Quentin putting together a simple DTP-like application as a demo on the first night.

On Wednesday, we walked down to campus and got our door access set up properly.

One of the things we've been planning for 0.5 is a unified message store and delivery system. Something similar was outlined in The Humane Interface, but without the implementation details. This is something that several of us are likely to implement parts of, so it was important to get a coherent model of how all of the bits would fit together.

Quentin: We ended the morning on the CoreObject unified message store (mail, im, rss etc.) which had been discussed several times on the lists over the previous year. We tried to work out all the involved interactions with GML on the whiteboard. We finally settled on a solution which is still to be implemented. The whiteboard result was the usual multi-layered arrow/box soup we all appreciate ;-)

This year we managed to schedule the hackathon to occur during the university term. This had the advantage that it was possible to find food on campus, so a little while after we'd arrived we headed to an adjacent building for food. We then spent the rest of the afternoon hacking. I spent some time fixing various platform-specific bugs in LanguageKit then hacked a bit more on the Project Manager, while Quentin worked on EtoileUI:

Quentin: After the lunch, I was busy testing the last part of the new EtoileUI event handling on GNUstep. I was happily surprised to discover a large part was working almost flawlessly. In the middle of the afternoon, I started worrying on how I was going able to split all this work in small commits without spending a week on the merge. Although I had already committed many changes related to the new event handling model, the last chunk was a lot bigger and was tangled with other unrelated code I had to fix on the way. Various uncommitted new features/classes and the cross development on both Mac OS X and GNUstep was making everything harder.

Nicolas spent most of the time working on Code Monkey, the Smalltalk IDE build on top of LanguageKit. I spent some time on Wednesday afternoon writing code that could inspect a C stack and extract the Smalltalk StackContext objects, allowing for complete introspection of the stack.

Nicolas: Wednesday found us at the University, where we started hacking on EtoileUI, ETSerialize, Project Manager and CodeMonkey. Commits started to flow in as the day passed, and we ended up the day drinking some wine and eating a great cheese fondue with other Swansea friends.

In the evening, I made a fondu. Apparently this is a specialty of the region of France where Quentin is from (as is Reblochon, one of my favourite cheeses), so he watched very closely while I made it to ensure I wasn't doing anything, as he put it, 'heretical'.

Quentin: After the dinner, Eric convinced to push the big chunk in a single go. I was relunctant to do so, but Git decided to get in the way in the middle night and wrongly rename files randomly in my local branch when rebasing to the master. Happily Nicolas was sitting right next to me and he managed to get Git works a bit more sanely than I was able to. I finally gave up the last bits of my developer civility and pushed almost 10000 lines in a single commit. At the same time, I extracted few work-in-progress bits here and there to commit them separately. I added a very rough Magritte-like model description to EtoileFoundation with ETModelDescription and also ETPaneLayout which will supercede ETPaneSwitcherLayout and the PaneKit when finished. The model description stuff is demoed in the FormExample of EtoileUI were ETFormLayout uses to generate a basic form-like UI. The ETTemplateItemLayout infrastructure behind ETFormLayout is also used by the new ETIconLayout which was added during the hackathon. All that stuff is still quite unpolished… Don't expect too much from it.

On Thursday, we knew Fred Keifer, the GNUstep AppKit maintainer, was going to join us at some point, but weren't sure when. He and his girlfriend were spending some time hiking in Wales, and planned their trip so he could spend a bit of time with us.

So that Fred wouldn't get to university and find us absent, we got up early (I, once again, had gone to sleep around 12:30, but apparently everyone else stayed up for a few hours after).

Nicolas: Thursday we worked a bit at the university, but as our room was to be occupied for a couple of hours during lunch time, we went to eat in a small coffee place at the Swansea marina, walking from the University to the marina along the beach.

At the last hackathon, Quentin had somehow managed not to see the beach so it was nice to stroll along to the marina and drink some nice coffee and hack on a sofa while we enjoyed the free WiFi and good food.

Quentin: Most of the day was spent improving and debugging the new EtoileUI event handling which makes event handling logic reusable in form of tools/instruments (in the spirit Taligent and OpenDPI) and allow to dispatch semantic actions instead of input device specific events, time to time I side tracked to discuss other stuff like CoreObject or LanguageKit.

Nicolas: In the late afternoon we finally had some news from Fred, who was waiting for us at the University, and we went back for a productive few hours, discussing many issues with him and some coding session (notably I ended up writing a live resize version of NSSplitView which turned out to be reasonably fast even on slow computers).

We had a discussion with Fred a while ago over the split between GNUstep and Étoilé and we wanted to talk in person about merging some of our changes (better horizontal menu support and theming, in particular) back upstream into GNUstep. Fred wasn't able to stay very long (and I ran away for a couple of hours for my weekly game of Ultimate Frisbee) but the discussions were productive and hopefully will lead to better collaboration between GNUstep and Étoilé in the future.

After Fred left we went back home and assembled CorePizza 2.0. This year we sent Nicolas to find the beer and he didn't get lost and Eric displayed an unsuspected skill in creating even pizza bases.

Quentin: David deserted around midnight. Eric, Nicolas and I were busy discussing how to improve CodeMonkey UI. What were the most important missing pieces now that debugging was underway with the new stack introspection support written by David yesterday. This was the night of CodeMonkey meets CoreObject. Sadly we gave up in the middle of the night on getting CodeMonkey works with CoreObject. In the end, Eric discovered two days later, the main problem was that the message name that was expected to trigger the persistency was mistyped :-/ … and we somehow missed it.

Friday morning started a bit later. We didn't have to get up for any specific reason. We did make it in to campus in the morning, but only just. The extra sleep was helpful for me, and my cold had almost gone away by Friday night.

Quentin: Friday was the talks day. I initially prepared a long talk on CoreObject, but the planning was 20 mn time slot, so I tried to quickly rearrange my slides with Eric's help to drastically reduce their number. In addition, I put together some slides on EtoileUI when Nicolas was preparing some slides to present GNUstep and Étoilé history.

Friday was the last day I was here, and we had talks planned for the afternoon. After quickly preparing some slides, we gave talks about the history of gnustep and étoilé, language kit, core object and étoilé UI -- all talks that you can now watch online :)

A lot of the morning was spent fixing up slides and then panicking that no one knew where the equipment were were planning to use to record the talks was. In the end we used the built-in iSight on Nicolas' MacBook Pro for video and my external one as a microphone.

Nicolas left shortly after the Friday talks finished, to recover from Nicolas' departure, Eric, David and I then went to the pub just outside the university campus to drink beers. We had an entertaining discussion on widely incoherent subjects: mountains and snow, the history of Canada, France and UK and a bit of Étoilé stuff here and there. LanguageKit and Étoilé on the web were the most discussed topics on the Étoilé side. We debated what was the best way to implement exporting EtoileUI-based applications as web applications and the various approaches we could take to do so… Then there was a long discussion which started with the organization of a ski-oriented hackathon in North America, turned into a wine oriented discussion (even French wine export troubles :-) then finally back to the holy Newton and the sacred Psion. David was motivated enough to take us for a walkthrough of the Psion 3A inside an emulator on his MacBook.

The Psion emulator was originally written for DOS, and runs inside DOSBox very nicely. I originally got my copy off a Computer Shopper (I think) cover CD, but you can probably find a copy online. The Psion did a lot of things very well, such as always saving state when you closed a document, so the user never really noticed the difference between applications and documents being closed and just being in the background. Anyone designing mobile devices should copy as much as they can from the Psion (and the Newton).

Friday was the last official day of the hackathon, but Eric and Quentin stayed until Monday morning, when they both headed off to Paris for Quentin to show Eric a sunnier bit of Europe.

Over the weekend, Eric and Quentin continued to hack on CoreObject and EtoileUI. Quentin tidied up the old Object Manager code which lets you visually inspect the AST of a Smalltalk program, while Eric did some work on the overlay shelf.

On Sunday we walked along the coast to Mumbles and had coffee and cakes in a sea-front cafe. For someone who lives amongst the patisseries of Paris, Quentin has a surprising obsession with Welsh cakes.

Quentin: That day I ate a lot of welsh cakes, scones and chelsea buns, that's almost all I can remember :-D As a last hacking objective, I got my generic Object Manager written in Smalltalk more or less working… the next step is now to improve CoreObject to get the whole thing more usable. The other part was mostly the continuation of the rewrite of various event-related methods in EtoileUI and a more exhaustive test coverage of the event handling.

On Saturday evening some friends of mine were playing in a tango band. Given the choice between coming out and dancing with beautiful women, or staying in and hacking, Eric and Quentin both chose to hack. Two nights in a row. When you next run Étoilé, remember their dedication…

Overall, we got a lot done. Nicolas did a lot on Code Monkey. It is now integrated with CoreObject for data storage and the final, intended, user interface was discussed in a lot of detail. Quentin and Eric did a lot with CoreObject and EtoileUI. Eric also did some hacking on LanguageKit, including some of the C++ bits that previously no one except me understood. I spent a lot of time tiding up bits of code, working on LanguageKit and the Project Manager, and finally got to spend some time improving the Objective-C 2 compatibility library. Not all of my commits are visible in Étoilé svn, because I also spent some time hacking on clang, including implementing support for Objective-C 2 declared properties on the GNU runtime.

Hackathon Statistics:

Total svn commits: 58. Total svn diff size: 20681 lines. GML diagrams drawn: 3. Welshcakes consumed: More than fit in a SmallInt.

LanguageKit SmallInt Improvements

Posted on 2 April 2009 by David Chisnall

Yesterday, I spent some more time hacking on clang. This time, it was implementing -ftrapv, a command-line option which automatically inserts overflow checking on integer arithmetic operations. This was made very easy to do because LLVM 2.5 now supports this in the IR. There are now intrinsic functions like llvm.sadd.with.overflow which return a pair containing the result of the arithmetic operation and a flag indicating whether it overflowed. On x86 (and some other back ends - I don't know exactly which ones now, but eventually it should be all of them), this just provides access to a condition flag in a register.

The implementation of -ftrapv in GCC just aborts in case of overflow, but in clang I had it call a function. This function takes the two operands some some information about the operation as arguments, and returns a value which is used in place of the overflowed result. I contributed a simple default version of this to clang, which simply printed a helpful error message and aborted.

For LanguageKit, I did something a bit more interesting. The implementation of arithmetic operations in Smalltalk originally was quite naive. A SmallInt in LanguageKit is an integer squeezed into a pointer. On a 32-bit platform, it is a 31-bit integer, while on 64-bit platforms it is a 63-bit integer. In both cases, the remaining (low) bit is set to 1, to distinguish it from an object pointer (object pointers are always word-aligned and so their least-significant bit is always 0). For each operation, I was doing something like this:

a >>= 1; // Turn the SmallInt into an int
b >>= 1; // Turn the SmallInt into an int
c = a + b; // Do the actual addition
if (c > SMALL_INT_MAX) { handle overflow } // Do some simple overflow checking
c = (c << 1) | 1; // Shift the int back to a SmallInt and set the low bit
return c; // Return the result

One of the problems with this is that every operation then added at least one pair of shifts which the LLVM optimisers could not remove. Detecting that a value right shifted and then left shifted by the same amount is the same as the original is nontrivial, and requires knowledge of the values being stored.

The new code is much simpler. This is the real code from the addition implementation, a function which takes two pointers obj and other as arguments:

OTHER_OBJECT(plus);
intptr_t val = ((intptr_t)other) & ~ 1;
return (void*)((intptr_t)obj + val);

The first line is a macro which tests if other is an object (typically a BigInt, but maybe an NSNumber of similar). If it is, then it promotes obj to a BigInt and tries the addition again. The second line clears the low bit on the other object pointer. The final line just adds the two SmallInts together. Because the low bit on one is always 1, and the low bit in the other is always 0, this addition does the same thing as the two right shifts, addition, left shift, and or in the older version. Even better, because we are now adding word-sized numbers, the hardware overflow checking can now be used correctly. If the addition in the last line fails, the handler will be invoked. This is implemented in the LanguageKit runtime framework and automatically inserts a BigInt pointer in place of the SmallInt value.

I've made similar improvements to the multiplication and subtraction routines. Previously, I was using a very naive approach to checking for multiplication overflow. If either of the operands was bigger than the maximum value you could store in a half-word then I was promoting the operation. Now, multiplications are only promoted if the result does not fit in a 31-bit or 63-bit signed value, just like addition and multiplication.

So, what does this mean performance-wise? I wrote a simple benchmark (in Etoile/Languages/Benchmark) to test this a while ago. This implements the most naive way of calculating the Fibonacci sequence (recursively call fib(n-1) and fib(n-2)) in both Objective-C and Smalltalk. The code measures the amount of CPU time taken to calculate fib(30) 100 times in each version.

When I first wrote this benchmark, some time before Christmas, it showed that the Smalltalk implementation took around 30 times as long as the Objective-C version. This wasn't too bad; I never intended Smalltalk to be used for heavy number crunching. The entire point of targeting the Objective-C ABI was that you can use Objective-C, C, or even inline assembly if you really care about speed. Looking at the generated bitcode, I saw that there were a lot of redundant shifts, and so a great deal of potential for optimisation. Running it again, with the new SmallInt code, I get this:

ObjC fibonacci execution took 23.773438 seconds.
Smalltalk fibonacci execution took 35.953125 seconds.  
Ratio: 1.512323

Smalltalk now takes 1.5 times as long as Objective-C. It's worth mentioning in this case that using a slightly better algorithm in Smalltalk is around a factor of 100 faster; good algorithms with bad compilers almost always beat good compilers with bad algorithms.

I didn't actually believe these results when I first looked at them, and spent quite a while going through the code looking to check that they are correct. This test is pretty much the worst possible case for Smalltalk, so in general use I'd expect Smalltalk to be even closer to Objective-C in terms of performance.

By the way, this test also now shows that the LowerIfTrue transform is actually useful. This is an AST transform in one of the LanguageKit plugins that turns -ifTrue: and related message sends in the AST into conditionals. When I wrote it, I tested this benchmark with and without it and there was no measurable difference; the overhead from the arithmetic was dwarfing the overhead from the extra message sends. This is not the case now. Disabling this optimisation gives this result:

ObjC fibonacci execution took 23.773438 seconds.
Smalltalk fibonacci execution took 60.390625 seconds.
Ratio: 2.540256

The Smalltalk implementation sends an -ifTrue: message when testing whether the argument is less than 2 (in which case it returns 1), while the Objective-C version just uses an if statement. With the transform, the Smalltalk version does not need the extra message send to the block and so compiles to almost the same machine code. I am not a huge fan of this optimisation in theory (since it makes the language marginally less expressive), but in practice it is unlikely to affect any existing code and is implemented by most other Smalltalk compilers.

GNUstep and Clang

Posted on 31 March 2009 by David Chisnall

I'm going to break from tradition slightly now and talk a bit about something that is not directly part of Étoilé. For a while, I've been intermittently hacking on clang, the new C/C++/Objective-C front end for the LLVM compiler infrastructure. If you've done any development for the iPhone or for recent versions of OS X, you will probably have used llvm-gcc, which uses the same back end but a front end based on GCC.

Since GCC switched to GPLv3, Apple has been slowly migrating away from it. The GPL wasn't great for Apple anyway, since they wanted to be able to use the same front end code in XCode for syntax highlighting, autocompletion, and error reporting as they use in the compiler, and the GPL didn't allow them to do this without releasing XCode under a GPL-compatible license. A couple of years ago, they started the clang project to replace the GPL'd front end and create a reusable set of libraries that handled parsing, analysis, and LLVM bitcode generation for C-family languages.

Why is this relevant to Étoilé? For some very simple reasons:

• Apple are no longer contributing much to the main GCC tree.
• No one outside Apple contributes much to Objective-C in GCC.
• Étoilé needs an Objective-C compiler.

The original Objective-C code in GCC was contributed by NeXT (after being attacked by the FSF's lawyers) and only supported the (closed) NeXT Objective-C runtime library. Over time, the NeXT and GNU runtimes have diverged a lot. NeXT, and later Apple, maintained their own branch of GCC and never merged support for the GNU runtime. The GNU branch maintained support for both, via an unreadable mess of #ifdefs in a single 10,000-line file.

Clang, starting from scratch, has a clean abstraction layer between the runtime-specific and runtime-agnostic bits of code. Each runtime library needs to implement a subclass of the CGObjCRuntime C++ class, which implements hooks for generating runtime library data structures and calls.

I started working on the GNU runtime support a little while ago, and it is now in a usable state. For a little while, the things that have stopped GNUstep compiling with clang have been missing support for GCC extensions to C. Now, these have all been fixed. GNUstep-base now compiles with clang, but doesn't link. The remaining issue is that LLVM does not support the __builtin_apply() family of GCC extensions. These are not actually used by GNUstep in most cases - their functionality is replaced by either libffi or libffcall - but they are still called by a few unused methods. A small restructuring of the GNUstep-base code will allow it to be compiled with the current svn version of clang. I've also tried compiling a few Étoilé frameworks with clang, and so far they've all successfully built (although I haven't tested if they work yet...).

That's not to say that clang is completely ready to replace GCC as the compiler for GNUstep and Étoilé. In summary:

• All of the NeXT-era Objective-C stuff is supported.
• Fast enumeration works (and is implemented for some of the GNUstep collection classes).
• @try/@catch and so on do not work yet (although the old-style exception macros do).
• A very small subset of declared properties may work (untested).
• Garbage collection does not yet work.
• Lots more testing needs to be done.

If you want to get involved, take a look at the CGObjCGNU.cpp file in lib/CodeGen in the clang repository. It contains a number of empty methods, which need implementing to bring the GNU runtime implementation up to feature-parity with the Apple version. Alternatively, try compiling your Objective-C code with the latest clang and report any errors to me and to the clang team.

Exception handling is probably the next thing to implement. The code in CGObjCMac.cpp contains a lot of things that probably should be factored out into runtime-agnostic code, so if you're interested in working on this, take a look in that file for inspiration.

Étoilé 0.4.1 Packages

Posted on 24 March 2009 by David Chisnall

We released 0.4.1 a few days ago. If you want to try it out, but don't want to have to build the system yourself, you now have several alternatives.

Arch Linux now has a package for 0.4.1.

FreeBSD has had a metaport of Étoilé since 0.1 back in 2006. It has now been updated to 0.4.1. Note that this metaport also includes some older things that are not part of 0.4, typically because they are not stable enough to be included in a release.

Gentoo has Étoilé 0.4.1 in the GNUstep overlay (not official link, because I can't work out how you're supposed to navigate the Gentoo packages site).

We're interested in hearing from anyone running Étoilé on other platforms. We've had reports of it working successfully on Debian and Ubuntu, although packages are not available for these platforms (Ubuntu has a Brainstorm suggestion and bug relating to Étoilé packages). I don't know if anyone has run recent versions of Étoilé on OpenBSD or Solaris, but I'd be interested in fixing any problems encountered on these platforms.

If you're interested in adding packages for Étoilé to your favourite operating system then send questions to our packagers mailing list or ask in SILC.

Étoilé 0.4.1 Released

Posted on 17 March 2009 by David Chisnall

I've just branched the 0.4.1 release and uploaded the tarballs to usual location. This is an incremental release, providing bug fixes and feature enhancements to 0.4.0.

The main focus for this release is an improved LanguageKit. This is now better-integrated into the whole Étoilé system and can be used to load bundles providing extra features to existing applications without the need for source-code availability.

As usual, we only recommend releases for people creating packages. If you are compiling from source, please check out the trunk or stable branches in subversion.

Étoilé interviewed on FLOSS Weekly Podcast

Posted on 16 February 2009 by Jesse Ross

David and Quentin were on the latest episode of the FLOSS Weekly podcast with Randal Schwartz and Leo Laporte. Listen to the interview here.

LanguageKit Updates

Posted on 31 January 2009 by David Chisnall

Lots of fun things have happened with LanguageKit recently and since I haven't written anything on this blog for a while I thought I'd write about some of them.

Fixing Bugs

A bit before Christmas, I found a subtle bug in clang that was causing the class names not to be NULL-terminated in the support file. This worked fine in some cases and crashed on others, dependent entirely on the location of the generated code in memory. Eric was very confused by the fact that code was working when he added a constant string and failing when he didn't. When this was fixed, a lot of heisenbugs went away.

Returning Structures

I implemented some very basic understanding of calling conventions. It turns out that LLVM doesn't know anything about the host ABI's structure returning mechanism, which was causing some very strange results. This was especially confusing to me since it worked on FreeBSD, but not Linux and everyone I asked told me that they used the same calling convention on x86. It turns out this isn't quite true. FreeBSD uses an in-register mechanism for returning small structures, while Linux uses the old PCC calling convention, passing them on the stack. Adding basic ABI-awareness meant that you can now call functions that return NSRange and similar structures and pass them as arguments. When you get an NSRange returned from a method, it is boxed in an NSValue. When you pass it to a method expecting a range, -rangeValue is called. This allows you to substitute any other object that responds to -rangeValue.

Stack Unwinding

The biggest missing feature before Christmas was support for non-local returns in Smalltalk. It is quite common to want to write code like this:

(some guard condition) ifTrue: [^nil].

The problem here is that the ^nil is inside a block. In Pragmatic Smalltalk, this will be in a nested stack frame. The return, however, is meant to return from the method, not from the block. This is a really ugly bit of Smalltalk, but one that apparently people use.

This month, I finally got around to implementing it. This is done via the zero-cost exception mechanism. The return statement in the block is transformed in the code generator to a self nonLocalReturn:nil. This method is written in Objective-C and looks up the scope chain until it gets to the method. It then throws a new kind of exception which unwinds the stack until it gets to the method.

This means that it will work with intermediate stack frames written in any language. If you have a @finally block and are compiling with native exceptions in Objective-C then this should be called on the way up the stack. Similarly, any cleanup that intervening Smalltalk stack frames need to do will also be done.

This is a very expensive operation. For each stack frame between the block and the method there is a call to the unwinding library, at least two calls to the LanguageKit personality function, and a call to the frame's cleanup code. Fortunately, non-local returns are mainly used for guard clauses so it's not likely to be invoked in the most common code paths. In future versions there will be more aggressive block inlining and some of these returns will be transformed into local returns.

Loading Bundles

One of the things I'd been planning to do from the start is allow bundles of Smalltalk to be run. I've now committed support for loading bundles which contain a LKInfo.plist resource. This specifies a list of Smalltalk (or other language...) files that will be compiled and a list of frameworks that will be loaded.

When used in conjunction with edlc there is some dynamic patching of the NSBundle class so that the loaded bundle will be returned in response to a +mainBundle message. This allows NSApp to find it and load resources easily. As a demonstration of this, Eric has committed the first pure-Smalltalk application to svn recently. This uses a simple shell script as the application binary:

#!/bin/sh
edlc -b `dirname $0`

You can launch this application with openapp or it will run as any Objective-C application would. The class specified by the PrincipalClass field in the plist is instantiated and sent a -run message. This does the same thing as the NSApplicationMain() function that most Objective-C apps use:

NSObject subclass: PhotoManager
[
    run [
        | nsapp |
        nsapp := ETApplication sharedApplication.
        NSBundle loadNibNamed: 'MainMenu' owner: nsapp.
        nsapp toggleDevelopmentMenu: nil.
        nsapp setDelegate: self.
        nsapp run.
    ]
]

Since LanguageKit supports static compiling as well as JIT, I eventually plan on extending this to emit a shared library for each bundle and load it rather than recompiling if the code hasn't changed.

Application Patching

One of the nice features of GNUstep is the idea of an AppKit user bundle. These are specified in defaults and will be loaded when AppKit initializes. Étoilé provides three of these. One relates to theming, one relates to the menu bar, and the final one is for general behaviour.

Part of the refactoring I did recently to LanguageKit moved the code generation component, which is huge, to a separate framework which is now dynamically loaded on demand. This means you can link LanguageKit against other applications without much overhead, and this is exactly what we're now doing.

If LanguageKit is installed then the EtoileBehavior bundle will load it into any GNUstep application. It will then look in $(GNUSTEP_USER_ROOT)/Library/LKPlugins for a directory matching the name of the application. If one exists, then it will load every bundle with a .lkplugin extension. This allows you to add arbitrary code to any application. Using categories, you can even replace existing methods.

Why is this important? Take a look at this from the Free Software Foundation:

The freedom to improve the program, and release your improvements (and modified versions in general) to the public, so that the whole community benefits (freedom 3). Access to the source code is a precondition for this.

In an Étoilé environment, access to the source code is now not a precondition of this freedom. You can improve a program, in a high-level language, without access to the source code.

AST Transforms

There is now a simple API for adding transforms on the LanguageKit abstract syntax tree. You just need to adopt a simple protocol and you will get the opportunity to replace each AST node in the tree. You can also use this mechanism to collect information about the tree.

Plugins using this mechanism will be collected in Languages/LKPlugins in svn. Currently there is just one simple demo, which replaces every comment with a log message. This allows you to record every time a line containing a comment is passed.

Code Monkeys

The final, but possibly most exciting, development recently is CodeMonkey, a new IDE being written by Nicolas. This uses LanguageKit for compiling and for for syntax highlighting using code based on Günther's pretty-printer.

It's still at quite an early stage of development - not even a 0.1 release yet and probably won't make it into 0.4.1 - but it's very promising. This does a few neat things with AST transforms. If you add a new method to a class then it will emit a new category on the class, rather than a new copy of the class (which would confuse the runtime). More interestingly, if you add an instance variable to a loaded class it will turn every reference to this into KVC calls. When you restart the application, the transform won't run and they will be direct ivar accesses.

The medium term plan for CodeMonkey is to turn it into a bundle. EtoileBehavior will then add a new menu item which will load the bundle and run the UI. This will give you Squeak-like introspection and modification of running applications.

Prototypes and LangaugeKit

Posted on 20 December 2008 by David Chisnall

Over the last few days, I've been playing around with rewriting the prototypes code not to require a modified runtime. The old code used a new (backwards-compatible) version of the GNU Objective-C runtime that added a 'third chance' dispatch mechanism, allowing objects to implement their own method lookup function.

The new version works by doing a hidden class transform. One of the unused flags in the class structure has been set to define whether a class is a hidden class or a real one. Hidden classes are just like normal classes, but they are generated at runtime and inserted at the end of the inheritance chain. When you send an object a -becomePrototype message, it inserts a new class/metaclass pair as its class, with the old class becoming its new superclass.

Since I'm inserting new classes, this gave the opportunity to add some other information, most usefully an NSMutableDictionary for storing extra values. In Smalltalk, there is a mechanism a lot like this. Instance variables in the Objective-C sense are called 'indexed instance variables,' but others stored in a linked list are also possible (but slightly slower to access).

The hidden class implements a few methods of its own. The most important of these are the two related to Key-Value Coding, meaning that you can use the standard KVC methods to access this hidden dictionary. In Objective-C, you simply do something like this:

id setValueIMP(id self, SEL _cmd, id aValue)
{
    [self setValue:aValue forKey:@"TestKey"];
    return self;
}
id getValueIMP(id self, SEL _cmd)
{
    return [self valueForKey:@"TestKey"];
}

int main(void)
{
    id pool = [NSAutoreleasePool new];
    id proto = [[MyObject new] autorelease];
    [proto setMethod:(IMP)setValueIMP forSelector:@selector(setTestValue:)];
    [proto setMethod:(IMP)getValueIMP forSelector:@selector(testValue)];
    [proto setTestValue:@"A string"];
    NSLog(@"Test value: %@", [proto testValue]);
    return 0;
}

This will output something like this, when run:

2008-12-20 16:16:29.054 test[6813] Test value: A string

Of course, this is not possible in Smalltalk, since Smalltalk doesn't have the ability to write functions outside of classes. It does, however, have the ability to create blocks.

This week, I tweaked the code slightly so that if you insert a block into this dictionary as an object, it implicitly registers a trampoline method for the key as a selector. When you call the method, the trampoline uses the _cmd argument (the hidden argument in all Objective-C methods giving the selector used to look up the current method) to get the block from the dictionary and then sends it a -value: message, or a value:value: message (and so on) depending on the number of arguments that the block expects.

This lets you write code in Smalltalk like this:

NSObject subclass: SmalltalkTool
[
    run 
    [
    | a |
        a := NSObject new.
        a becomePrototype.
        a setValue:[ :object :aValue | ETTranscript show:aValue; cr ] 
        forKey:'logValue:'.
        a logValue:'A string'.
    ]
]

When you run this with edlc, you get:

A string

Obviously this is slower than a direct method call, since you first enter the trampoline, then look up the block in a dictionary, and then finally call the block, but it should be a lot faster than using forwarding. On GNUstep, sending a message via forwardInvocation: takes around 300 times as long as a direct message send. A message send to a block costs around 3 times as much as a message send to an Objective-C object. A message send to a prototype costs around ten times as much. Note that a lot of the extra overhead from the block (which has a similar overhead to calling a Smalltalk method) is due to the need to clean up the object context at the end of the call.

Étoilé 0.4 Hits FreeBSD Ports

Posted on 8 December 2008 by David Chisnall

Jesse just pointed out to me that the FreeBSD ports for Étoilé were updated to 0.4.0 last week. FreeBSD was the first operating system to package Étoilé, thanks to some excellent work by Dirk Meyer, who also maintains a large number of GNUstep ports for FreeBSD.

Dirk has added all of the 0.4 release to the ports tree, and you will also find a number of things that were part of the 0.2 release but weren't considered polished enough for 0.4.0 (but should be reappearing over the 0.4.x series after some more testing). There is a metaport called etoile which will install all of these. Simply do:

# cd /usr/ports/x11/etoile
# make install clean

Or, if you have portupgrade installed:

# portinstall etoile

And you will have the full set of Étoilé packages installed. If you want to package Étoilé for your own favourite operating system, then please subscribe to the etoile-packaging list and post any questions you might have there.

Massive LanguageKit Improvements

Posted on 6 December 2008 by David Chisnall

This weekend, I rewrote large chunks of LanguageKit, simplifying a few things and upgrading a few other things. If you are using LanguageKit / Smalltalk, make sure up update both LanguageKit and SmalltalkKit - they contain interrelated changes.

The changes involved the way in which local variables are allocated on the stack. Every stack frame now contains a StackContext object. All of the local variables are now stored as instance variables in this object. The BlockClosure object is now stateless - all of the state information is stored in the context hierarchy.

This simplified the symbol table a lot, and meant that a lot of ugly code related to closures has gone away. The most obvious side effect of this is that accesses to variables from blocks now works correctly. In addition to contexts, blocs are also now allocated on the stack. This should speed things up a bit and have the nice side-effect that it gets rid of a potential memory leak that used to exist.

The other side effect is that it greatly simplified the task of implementing upward funargs. If you want to keep a block around for longer than the duration of the parent scope, then you just need to send it a -retain message (assigning it to an instance variable in Smalltalk does this for you). When you do this, you get a copy of the block, rather than original (you don't want to store pointers to things that are on the stack - bad things happen if you do).

The following program illustrates this:

NSObject subclass: SmalltalkTool [
    | block |
    run [
        self setBlock.
        self callBlock.
    ]
    setBlock [
        | a |
        a := 'Local variable'.
        block := [ a log.  ].
    ]
    callBlock [
        'Testing retained block:' log.
        block value.
    ]
]

When you run this, you get the following output:

$ edlc -f retainblock.st
2008-12-06 17:27:36.530 edlc[40803] Testing retained block:
2008-12-06 17:27:36.566 edlc[40803] Local variable

As you can see, the block is stored in an instance variable and is still able to access the variable a from the setBlock method, even after that method has exited.

The block you acquire still contains a pointer to the context on the stack, however. This means that if you use it before the enclosing function, method, or block, returns, you will still get any changes to other variables propagated back up the stack.

When a copy of a block is made, it calls -retainWithPointer: on its scope. This method stores the address of the pointer which references the block and swizzles the block's isa pointer so that it becomes a RetainedContext object. At the end of any LanguageKit-generated block or method, a simple test is performed on the context to see if it is a RetainedContext. If it is, it is sent a -promote message. This then copies it to the heap and updates all of the pointers referencing it.

As you can imagine, this is quite an expensive procedure. The good news is that it's rarely needed, and only incurs one test at the end of each method or block when it isn't. Since blocks are now allocated on the stack, I expect that the new version should be faster than the old one (although I haven't actually benchmarked it).

The new version is in trunk now, so please test it. People watching svn will have noticed that the BlockContext object currently has an unused instance variable: char **symbolTable. This will be set to an array of the variable names for every variable in the context, which will allow contexts to be introspected.

Étoilé 0.4.0 Release Announcement

Posted on 16 November 2008 by David Chisnall

Étoilé intends to be an innovative, GNUstep-based, user environment built from the ground up on highly modular and light components. It is created with project and document orientation in mind, in order to allow users to create their own workflow by reshaping or recombining provided Services (aka Applications) and Components. Flexibility and modularity on both User Interface and code level should allow us to scale from handheld to desktop environments.

0.4 is a developer-targeted release on its way towards this goal. As a developer-focussed release, this predominantly consists of frameworks. A few demonstration applications are also included. More will be added during the 0.4.x release series, leading to a user-focussed 0.5 release next year.

Highlights

CoreObject is a framework for describing and organizing model objects. It supports automatic persistence and versioning by recording messages sent to objects. It offers a flexible versioning scheme where both individual objects and their entire object graph can be versioned separately. The built-in object model is a generalization of the property model used by the AddressBook framework. Foreign model objects can be also integrated by wrapping them with a special proxy. CoreObject uses the EtoileSerialize framework which, in many cases, allows objects and messages to be automatically serialized with no extra code being written.

LanguageKit is a compiler kit built on top of LLVM for creating dynamic language implementations using an Objective-C runtime for the object model. This is used by SmalltalkKit, implementing Étoilé's Pragmatic Smalltalk, a Smalltalk JIT compiler which generates code binary-compatible with Objective-C, allowing classes to be written in a mixture of Smalltalk and Objective-C.

EtoileFoundation is the core framework for all Étoilé projects, providing numerous convenience methods on top of the OpenStep foundation and significantly better support for reflection. This includes EtoileThread which allows objects to transparently be run in a separate thread. It also includes a number of extensions to the Objective-C object model, allowing traits and mixins. This framework is used by most of the rest of Étoilé and provides a number of core functions, such as UUID and XML handling.

EtoileUI is also available as an early preview release and should not be considered stable. EtoileUI is a high-level, object-oriented, user interface toolkit that provides a uniform tree representation for graphical objects on top of the AppKit. All User Interface concerns such as layouts, event handlers, styles, model objects etc. will be implemented as pluggable aspects. It also shares the same interfaces as other CoreObject systems. The combination of these three key features makes possible to inspect and reshape both User Interface and model objects at runtime through direct manipulation. It comes with a library of layouts where each one encapsulates a custom and pluggable visual presentation.

Other frameworks, such as LuceneKit, providing full-text indexing and searching, and OgreKit, a powerful regular expression framework are also included. UnitKit is a simple and flexible unit testing framework used by much of Étoilé. A new addition is MediaKit, a framework used to provide support for sound playback and recording and, in future, video. SystemConfig has received a number of improvements since our last release, including support for modifying basic X11 keyboard settings and monitoring the battery level.

Several applications are part of this release, such as Mélodie, a music jukebox using CoreObject for the music library and MediaKit for playback. Étoilé applications which use ScriptKit are scriptable from outside using Objective-C or Smalltalk. This is used by the hot corners and gesture recognition tool to run arbitrary commands in response to corner activations or mouse gestures, and by ScriptServices which allows arbitrary shell or Smalltalk scripts to be invoked on the current selection from any GNUstep or Étoilé application.

Screenshots

Étoilé in the Dictionary	About and Vindaloo PDF reader

Availability

Étoilé 0.4.0 is currently available in code source form only and may be downloaded at http://download.gna.org/etoile/etoile-0.4.0.tar.gz It may also be obtained from Subversion with the following command:

svn co svn://svn.gna.org/svn/etoile/tags/Etoile-0.4.0

If you wish to use the latest stable release, then you can download http://download.gna.org/etoile/etoile-0.4.0-svn.tar.gz before running svn up to seed your source tree.

More Information

Visit our website: http://www.etoileos.com/ and blog: http://etoileos.com/news/ Or subscribe to our mailing lists: https://gna.org/mail/?group=etoile Or join our SILC channel: silc://silc.etoileos.com/Etoile

Static Compiling Smalltalk

Posted on 10 November 2008 by David Chisnall

One of the things I wanted to do with Smalltalk was allow static compilation. This is possible with LLVM as the back end. The compiler creates LLVM IR, a low-level intermediate representation form, which is then used to perform optimisations and can be compiled or interpreted. I was using this for the JIT - the IR was created when the code was loaded but turned in to native code on-demand, when each method was used.

Today I committed a few changes to LanguageKit to allow the bitcode to be written to a file instead of loaded. This was slightly more complicated than you might imagine. I use a trick with the JIT where each Smalltalk module uses the set of functions defining small integer messages as a template. This allows them to be inlined nicely without having to worry about cross-module optimisations. For static compilation, this is not desirable, so the biggest change was allowing it to reference these functions externally or internally depending on how the code generator was being used.

Once this was done, I added a new -c option to edlc. If you now do:

$ edlc -c -f test.st

You will get a file test.bc as output. This contains the LLVM bitcode for the Smalltalk file. The next step is to link together all of the .bc files, including the MsgSendSmallInt.bc file which contains definitions of small integer messages:

$llvm-link $(GNUSTEP_LOCAL_ROOT)/Library/Frameworks/LanguageKit.framework/Versions/0/Resources/MsgSendSmallInt.bc test.bc -o smalltalk.bc

This outputs a single file, smalltalk.bc, containing all of the bitcode from the various modules. If you compiled more than one Smalltalk file then list all of the .bc files here. This is completely unoptimised, so let's run some optimisations on it:

$ opt -O3 smalltalk.bc -o smalltalk.optimised.bc

This runs the same set of optimisations that llvm-gcc runs at -O3. I haven't actually done any sensible tests to see if this is sensible, but hopefully it is (if anyone can come up with a good list of optimisations before I get around to doing some sensible testing, please let me know).

Now we have an optimised bitcode file, we want to turn this into object code. This is a two-step process:

$ llc smalltalk.optimised.bc
$ gcc -c smalltalk.optimised.s

The first step produces assembly code, and the second step assembles it (you can use as for the second step, but I was lazy and just threw it at the GCC compiler driver). You now have a file called smallltalk.optimised.o, an object code file that you can link in to your executable just as you would an object code file compiled from Objective-C.

This sounds a bit complicated, and it is. It's actually more steps than the first C compiler I ever used (where preprocess, compile, assemble, and link were all separate steps) required. Fortunately, Nicola Pero is working on adding support for it to GNUstep Make, so soon it should be just a matter of putting SMALLTALK_FILES=... in your GNUmakefile.

The bad news is that this is too big a change to be properly reviewed in time for 0.4.0, so unless you are running trunk you will have to wait for a bit to see it. 0.4.1 should be out around the new year, so you don't have too long to wait...

Packaging Étoilé

Posted on 27 October 2008 by David Chisnall

Nicolas just sent me a link to this Ubuntu brainstorm idea. Someone is proposing full Étoilé packages for inclusion in Ubuntu.

Currently, Étoilé is a bit of a moving target. It's been a long time since our last release. FreeBSD has ports for this release, but so much has changed in subversion since then that these are no longer a good introduction to Étoilé.

Hopefully this will change next month. We are planning on releasing Étoilé 0.4.0 on the 31st of October. If you want to get an idea of what it will contain then take a look at the current stable branch - this will be tagged 0.4.0 in under a week.

After this, we will be moving to a time-based point release schedule, with Étoilé 0.4.1 being released at the end of the year, 0.4.2 at the end of February, and so on.

Hopefully this will make life easier for packagers. We aim to only require released versions of dependencies for release versions of Étoilé (0.4.0 will require LLVM 2.4, for example, which is due for release on October 30).

If you are interested in providing packages for your platform, then please get in touch and let us know what we can do to help.

The road to CoreObject Part 3: Mixing Temporal Object Store and Name Service

Posted on 19 October 2008 by Quentin Mathé

CoreObject is a central piece of Étoilé, often discussed but rarely seen :-) The good news is that it's currently shaping up pretty well. But before entering in the details and illustrating CoreObject persistency with an example in a next post, I'd like to give a brief overview of it.

The overall architecture has evolved substantially over the past two years. The implementation started with the writing of EtoileSerialize by David, and CollectionKit then OrganizeKit by Yen-Ju. These last two frameworks were built to provide a semi-structured object model inspired by AddressBook framework, that can be used to write applications managing collection of objects (music, photo, contacts etc.). This summer, Eric wrote a music manager named Mélodie based on this reusable object model. As such, Mélodie is the first Étoilé application that truly uses CoreObject.

Until recently, CoreObject mostly existed as a fork of OrganizeKit in Étoilé repository. The persistency model was to store the whole core object graph into a single property list, or multiple property lists but without the possibility to reference core objects across these property lists. This was a very important limitation that prevented concurrency control and versioning of objects through EtoileSerialize. Moreover each time a process wanted to access a core object, the entire core object graph had to be deserialized. Over the past two months, I have revisited CoreObject, in order it fully leverages EtoileSerialize for persistency, supports the loading of the core objects in memory on demand, interacts with a metadata server to track stored objects, and provides a better control over the history of core objects.

The updated version of the semi-structured object model also brings a very transparent approach to persistency, you don't need to call EtoileSerialize explicitly or even use a proxy to wrap your objects.

Now let's look at the various building blocks of the framework. The basic idea behind CoreObject is to provide a reusable model for organizing objects and handling their persistency. The low-level persistency logic is implemented by EtoileSerialize, CoreObject extends it with:

• a protocol to organize core objects into groups (COObject and COGroup protocols)
• a main backend that provides a semi-structured object model (COObject and COGroup classes)
• additional backends to attach external object graphs (for example mounting a filesystem or exporting an application UI into the core object graph)
• COProxy for integrating persistent model objects not derived from the COObject class
• a metadata server to track stored objects and index both metadatas and content of core objects (COMetadataServer class)
• a per process object factory and cache that is used to handle the faulting and uniquing of core objects (COObjectServer class)

So CoreObject mostly adds a name service on top of a EtoileSerialize and persists the name service structure and the objects bound to it in the same uniform representation. This representation is the core object graph, where each object and each group is stored as a persistent root by EtoileSerialize. Each persistent root is identified by an UUID/URL pair. Persistent roots are currently stored as object bundles on the filesystem. An object bundle is a directory the contains the history of the object in term of snapshots and deltas. Deltas are serialized invocations that represent logical changes. EtoileSerialize defines a protocol for the storage model, so new ways to store the objects could be defined. For example, changing the layout of object bundles, storing all the objects in a single flat file, over the network or other kind of data stores such as ZFS DMU (the low-level ZFS transactional store on which the filesystem is built).

The UUID/URL pairs are stored in a metadata server, which defines all the objects that belong to a core object graph. In future, the core object graph should thus be able to span multiple computers or data stores backed by a single metadata server. The metadata server is currently based on a PostgreSQL database.

For this first approach, multiple users cannot share a single core object graph and the access rights are simply defined by the permissions set on the object bundles at the filesystem level.

Out of the box, EtoileSerialize provides the basic infrastructure for per object history. This allows to support undo/redo per object. However objects such as photos, music, contacts are usually organized into libraries and it is expected undo/redo will operate on the last modification for the currently opened library, when you use a photo manager or a music manager. If that wasn't the case, undo/redo would only work if one or several objects are selected as targets for undo. This also means the user would have to remember the last modified object if he changed the selection after editing this object.

To solve this problem, CoreObject introduces the notion of object contexts. An object context is a pool where you insert related core objects. The object context records an history that is the interleaved histories of all the objects that belong to it. By this mean, it becomes possible to navigate and restore the history per object and per context.

Most of the elements of the architecture outlined at the beginning have already been implemented, if we put aside the indexing service. Various key pieces remain to be written though: concurrency control, update feed to push object changes to client applications, in-store deletion model and history cleaning.

In a more broad perspective, integration with the branching support of EtoileSerialize, exporting core objects to other formats and collaborative editing, will also have to be fully worked out. Finally the versioning of structured documents will require additional support to be truly convenient and integrate perfectly with EtoileUI.

So, you want to invent a language?

Posted on 12 October 2008 by David Chisnall

I posted a little while ago about the Smalltalk compiler in Étoilé svn. Since then, Truls Becken has rewritten my parser (which was quite bad, and is now quite good) and tidied up the code a little. I've also refactored it into two frameworks, LangaugeKit and SmalltalkKit. LanguageKit contains all of the abstract syntax tree and code generation stuff, while SmalltalkKit contains all of the Smalltalk-specific parts.

The total line count for the Smalltalk-specific part is a shade over 500 lines of code. This means that writing a new front-end for something Smalltalk-like is very easy (I plan on adding some things to LanguageKit to make slightly less Smalltalk-like languages similarly easy).

If you want to play, then the first thing you need is a subclass of LKCompiler, which implements two methods: +fileExtension and +parser. The first returns the extension used by scripts in your language (@"st" for Smalltalk), while the second returns the Class implementing your parser.

Then you need to implement the parser. This just needs to implement one method, parseString: which takes a string as an argument and returns an AST. For Smalltalk, I have a hand-written tokeniser and use LEMON (from the SQLite project) for the parser. The tokeniser simply turns the string into a stream of tokens and then passes them one at a time to the parser (it might be simpler if I wrote it using something like Lex, but since it's only 200 lines of code now I can't really be bothered). The parser is generated from a BNF-like description of the grammar, with instructions in Objective-C on how to generate the AST from this.

Now that Truls has rewritten it, the Smalltalk grammar is a fairly good example of a LEMON grammar. If you want to write a new language, a good first step is tweaking Smalltalk a bit. If you find that you want a semantic construct that isn't supported by the AST, drop in to SILC and talk to me - adding static flow control (if statements and while loops) is high on my list of priorities, as is support for primitive (non-object) types that aren't auto-boxed.

Scripting and Gestures

Posted on 13 August 2008 by David Chisnall

Two of the things that have been on my TODO list for about two (maybe three) years are cross-app scripting and mouse gestures. StepTalk had some preliminary support for cross-app scripting, but I don't think it made it into a release. I never really liked its approach, since it seemed horrible over-engineered (for reference, the Smalltalk interpreter in StepTalk is about twice as much code as the Pragmatic Smalltalk compiler and support library).

Yesterday, I committed the first version of ScriptKit. This is a very lightweight cross-app scripting framework built on top of Distributed Objects. It simply exports a dictionary containing a set of named objects for scripting. By default, NSApp (the application object) is exported. If you don't want to give unrestricted access to remote scripts you can export your own object with the 'Application' key and filter out some messages. You can also export other objects with their own names. In future we will define a set of standard-but-optional ones that Étoilé services should export (e.g. the current document, some CoreObject related things and so on).

For the paranoid, I plan on adding a 'Paranoid Mode' which uses a pre-shared key to prevent unauthorised scripts from controlling the app.

The nice side-effect of using DO as the core is that it is also trivial to send scripting events from Objective-C. Anyone who has tried doing this with Cocoa has probably given up and just generated a string containing AppleScript code and passed this to the scripting engine. Since we are using a Smalltalk which is toll-free bridged with Objective-C, it makes sense to just expose scripting objects as Objective-C / Smalltalk objects (well, object proxies) and use them directly, without a confusing abstraction layer.

The other thing I added yesterday was a gesture recognition engine. Today I remembered that 'x is a-cross' and fixed it so that it actually works. This is embedded in Corner.app, which currently handles hot corners for Étoilé (allowing scripts to be run when the mouse enters and leaves a screen corner). If you hold down control and shift, it enters a gesture quasi-mode. It then tracks mouse movements. Each movement is treated as an approximation of a movement in one of 8 directions, numbered 1 to 8 clockwise from the top (i.e. 1 is up, 5 is down, and so on). Complete gestures are therefore turned into strings ('gesture words'), so an 'h' shape would be '5135' (down-up-right-down). Distance moved in each direction is ignored because when doing mouse gestures I am rubbish at getting distances right, while with this system I can consistently do the gesture I was trying to.

Corner maintains a dictionary mapping gesture words to objects. These objects can be written in Smalltalk or Objective-C. They have to implement a -gesturePerformed method, and this will be called whenever the gesture they are associated with is drawn. Now that cross-application scripting is working, this can be used to control any application, for example locking the screen and setting an away message in the Jabber client.

Currently, there is one default gesture - drawing an h hides the active application (if the active application supports scripting, otherwise it does nothing). Others will probably be added in time for 0.4.

OgreKit Tutorial #3

Posted on 6 August 2008 by Yen-Ju Chen

OgreKit also comes with a find panel. It can work on NSTextView, NSTableView and NSOutlineView. The later two are not ported yet, but the architecture is extendable to other graphic interface. An example of using OgreKit find panel is under '/Etoile/Developer/Examples/OgreKitExample'. First, we need to connect the find panel to the text view:

- (void) awakeFromNib
{
    textView = [scrollView documentView];
    textFinder = [OgreTextFinder sharedTextFinder];
    [textView setRichText: NO]; /* Use Plain text adaptor */
    [textFinder setTargetToFindIn: textView];
}

OgreKit find panel can search both plain text and attributed text. Here, text view is set to use plain text and the right adaptor will be used by OgreKit find panel automatically. To connect find panel and text view, use -setTargetToFindIn: from OgreTextFinder. That's all.

To bring up the find panel, add this action into menu:

- (void) findPanelAction: (id)sender
{
    [textFinder showFindPanel: sender]; 
}

Now, you have a find panel which supports regular expression by default. Here is a screenshot:

OgreKit Tutorial #2

Posted on 1 August 2008 by Yen-Ju Chen

Here are some examples of using OgreKit:

In NSMutableString, -chomp remove all newlines ('\n') anywhere in a mutable string.

NSObject subclass: SmalltalkTool
[
    run
    [
        | target |
        target := NSMutableString stringWithString: 'alphabetagammadelta\n\n\n'.
        target length log.
        target chomp.
        target length log.
    ]
]

In OGRegularExpression, -replaceAllMatchesInString:withString: replaces all matched strings.

NSObject subclass: SmalltalkTool
[
    run
    [
        | regex target result |
        regex := OGRegularExpression regularExpressionWithString:'a[^a]*a'.
        target := 'alphabetagammadelta'.
        result := regex replaceAllMatchesInString:target withString: '###'.
        target log.
        result log.
    ]
]

You can even swap the matched substring like this:

NSObject subclass: SmalltalkTool
[
    run
    [
        | regex target result |
        regex := OGRegularExpression regularExpressionWithString:'(a)([^a]*a)'.
        target := 'alphabetagammadelta'.
        result := regex replaceAllMatchesInString:target withString: '(\2)(\1)'.
        target log.
        result log.
    ]
]

OgreKit also supports various regular expression syntax:

OgrePOSIXBasicSyntax    POSIX Basic RE
OgrePOSIXExtendedSyntax POSIX Extended RE
OgreEmacsSyntax     Emacs
OgreGrepSyntax      grep
OgreGNURegexSyntax  GNU regex
OgreJavaSyntax      Java (Sun java.util.regex)
OgrePerlSyntax      Perl
OgreRubySyntax      Ruby (default)
OgreSimpleMatchingSyntax    Simple Matching

Now, let's go back to Objective-C. Instead of using regular expression to replace string, you can have a delegate method for that. Use -replaceAllMatchesInString:delegate:replaceSelector:contextInfo: to specify the delegate and method, then write your own replace method. Here, the replace method is -count:contextInfo:, which will return the number of matched letter.

(void) testReplaceDelegate
{
    OGRegularExpression *regex = [OGRegularExpression regularExpressionWithString: @"a[^a]*a"];
    NSString *target = @"alphabetagammadelta";
    NSString *result = [regex replaceAllMatchesInString: target
                                               delegate: self
                                        replaceSelector: @selector(count:contextInfo:)
                                            contextInfo: nil];
    NSLog(@"Target %@", target);
    NSLog(@"Result %@", result);
}

- (NSString *) count: (OGRegularExpressionMatch *) match
         contextInfo: (id) contextInfo
{
    return [NSString stringWithFormat: @"(%d)", [[match matchedString] length]];
}

The result will be:

Target alphabetagammadelta
Result (5)bet(3)mm(6)

OgreKit Tutorial #1

Posted on 31 July 2008 by Yen-Ju Chen

David asked me to write an example of using OgreKit framework. I figured it might be interesting to do that in combination with SmallTalk. This post shows you how to set up everything on Ubuntu 8.04. Of course, you need to have GNUstep installed first.

Dependencies for LLVM are 'lemon', 'flex' and 'bison'. They can be installed from Ubuntu packages. It is necessary to use LLVM trunk for Smalltalk. Based on LLVM User Guide, you can download LLVM trunk with

svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm

After configuration with './configure', compile it with 'make ENABLE_OPTIMIZED=1' for release build (10x faster than debug build) and install it with 'make install'. It is not necessary to install the frontend for this purpose.

Once LLVM is ready, compile and install 'EtoileFoundation' and 'Smalltalk'. To avoid debug information from Smalltalk, use 'make debug=no' for compilation. Use 'st -f test.sh' under Smalltalk directory to testing Smalltalk. There are also a few examples under 'examples' directory.

For OgreKit, you need to have oniguruma from Ubuntu packages. Then compile and install OgreKit as usual.

Finally, this is a small script to check everything.

NSObject subclass: SmalltalkTool
[
    run
    [
        | regex matches |
        regex := OGRegularExpression regularExpressionWithString:'a[^a]*a'.
        matches := regex allMatchesInString:'alphabetagammadelta'.
        matches foreach:[ :x | x matchedString log.].
    ]
]

Save it in a text file called 'ogre.st', for example, and run it with 'st -f ogre.st -l OgreKit'. Parameter 'f' refers to the file name and 'l' refers to the OgreKit framework. The regular expression patterns is 'a[^a]*a', which means a string that the first and the last letter is 'a', but none of the letters in-between is 'a'. Using this pattern to match a string 'alphabetagammadelta' will give 3 substrings: 'alpha', 'aga', 'adelta'. Results are stored in an array of OGRegularExpressionMatch. Use '-matchedString' to retrive the matched substring.

Fun With Threads

Posted on 29 July 2008 by David Chisnall

This weekend I started working on a replacement for MultimediaKit. This has been on my TODO list for a while, since the current one is GPL-tainted. I started working with libavcodec and libavformat directly, since these are LGPLd.

In order to get consistent latency, ideally I wanted the decoder running in its own thread. Since we have a threading library in svn, I thought I'd try using it (okay, I wrote it, but I've not actually had the need of a threading framework since then). The first thing I needed to do was create the player object and put it in its own thread:

MusicPlayer *player = [[MusicPlayer alloc] initWithDefaultDevice];
// Move the player into a new thread.
player = [player inNewThread];

Actually, that's all I needed to do. After putting some files in the player's queue, I could periodically query its state, like this:

// Periodically wake up and see where we are.
while (1)
{
    id pool = [NSAutoreleasePool new];
    sleep(2);
    NSLog(@"Playing %@ at %lld/%lld", [player currentFile], 
        [player currentPosition] / 1000, [player duration] / 1000);
    [pool release];
}

Note the complete lack of any locking or thread operations here. The player object, after the call to -inNewThread is really a proxy which maintains a lockless ring buffer storing messages between the player and my main thread. When I send it a currentFile message, it adds it to the queue and returns a proxy. If I try to use the proxy (here, NSLog will do so by sending it a -description message) then my calling thread will block. The other two messages return primitives, so they block immediately.

When I am not sending the player messages, the run loop managed by EtoileThread sends it a -shouldIdle message whenever the message queue is empty, and if it is then it sends it an -idle message. The -idle method reads the next frame from the audio file, decodes it, and passes it to the output device. All of these are synchronous, blocking, calls (although the output device does some buffering) and so it's very simple code. Neither thread needs to spend much time waiting on a mutex - the structure used to send messages between threads is a hybrid ring buffer, which runs in lockless mode unless it has spent a little bit of time spinning (at which point it uses a mutex).

This means that, while playing, the cost of checking for new messages is very cheap (one comparison operation, in fact). While paused (and not receiving messages), the object will automatically switch to locked mode and wait for a condition variable to wake it up, so you aren't wasting CPU.

The best thing is that all of this is hidden away in EtoileThread (in EtoileFoundation), so any of your objects can use the same mechanism with almost no code. Just adopt the Idle protocol if you want to do something when your object isn't receiving messages from another thread, and send it an inNewThread message just after creation.

Pragmatic Smalltalk 0.5

Posted on 12 July 2008 by David Chisnall

I've been calling Étoilé 'a pragmatic Smalltalk' for a long time (although Nicolas, I believe, was the one to coin the expression). Smalltalk is a really great language, but it has two disadvantage:

1) It tends to be bytecode-interpreted, which is not very fast. 2) Implementations tend to be all-or-nothing.

The first is less of a problem now that CPUs are so fast they spend 90% of their time idle in a typical desktop workload. The second is much more of a problem. Smalltalk-80 includes a complete GUI and common implementations, such as Squeak adopt this model. This means that Squeak applications and 'native' applications are entirely separate. If there is one thing that Squeak doesn't have that you need, then using Squeak is not easy.

This week, I committed the first version of the Smalltalk compiler I have been working on to Étoilé svn. Unlike other Smalltalk implementations, this is designed from the ground up for interoperability. Smalltalk objects are compiled (to native code) as Objective-C objects. This means that they can subclass Objective-C objects, and can even implement categories on Objective-C objects. There is no C function interface - if you want to call C functions then call them from Objective-C.

The compiler is in three components. SmalltalkKit contains everything required to take a string containing Smalltalk code and compile it to a set of Objective-C objects.

The Support library contains things needed by Smalltalk but not Objective-C. The most important class here is the BlockClosure class, which implements a Smalltalk block as an Objective-C object with a function pointer as an instance variable and pointers to bound variables and space for promoting other variables (eliminating the need for garbage collected stack frames). There are also a few categories, such as map: and related methods on NSArray which take blocks as arguments. Note that these are implemented in Objective-C even though they are used by Smalltalk - they could, in most cases, easily be implemented in Smalltalk instead.

The final part is a tool which compiles a Smalltalk file, instantiates a specified class, and send the instance a run message. This is very small and shows how the compiler can be used, and will serve as the framework for writing complete applications in Smalltalk.

The parsing is done in Objective-C, using the Lemon parser generator from SQLite. The abstract syntax tree (AST) is constructed out of Objective-C objects, which means it's exposed to Smalltalk. As a result, Smalltalk programs can generate code easily by constructing the AST and invoking its compileWith: method, or by instantiating a parser and giving it a string.

Currently, the compiler only works in-process. It uses runtime introspection when constructing the AST. Code generation, however, is done via LLVM, and involves generating an LLVM intermediate representation (IR) version of the AST, running LLVM optimisation passes on this, and then compiling it to native code. With minor modifications, it is possible to emit the LLVM IR as bitcode and then run extra optimisations on it or compile and link it as a native library. Whether this is interesting depends on how long it takes to run the compiler. For the simple test I've done so far, program startup has taken much longer than parsing and code generation (and I'm using a debug build of LLVM, which is about 10% the speed of a release build). For larger programs, it might be worth statically-compiling. If parsing is a major overhead, it might be worth caching the bitcode for each Smalltalk input class.

So far, it is a fairly naive implementation. Lots more optimisations are possible (some are very easy) than are currently done. My aim, however, is to move as many as possible into LLVM passes, so that they can be used when compiling other dynamic languages. The code representing the Objective-C object model is taken from code I wrote for clang, the new C language family front end for LLVM, and so is also used for compiling Objective-C with LLVM.

Building a Better Garbage Collector

Posted on 6 July 2008 by David Chisnall

One day a student came to Moon and said, "I understand how to make a better garbage collector. We must keep a reference count of the pointers to each of the cans." Moon patiently told the student the following story:

"One day a student came to Moon and said, "I understand how to make a better garbage collector...

I am not a fan of many of the things Apple has done to Objective-C recently. The one thing I liked the idea of in principle was garbage collection. Unfortunately, they seem to have done this very badly, so I set about seeing if there was a better way. First, some background:

There are, generally speaking, two kind of garbage collection: reference counting and tracing. With reference counting, every assignment increments the reference count of the new value and decrements the reference count of the old value. When an object's reference count hits zero, it is freed. This is what is traditionally done with OpenStep, via the -retain and -release methods.

The other alternative is tracing. This requires every object to be known to the garbage collector. Globals are identified as 'roots' and periodically the collector attempts to navigate from the roots to every reachable object. Those that can not be reached are freed.

In 2004, some very bright guys at IBM's T.J. Watson Research Center (a nice place to visit, by the way - it's on top of a hill, with huge windows and overlooks some gorgeous scenery) came up with a Unified Theory of Garbage collection in which they propose that these are really equivalent. A tracing garbage collector needs to set a flag indicating that an object has been reached, and this can be seen as a special case of a reference count (one capped at one). Reference counting garbage collectors need some extra mechanism for detecting loops, and this is equivalent to the tracing operation.

When Apple added tracing GC to Cocoa, they threw away the reference counting mechanism. This was a shame, since all that is needed to turn reference counting into full GC is the addition of a cycle detecting algorithm. If a has a reference to b and b has a reference to a then, with pure reference counting, both a and b will leak. The rôle of the cycle detector is to run periodically and make sure this does not happen.

Fortunately, the two of the same guys who came up with the unified theory had, a few years earlier, published another paper in which they describe a mechanism for adding an efficient (i.e. fast) cycle detector to a reference counting system.

I have implemented this for GNUstep, and it shows promise. I've made a few modifications to NSObject - retain counts are now stored in a 16-bit value (if you have more than 65535 references to a single object, you probably have a bug) and the other 16 bits are now used to store flags, including a colour. The colour is set by the cycle detection algorithm, which is invoked periodically when a buffer of objects which have been released but not freed becomes full.

One minor problem with this is that objects can now exist safely in loops and -retain can be called on an object which is currently executing -dealloc. This can lead to an infinite loop, and some careful juggling is required to ensure that no objects deallocate themselves while freeing loops.

The code works, although it has a few (easily fixable) limitations. I created a small graph of five Pair objects. Each one has a reference to itself and to the next one in the ring. The code correctly determines that this contains a loop, and destroys all five objects when the autorelease pool is destroyed.

The only major limitation is that I've only written code for atomically accessing the colours on x86. This can be fixed trivially by simply writing these functions. A smaller issue is that a number of GNUstep classes indulge in premature optimisation by calling NSDeallocateObject() in their -dealloc method, rather than calling [super dealloc].

I currently use a modified version of the algorithm in the paper which uses an NSHashTable to store pointers to objects that might contain loops. Since there's space in the flags field for a 'buffered' flag, I can easily extend it to use this, and replace the hash table with a static array. This is better for two reasons: it should be faster, and it means that we can use thread-local storage without having to worry about explicit destructors (which are currently called by listening for a thread terminating notification, which is slightly fragile and will potentially fail to catch things released when a thread is dying).

Since some code already handles loops via unretained references, the current code has problems. To avoid this, I introduced an extra colour (transparent), and any objects with this colour are assumed to always be acyclic. This allows automatic loop detection to be turned on on a per-object, or per-class basis. In future, this can be used in reverse: to turn off loop detection for intrinsically acyclic data structures (e.g. trees).

My main motivation for this is for the Smalltalk compiler I am currently working on. Since Smalltalk expects garbage collection, and Objective-C does not provide it, this presents a small problem. A tracing collector can be used, however this is very tricky when integrating with a C-like language, since it means that everything which may potentially contain pointers has to be checked, including integers and untyped buffers.

On the way to EtoileUI, Part 1: Back to the Hackathon

Posted on 6 June 2008 by Quentin Mathé

For the Swansea hackathon, I gave a quick overview of EtoileUI. When I came back, I intended to upload it and wrote a post on the subject but the time went by faster than I expected :-)

During April and May, the framework has steadily improved to the point of being now usable on both GNUstep and Cocoa, but it is still quite experimental and many things remain to be worked out. In the early days of April, the stability of EtoileUI on GNUstep wasn't really satisfying too, since I initially wrote a large part of it on Mac OS X before backporting it this winter.

Before trying to explain what is EtoileUI in upcoming posts, here is the link to the presentation (PDF) I did.

As you probably know from the previous post, the integration of CoreObject and EtoileUI is moving forward, especially now that Eric has started to use both in a real application named EtoileTunes. This recent project has also motivated me to be quicker at squashing annoying bugs in these frameworks. On my side, a generic object manager based on the example of the last slide is coming along nicely. For now, it is mostly a CoreObject-based file manager which supports several views (icon, list, column, etc.), although it can be used to browse and mutate any object graphs that comply to a simple object collection protocol (declared in EtoileFoundation).

EtoileTunes

Posted on 26 May 2008 by Eric Wasylishen

For the past few weeks I've been working on EtoileTunes, a music player for Etoile. My goals were to try improve my Objective-C/GNUStep knowledge, to try out programming with EtoileUI and CoreObject, to work on a replacement for MultimediaKit, and to hopefully end up with a good enough music player to use regularly.

It's still in fairly early stages, but I have something that uses TagLib to read music file metadata, then constructs a CoreObject with that metadata. It then puts these into a COGroup subclass which represents a playlist, and displays this group with EtoileUI. The eventual goal, as I understand it, is for EtoileTunes not to be a separate application, but just a pre-defined layout which the normal object-manager UI in Etoile can take on. Jesse, Quentin, and I discussed some ideas for what would make a good music manager UI, so hopefully I can build some working mockups with EtoileTunes.

On the MultimediaKit replacement side, I have an incomplete, but working, Objective-C wrapper around Xine-lib which provides an API similar to MultimediaKit's (only for music, not video playback so far.) I'm also working on a GStreamer backend. A future task might be to write a MultimediaKit framework from "scratch" - an Objective-C framework that would fit between Etoile apps/services and the operating system's/sound server's audio API. It could use the ffmpeg project's libavcodec/libavformat libraries to do all of the hard work of decoding media formats. The advantages of this would be having something that exactly fits Etoile's needs, and fits in to the system perfectly (for example, the same code for decoding music could be used for transcoding between different formats, and be integrated into a system Etoile might provide for converting file formats), and being able to ensure details like gapless playback work perfectly. This might be a lot of work though - especially if it has to handle video playback and recording, and multiple OS backends, so it may be best to stick with GStreamer/Xine for now.

Here's a screenshot of EtoileTunes:

If you would like to try it, the code is in /branches/ericwa/EtoileTunes. It's still very work-in-progress, though.

Lastly, any suggestions for a better name? :)

Compiler Fun

Posted on 12 May 2008 by David Chisnall

Anyone following the Étoilé svn logs recently will notice that I haven't been committing much for a few weeks. The reason for this is that I've been taking a short break to do some compiler hacking.

Objective-C support was first added to GCC by some guys at NeXT. They didn't want to release their code, but were eventually forced to by the FSF. They did not release the code for their runtime library, and so this code was completely useless to anyone else. RMS wrote a drop-in replacement for this library, which became the GNU Objective-C runtime. Gradually the GNU and NeXT runtimes diverged and the Objective-C support code in GCC became littered with #ifdefs.

After Apple bought NeXT, they continued developing their version of GCC in a branch. This branch was slightly cleaner, since it never had support for the GNU runtime, but no use to anyone on platforms other than Darwin for the same reason. This code is no fun at all to work with - Objective-C structures are lowered to the corresponding C structures, so there is no clean Objective-C AST to work with and runtime-specific code is interleaved with the abstract representations. When Apple add a new language feature, they add it to their branch, and if anyone else wants to use it then they have to merge the changes into the main trunk. Unfortunately, no one is doing this and Objective-C support in GCC is in a rather depressing state (bugs in Objective-C are not seen as show stoppers for a release, as we saw in the early 4.x series).

Recently, GCC switched to GPLv3. Apple corporate policy is that they will not touch GPLv3 code, and so the Apple branch is now a fork of GCC 4.2. Features added to GNU GCC will not find their way into Apple GCC and vice versa, unless explicitly licensed in a compatible way by their contributor.

Apple have also started looking at a new compiler, known as LLVM. This is a modular infrastructure for building compilers. It currently has an Objective-C/C/C++ front end based on Apple's GCC. This combination of an LLVM back end and a GCC front-end is typically known as llvm-gcc. It is found in the iPhone SDK and is likely to be found in the OS X dev tools soon. GCC isn't really designed to be split apart like this, however, and so the Apple guys have been working on a new one.

Unlike GCC, clang has very clean layering. This is intentional, since Apple also want to use it in XCode for syntax highlighting and refactoring tools. This means that every single Objective-C language construct gets corresponding AST nodes which are then passed to another part of the program which emits LLVM intermediate representation (IR) code - single static assignment assembly language - which is then turned into native code for the desired platform.

When I first looked at clang, most of the parsing code for Objective-C was done, but none of the code generation part. This meant that I was free to add any interfaces I wanted. Clang now has an abstract class encapsulating all of the runtime-specific behaviour and hooks in the generic code that call this. I have also written a complete implementation of this for the GNU runtime and an almost-complete one for the Étoilé runtime. As a result of this, clang can now compile about 90% of the files in GNUstep-base without issue. The remaining ones are failing due to a couple of outstanding bugs with implicit casts (the LLVM type system is a lot more strict than the Objective-C one and so casts which are implicit in Objective-C need to become explicit in the IR) and a few C features. GNUstep uses variable length arrays in a few places, for example, and I have only added partial support for these.

My changes to Clang are currently undergoing code review, but after this has happened and I've made the required changes they should go in.

Objective-C isn't the only thing that makes this interesting. Since the object model code is all isolated in a separate class, it is possible to plug this into other compilers trivially. Generating classes, protocols and categories, selectors and message sends that use the underlying GNU runtime (and soon the Étoilé runtime) functionality is trivial when using this class (each high-level construct is mapped to a method call). I am currently in the process of writing a Smalltalk compiler that uses this same back end. LLVM supports both JIT and static compilation, so we will be able to JIT-compile Smalltalk while developing, dump it to a file, and static compile it for distribution.

This means that Smalltalk will be a first-class citizen of the Étoilé ecosystem. Applications will be able to be written in Smalltalk and Smalltalk classes will be able to inherit from Objective-C classes. There is no bridging - Smalltalk methods will be compiled to native code and attached to the same structures as Objective-C methods. Once this is finished, I will be recommending Smalltalk as the development language-of-choice for new Étoilé applications. If you discover that a particular piece of code is too slow (after profiling) then you might want to rewrite it in Objective-C (or even pure C), although I don't expect Smalltalk to be much slower than Objective-C.

Smalltalk is not the only high-level language we will implement in this way - just the first. Expect Io, JavaScript and maybe even Self implementations later. These languages are all prototype-based, however, and so require a few features that are not found in the GNU runtime (but are in the Étoilé runtime) for full support.

Hackathon Recap

Posted on 11 April 2008 by Quentin Mathé

Étoilé hackathon ended ten days ago, I left Swansea on Tuesday afternoon to flight back to Paris after a mix of train and bus. Afterwards David and Damien Pollet succeeded in closing the hackaton by writing the first bits of a bridge between GNU Smalltalk and Objective-C.

Following Friday talks, we wandered around the university looking for a pub where we could eat. We finally ended up at David's place and had a nice break around beers (that aren't beers but ales), coffee, saucissons and we ordered some real food too. Afterwards Nicolas was still motivated to learn more about EtoileUI. His interest for EtoileUI had the bad side-effect of shortening the night quite dramatically!

During the week-end, we discussed the proper way to implement a semantic editor and how to integrate it with EtoileUI and CoreObject. Nicolas spent most of his time writing a new prototype. He already wrote a very rough one few years ago.

Damien arrived from Lugano on the second day of the hackaton. After having installed Étoilé, he started to play with the possibility to hack a Smalltalk bridge. In the meantime, David continued his work on EtoileSerialize, while I spent a large part of the week-end cleaning stuff here and there and hunting bugs of EtoileUI on GNUstep (Saturday and Sunday night especially :-).

We also began to clean EtoileFoundation a bit, David moved EtoileThread to it, and last week I recently managed to bring EtoileXML too and have both compiles fine as subframeworks (EtoileFoundation.h playing the role of an umbrella header). In the middle of our discussions centered around CoreObject and EtoileSerialize, I got the impression David was doing some coding on LLVM too, while Damien was fighting with autotools next to me. At least, I was able to follow David's presence on the #llvm channel right on the wall screen where Jesse appeared the previous day. I'm pretty sure everything went on rougly that way until Monday!

On Saturday evening, I lost myself in Swansea suburbs and the five minutes walk to buy a few ales became a one hour walk in the Welsh drizzle. The next day, the weather persisted to be quite unstable, constantly hesitating between storm, sun and waxy clouds. Monday, our own trinity (EtoileSerialize, CoreObject and EtoileUI) was expecting us for a last serious day of hacking. The sky was now in a better shape and could have been qualified of sunny and warm by previous day standards ;-) David had the good idea to plan a break at the pub for the evenining, so we could slow down a bit before really ending the hackaton. For the last day, I went to a coffee shop where I previously bought some nice scones and Chelsea cakes (iirc), then we left David's place for the university and our daily coffee session with a nice vista on the sea and the sound of the seagulls in background (or is it just my vivid imagination?).

The hackaton was just great, although the time went by very quickly. David had well organized everything and after our daily hacking sessions, he even managed to cook some very nice stuff (like weird and tasty squashes) for the hungry french hackers, so I'm looking forward to the next one. May be in France…

Hackathon Progress

Posted on 30 March 2008 by David Chisnall

The hackathon started on Friday, with Quentin and Nicolas arriving in Swansea in the early afternoon. After briefly settling in, the three of us gave a short series of talks to the postgraduate students in the department of computer science.

I managed to pursuade Quentin to write some documentation, which can be seen here:

Jesse, unfortunately, was unable to be with us in the flesh, but appeared on the wall in the middle of the afternoon as our very own Big Brother (or possibly Emmanuel Goldstein):

Most of the time has been spent working on CoreObject-related things. Nicolas has implemented a basic semantic text editor, which Quentin is wrapping up in CoreObject. I spent most of the time so far working on EtoileSerialise (or EtoileSerialize, as it is now known). It now passes the test automatically serialising nested structures of arrays of structures. The storage has also been abstracted away, and branching (finally) implemented.

Last night, after a day of hacking on various things, we implemented CorePizza, the official Étoilé project food:

Someone stole an hour from the middle of last night, so we're a bit tired today, but still making progress.

Summer of Code

Posted on 18 March 2008 by David Chisnall

The Google Summer of Code list was published last night. We have some good news and some bad news. The bad news is that Étoilé wasn't selected. The good news is that GNUstep was. Since Étoilé is built on top of GNUstep, everything that benefits them benefits us, so anyone interested in the summer of code and Étoilé should consider applying for one of the GNUstep places. We don't know how many places each project was awareded until later on in the process, but last year they got two so hopefully a couple of interesting projects can be finished as a result of the summer.

Spring Hackathon

Posted on 6 March 2008 by David Chisnall

The dates and location of the first Étoilé Spring Hackathon are now finalised. We will officially be starting on Friday the 28th of March and continuing until the 1st of April.

The Swansea University Computer Science department has very kindly agreed to provide us with a room for the duration of the hackathon in this building. Access to the building and room is via security card. I'll arrange some visitors' cards for attendees.

We're about fifteen minutes walk away from a pub that serves real ale and has free WiFi, which should come in handy for the evenings.

If you're coming, let me know when you're likely to arrive. I'll try to arrange some kind of social event on the Thursday evening before the real hacking starts.

Some Quick StepChat News

Posted on 13 November 2007 by David Chisnall

My last few posts have all been about very low-level stuff and have been sorely lacking in pretty pictures.

For the last day or so I've been working on adding vCard support to StepChat. It's not finished, but it now creates a 'Jabber People' group in your address book and adds any published vCards there. In future, it will merge any changes in a nice way. For now, it only gets vCards once (I need to poke the presence stuff to spot vCard updates). One nice bonus is that Jabber stores avatars in vCards, and once you can load vCards you get avatars almost for free. You will see in the screenshot that Dom has a nice line-art dragon for his avatar.

I also chased down the bug that was preventing colours displaying in the roster with GNUstep, so now that works too. Finally, one more thing you can see in the screenshot is that GNUstep now has nice menu dividers. This shot was taken with the Cairo backend (which works almost perfectly; the only bug I've seen with it is that text in the status message box on the roster gets horribly smeared. Since this doesn't happen in other text boxes it's probably a nib loading error).

Unfortunately, the bug causing windows to have the wrong titles is still there. I will have to have a hunt for it at some point (or just wait for Yen-Ju to fix it).

Another Day, Another Runtime

Posted on 10 November 2007 by David Chisnall

After spending a little while poking at the GNU runtime, I came to two conclusions:

• It was at least twice as complicated as it needed to be.
• GNU coding style really hurts my eyes.

I've spent a little while thinking about what I want from a runtime. One of my recent projects has been writing a Smalltalk JIT that targets the GNU runtime (still quite work-in-progress) and so I know the GNU runtime can support Smalltalk as well as Objective-C. Quentin, meanwhile, has been working on the Io bridge. While this works, it is a bit ugly because the Io object model doesn't really mesh well with the Smalltalk object model that the GNU runtime uses. More on this later.

Beyond better being able to support Io, I read a few interesting papers recently. The first was on Polymorphic Inline Caching. This is quite a neat idea, and allows you to eliminate the cost of dynamic method lookups in a number of cases. Unfortunately, this is quite hard to get right. Consider the simple Objective-C line:

[object message];

With the Apple runtime, this will be translated roughly into something like this:

objc_msgSend(object, @selector(message));

I'm cheating a bit here, and skimming over how the @selector() directive is expanded. In contrast, the GNU runtime does this:

IMP method = objc_msg_lookup(object, @selector(message));
method(object, @selector(message));

This is quite nice, since it means that a small compiler change is all that's needed to cache the method. We could replace this with something like this:

static IMP cached_method = NULL;
static Class cached_class = Nil;
if(cached_class != object->isa)
{
    cached_method = objc_msg_lookup(object, @selector(message));
}
cached_method(object, @selector(message));

Now you only need to bother with the (expensive) method lookup if you reach this bit of code with two different object types. There are a lot of places in code where you will get the same kind of object all of the time, and this can give a huge speed boost. In other cases, you get a number of different ones and this is where polymorphic inline caching comes in. Rather than keeping a single (class, method) pair cached, you keep a few. Profiling can determine the optimal size for this cache relatively easily.

Nice and easy? Well, there's a catch. Objective-C is a dynamic language. You can load bundles which will replace methods at runtime and languages like Io allow even different objects to have different methods. This means that you need to check that the cache is valid before you use it. A problem.

This, and the difficulty in supporting Objective-C 2.0 on the GNU runtime caused me to write a new one from scratch. This took just under 48 hours (after which I ate and went out to a well-earned salsa class). What's new?

First, inline caching can now be done safely. Rather than looking up methods, the runtime looks up slots. The slot contains (among other things) an IMP and a version. How does the version help? Consider two classes, A and B. A inherits from B, which implements a -foo method. Somewhere in my code, I call this method on an instance of A and cache the result. Two things can cause this cache to become invalid:

• B's implementation of the method being modified / replaced.
• A having an implementation of the method added.

The first case Just Works™ since a pointer to the the slot (which contains a pointer to the method) is cached. You can modify the method without any problems (great for debuggers and runtime optimisations). The second case is more tricky. When you add a method to A, it first performs a lookup on the selector. If this returns a non-NULL slot, then the version of the located slot is incremented. Any time you cache a slot, you should also cache the version; if there is a version mismatch with the cached slot then you need to perform the lookup again.

The slots also contain an optional offset. This can be used to implement very fast set/get methods. A lot of the time you wrap instance variables up in set/get methods to insulate users of the code from changes to the instance variable layout. This comes with a speed penalty. I can't make this go away completely, but the new runtime allows you to avoid the method call and just access the ivar directly, while maintaining the dynamic lookup. This can make things like KVO faster, since you can do direct ivar access while there are no observers and then switch to indirect access when there are some.

I said the new runtime was simpler (no exaggeration; it's roughly 10% of the code size of the GNU runtime). That's partly because it works in a slightly different way to other Objective-C runtimes. While the GNU runtime provides the functionality required to implement Smalltalk in C, the new one provides the functionality required to implement Self in C, and then implements Smalltalk in Self (which is very easy).
Every object has (or, rather, can have) its own dispatch table and its own lookup function. Classes really are just objects. Anything you do with classes, you can also do with objects; for example you can add a method to a single object at runtime (say hello to closures, prototypes, and all of the things Io and Lisp programmers have been mocking you for having to do without).

The class model contains a nod to that used by Animorphic Smalltalk. This used mixins as a base type. That's effectively what I'm doing although they're called classes so as not to scare off the old Objective-C programmers. Classes can be composed as mixins are, which is how concrete protocols are supported (the only difference between a concrete protocol and a mixin is that the compiler does type checking for a concrete protocol. From the perspective of the runtime they are the same).

Oh, and every object has an associated recursive mutex, so @synchronized can now be generated easily.

To get an idea of how the runtime is used, take a look at example.c, which contains some simple example Objective-C code in comments and the equivalent code the compiler should be producing. I'd love to see this supported in LLVM, so anyone familiar with that codebase who feels like lending a hand please let me know.

You can also find more information in the release announcement email, including a more detailed overview and the API docs.

None of the interfaces are set in stone yet, so any suggestions are welcome. You can grab a copy of the code by doing:

svn co http://svn.gna.org/svn/etoile/branches/libobjc_tr/

Labels: libobjc, Objective-C, runtime, shiny

Objective-C: Étoilé Vs Leopard

Posted on 18 October 2007 by David Chisnall

Mac OS X 10.5, codenamed Leopard, is due out in a week or so. One of the features it advertises is Objective-C 2.0. I've written a bit about Objective-C 2.0 before. In this post, I'm going to compare some of the new language features present in Étoilé with those present in Leopard.

Garbage Collection

The big feature all the Java programmers want is garbage collection. GNUstep has actually supported this for a little while. If you use RETAIN() and RELEASE() macros instead of sending retain and release messages, you get the correct stubs for the garbage collector generated when compiling with garbage collection enabled, or -retain and -release messages sent otherwise.

This support was originally begun in 2004, but I'm not aware of anyone who uses it. Part of the problem is that mixing GC and non-GC code is tricky, so it's really only an option for people with no legacy code. Another part is that it adds some runtime overhead.

Loose Protocols

Apple now allows you to specify that some methods in a protocol are optional. Apparently this is useful, but I can't think why. Objective-C gives two ways of accomplishing this already. The first is to use an informal protocol; a category on NSObject with a default (typically null) implementation of the methods. The other is to query at runtime with respondsToSelector: whether a delegate implements a method.

The point of using a formal protocol, rather than an informal one, is so that the compiler can check that you have implemented the methods. Another possible reason is to allow a runtime check for a set of methods at once. A loose protocol gives you none of these. It just moves things that should be in the documentation into the code. Great for people who believe header files are documentation, not so great for the rest of us.

Concrete Protocols

Concrete protocols are a potentially useful part of Objective-C 2.0. They allow protocols to contain default implementations of a method. In Étoilé, we have something similar; typesafe mixins.

Mixins allow you to maintain the separation of interfaces and implementations that Objective-C encourages. Mixins, unlike concrete protocols, are defined as classes. If you want to add a method to a class, you first define a class implementing it, like this:

@interface Mixin : NSObject {
}
- (void) method;
@end
@implementation Mixin
- (void) method
{
    NSLog(@"Method called");
}
@end

When you want to apply it to a class, you simply do:

[aClass mixInClass:[Mixin class]];

After this, all instances of aClass will responds to -method (it will not get a double helping of the methods declared in NSObject). You can even declare and use instance variables in the Mixin class. When you apply a mixin you will get an exception if one of the following happens:

• The types instance variables declared in the mixin do not match those declared in the class. The class can include more instance variables than the mixin, but it must include all of them. This allows mixins to directly access class ivars; something not possible with concrete protocols.
• The types of methods declared in the mixin conflict with the types of methods declared in the class (or a class the class inherits from).

Method Attributes and Properties

Method attributes might be really nice, but the number of the GCC function attributes that you are allowed to use is very small. Most of the ones that are actually useful can not be used in Objective-C without radically changing the way in which method lookup is handled; for example by adding an equivalent of Java's finally keyword.

Properties seem at first glance to be a nice idea. They are close to C#'s implicit set and get methods. In terms of expressiveness, they give nothing more than key-value coding already allows us. They may be slightly faster; the compiler could possibly add some code to translate them into ivar lookups if the implementation is for direct access to ivars. Something similar could probably be done with KVC, in the same way that polymorphic inline caching works. I'd be surprised if Apple has implemented this, however. At the moment, the only advantage is to add some confusing extra syntax.

The only remaining feature of Objective-C 2.0 that I recall is the foreach construct. Étoilé has a macro which works in a similar way. The following two lines are semantically equivalent:

FOREACH(anArray, string, NSString*)
for(NSString * string in anArray)

The latter is slightly faster, but requires anArray to implement a very messy countByEnumeratingWithState:objects:count: method, which retrieves 16 objects with a single call. The Étoilé version is slightly slower (although it does do IMP caching for you), but works with any collection that supports -objectEnumerator and so does not require multiple code paths. It's included with EtoileFoundation, so can be used on OS X too, including OS X 10.4 and earlier.

Prototypes and Futures

We've run out of new features for Leopard, but there's still one new one for Étoilé. We have support for prototypes in Objective-C. Any object that inherits from ETPrototype, or implements the ETPrototype protocol can use them. This required a small (binary-compatible) modification to the runtime system to allow delegation of method lookup to the class.

By using nested function (not supported on OS X), you can create and add methods at runtime, like so:

id anObject;
DEFMETHOD(method)
{
 //Code goes here.
}
[anObject setMethod:(IMP)method forSelector:@selector(foo)];
[anObject foo];

You can also declare methods with arguments, and declare them outside the scope of a function / method. Note that, as with nested functions, you can not call the method after the current function has returned if it references any local variables. Note too that instance variables in the method can only be accessed by casting self to the correct type and accessing them explicitly (e.g. ((MyClass*)self)->ivar).

These prototype objects can then be -clone'd, have KVC-accessible ivars added and removed using the -setValue:forKey: method, and be used just as prototypes in Self or Io. We don't restrict you to class-based programming.

Note, however, that prototypes do come with some runtime overhead and so should probably not be used everywhere. The same mechanism can be used for closures; if the nested function you add as a method uses lexical scoping, and is called immediately, then it will work as a block would in Smalltalk.

While not technically a language feature, as I mentioned earlier, we also have support for futures.

Playing with the Runtime Again

Posted on 9 October 2007 by David Chisnall

Everyone should have a hobby, and at the moment mine seems to be abusing the GNU Objective-C runtime. One of my spare time projects is writing a Smalltalk JIT that uses the GNU runtime to provide the object model (allowing you to subclass Objective-C objects in Smalltalk).

Last night, Quentin demonstrated the old maxim that the way to get anything done in an open source project is to tell the developers that it sucks because it can't do X; pretty soon they'll have it doing X, for any value of X. Quentin's criticism was that it wasn't possible to subvert the message dispatch mechanism very easily.

What does that mean? Let's take a look at what happens when you send a message in Objective-C. First, you write something like this:

[object doSomethingWith:aParameter];

The compiler then converts this into something like this:

SEL sel = sel_get_any_uid("doSoemthingWith:"); IMP imp = objc_msg_lookup(object, sel); imp(object, sel, aParameter);

Note that this is a simplification, and the selector will typically be cached somewhere. The important function is objc_msg_lookup, which returns the function that implements the method. These functions always take the object and selector as the first two arguments, and may take others.

For a language like Smalltalk or Objective-C, this mechanism makes sense. For something like Io, it almost does. The problem, for Io, is the implementation of objc_msg_lookup(). This looks at a sparse array structure in the class structure to find the mapping. This isn't helpful for a prototype-based language like Io, where instances might have different methods to their classes. For Io, you want to be able to alter the behaviour of the objc_msg_lookup() function on a per-object basis. for bonus points, you want to do this without breaking binary compatibility (the GNU C++ standard library people were very unpopular when they did this).

Fortunately, the Objective-C class structure has a field called info, which is a bitfield. Actually, it's half a bitfield; the upper half is used to store the id of the class in the system, limiting you to 64K different classes, and the lower half stores flags. These flags are used for various different purposes, including indicating whether a class has an +initialize method that needs to be called. Not all of them are used, so I added a new one. I then modified the objc_msg_lookup function to include a special case if this is set.

Now, if you set the flag on your class then the runtime will know it wants to handle message lookup itself:

+ (void) initialize { CLS_SETOBJECTMESSAGEDISPATCH(self); [super initialize]; }

You then implement a method like this for your class:

+ (IMP) messageLookupForObject:(id)anObject selector:(SEL)aSelector

This lets you store your own version of a dispatch table in an instance variable, so you can add methods to a specific object at runtime. Objects which are extended in Io can have this flag set on their classes at runtime, and use a separate dispatch mechanism for the Io methods.

It can also be used for more efficient proxying for local objects. The CoreObject proxy object wants to pass messages that don't change object state right through, without logging them. With this mechanism, it will be possible to implement a hashmap lookup with the same sort of cost as performing a normal message lookup for messages that are passed through (store an NSMapTable in a COProxy ivar containing selector to IMP mappings for the proxied object), and return the message forwarding IMP for those that aren't.

Another, potentially interesting, option would be to combine this with some runtime code generation to dynamically construct proxy methods that would log their arguments and then pass them on without needing to construct an NSInvocation.

Hopefully, this patch will make it upstream as far as the GNUstep version of libobjc, even if it doesn't make it all the way into GCC. Anyone who wants to play with it themselves can find the diff in this mailing list post.

Futures in Objective-C

Posted on 23 September 2007 by David Chisnall

I like concurrent programming, but I don't like threads. Like files, they're a nice abstraction for operating system designers but not so much for userspace hippies.

In functional languages, you can often get concurrency for free by having a clever compiler, instead of a clever human. This is good, because clever humans are expensive. Clever compilers are too, but they're easier to copy than clever humans.

Consider the following bit of Objective-C:

id foo = [anObject doSomehing];
...
[foo doSomethingElse];

In Smalltalk, objects were regarded as simple computers that communicated by message passing. The fact that this message passing was implemented with a stack was hidden. To the Smalltalk way of thinking, the objects were independent. This bit of code sends a doSomething message to anObject, and block until it sends a return message.

From here, it doesn't take long to realise that you don't actually need it to block until the [foo doSomethingElse] line. So, can we implement this in Objective-C in the general case? The answer is yes, and that's what the EtoileThread framework (soon to be in EtoileFoundation) does. It works on OS X too.

I actually wrote EtoileThread in a hotel in Dublin in June last year, but I recently rewrote a lot of it to be more efficient. I'd like to give a little overview of how it works.

There are three core components to this. The first is the ETThreadedObject class, which encapsulates an object in its own thread. It's an NSProxy subclass, and forwards messages to the real object. You create it typically via an NSObject category, which adds +threadedNew and -inNewThread methods. When you send a message to the object returned by either of these, the following sequence happens:

1. The invocation is caught by the ETThreadedObject and put into a ring buffer.
2. The forwardInvocation: method returns, and the calling code receives an ETThreadProxyReturn object.
3. The second thread retrieves the invocation from the ring buffer and executes it.
4. The second thread passes the real return value to the previously returned ETThreadProxyReturn.
5. Any calls to methods in the returned proxy block until this point.

What does that look like in practice? Well, we'll look at the simple example program included with the framework (ETThreadTest.m for those following in svn) and see. First, we define a simple class that has some trivial methods:

@implementation ThreadTest
- (void) log:(NSString*)aString
{
    sleep(2);
    NSLog(@"%@", aString);
}
- (id) getFoo
{
    sleep(2);
    return @"foo";
}
@end

The first just NSLogs whatever is passed to it, and the second returns a constant string. Next, in the main body, we create an instance of this in its own thread:

    id proxy = [ThreadTest threadedNew];

We then send this a log message:

    [proxy log:@"1) Logging in another thread"];

And then a getFoo message. Recall that the implementations of both of these messages had 2 second delay built into them. This was introduced to make it obvious which order everything was being executed in.

    NSString * foo = [proxy getFoo];

    NSLog(@"2) [proxy getFoo] called.  Attempting to capitalize the return...");

Then, we NSLog the return value from the getFoo method (for good measure, we'll send it a message and NSLog the result, rather than NSLoging it directly):

    NSLog(@"3) [proxy getFoo] is capitalized as %@", [foo capitalizedString]);

Finally, since we know we are calling a future, we get the real object out and NSLog it.

    if([foo isFuture])
    {
        NSLog(@"4) Real object returned by future: %@",
                [(ETThreadProxyReturn*)foo value]);
    }

What happens when we run this? Take a look:

$ ./ETThreadTest 
2007-09-23 21:59:39.718 ETThreadTest[25196] 2) [proxy getFoo] called.  Attempting to capitalize the return...
2007-09-23 21:59:41.695 ETThreadTest[25196] 1) Logging in another thread
2007-09-23 21:59:43.695 ETThreadTest[25196] 3) [proxy getFoo] is capitalized as Foo
2007-09-23 21:59:43.695 ETThreadTest[25196] 4) Real object returned by future: foo

Note that the NSLog from the main thread completes first. Note also the two second delays. Finally, note that the third and fourth log statements don't complete until after the getFoo method has run, since they depend on the returned value.

What's improved in the implementation of this in the last week? My first version was quite experimental. It used an NSMutableArray to store the invocation queue. This meant that every message going into the queue required these steps:

1. Acquiring a mutex.
2. Inserting an object into an NSMutableArray.
3. Signalling a condition variable (if the array was empty).
4. Releasing a mutex.

On the receiving end, you needed the following:

1. Acquiring a mutex.
2. Sleeping on a condition variable (if the array is empty).
3. Removing the first object from an NSMutableArray (quite expensive).
4. Releasing a mutex.

This is a minimum of four system calls (a maximum of six) and at least one expensive array operation. This isn't too bad if you are only very occasionally sending messages to your threaded objects, and are expecting them to take a long time to complete. If you are sending a lot of messages, however, the overhead quickly stops it being worthwhile to bother with the second thread.

The new implementation uses a lockless ring buffer in this situation. Inserting an object into this involves a subtraction and a comparison to see if it's full (we just spin using sched_yield() if it is, but with enough space for 128 invocations in the buffer that should be rare), inserting two objects in a C array (the invocation and the proxy value), an addition, and a memory buffer (on weakly-ordered platforms, not on x86). Removing an object switches back to the old-style locking mode if the queue is empty. We don't want to spin if there is no work to do for the consumer thread, because that could last a long time. Then, taking the objects out of the array is just copying their pointers out of the array and another addition.

The ring buffers, like those in Xen, use free-running counters and a mask of the lower bits to translate this into an address.

There are two special cases. The futures code only works for methods that return objects. For methods that return non-objects, we need to wait until the invocation has completed. For void return methods, we simply complete asynchronously, but don't return a proxy object.

If you want a more detailed overview of how it all works, there are lots of comments in the code. Have fun, and file helpful bug reports.

XHTML-IM Support

Posted on 27 August 2007 by David Chisnall

Ages ago, I wrote most of an XHTML-IM parser for StepChat. I didn't enable it in the default build, since it wasn't quite working. Over the weekend, I finished it off, tested it, and added support for generating some XHTML-IM. The code is not completely compliant with the specification, since it doesn't bother checking if the other party supports HTML. This will probably be added soon. It also contains a couple of work-arounds for libgaim-based clients, which interpret the standard quite creatively.

Below, you can see a conversation I had with my debugging persona (yes, I know talking to yourself is a sign of madness, especially when you use the Internet to do it). The first image comes from FreeBSD/GNUstep while the second one comes from OS X. Camaelon is not enabled for Jabber on my FreeBSD box because it was breaking things (I think it's fixed now, but I haven't got around to removing the default)

The more observant among you will notice that I fixed a bug in the handling of bold text in the middle of this conversation.

Labels: jabber, omgponies, stepchat, xhtml-im, xmpp

Drop Shadows

Posted on 9 August 2007 by David Chisnall

Shadows are nice. Since we got compositing managers for X11, we've had the ability to do this. The xcompmgr program does it, but in a gimmicky, eye-candy way, which doesn't really add much to usability.

Shadows are definitely something we want though, so yesterday I forked xcompmgr and put it in our repository. Today we have some (early) results. Drop shadows:

Three things to note. The first is that the dock and menu bar don't have shadows. The second is that the active window has a bigger one than the others. Finally, there are no shadows visible on the Typewriter window at the top right of the screen. The first two of these are intentional. The third is not, but only occurs in screenshots, not on the screen, and so is probably not very important.

Labels: shadow, shiny, ui

Étoilé 0.2 Troubleshoot

Posted on 3 August 2007 by Yen-Ju Chen

Since 0.2 release, there are some issues regarding setting up the Étoilé environment. Here are a few steps to narrow down the problem.

First, make sure your GNUstep is correctly installed. Current stable release of GNUstep is Make 2, Base 1.14, GUI/Back 0.12. You can also get them from GNUstep stable branch. Run a few GNUstep applications in any graphic environment to be sure your GNUstep is working. And remember to source GNUstep.sh in your profile. You can get help from GNUstep maillist if problems occur at this stage.

Second, after installing Étoilé, run a few user-level applications, such as Typewriter, Sketch, StepChat, Vindaloo, AddressManager and FontManager. They are regular GNUstep applications. You can run them in any graphic environment such as GNOME, KDE or WindowMaker. If you run 'setup.sh' during the installation, several bundles have also been installed. For testing purpose, you can remove them by:

defaults delete NSGlobalDomain GSAppKitUserBundles

and add them back by :

defaults write NSGlobalDomain GSAppKitUserBundles '(
/usr/local/GNUstep/System/Library/Bundles/Camaelon.themeEngine,
/usr/local/GNUstep/System/Library/Bundles/EtoileMenus.bundle, 
/usr/local/GNUstep/System/Library/Bundles/EtoileBehavior.bundle)'

Make sure the paths are correct. You can also try different combination of these bundles. Camaelon is the theme engine. EtoileMenus is the horizontal menu. EtoileBehavior handles various tasks behind the scene and you will not see any change on user interface with it.

Third, you should be able to set up Étoilé manually. With GNOME, you can log into a fault-safe session with xterm only. There should be something similar on KDE. Once xterm shows up, run these system-level applications one-by-one:

gdomap &
openapp Azalea &
openapp AZBackground &
openapp EtoileMenuServer &
openapp AZDock &

If they all run propertly, you are close to have a functional Étoilé environment. Log out the session by exiting from xterm and log into the fault-safe session again. Run 'etoile_system' tool (no openapp !) and all the system applications should launch automatically. If not, check your SystemTaskList.plist in your GNUstep/System/Library/Etoile/ or ~/GNUstep/Library/Etoile/. It contains all the applications you launched one step before.

Finally, to add Étoilé into your GDM, make sure these files exists:

1. etoile.desktop, in your xsession directory, such as '/usr/share/xsessions'. And it should contain a line 'Exec=/usr/local/bin/etoile'. You can get this file in Etoile/Services/Private/System/.
2. /usr/local/bin/etoile. This is the actual script to run 'etoile_system'. It should look like this:

   . /usr/local/GNUstep/System/Library/Makefiles/GNUstep.sh etoile_system

   There is a space between '.' and '/usr/local/GNUstep/System/Library/Makefiles/GNUstep.sh'. And be sure the permission is correct.

If you log into the fault-safe session again, you should be able to run this script '/usr/local/bin/etoile' to launch etoile_system, which will then launch all other system-level applications. By this point, you should have a Étoilé session in GDM for Étoilé environment.

If you want to set up the Login.app, see 'Etoile/Services/Private/Login/README' for details.

Update: There is a summary of latest GNUstep/Etoile on Solaris.

The Road to CoreObject Part 2: Why Bother?

Posted on 30 July 2007 by David Chisnall

Since the last post, a lot of people have asked me 'why are you doing this? What advantage does it actually give?' In this post, I'll try to explain.

One Abstraction, Two Uses

What is a file? Over the last year, I've asked a number of people that, from computer scientists to technophobes. None has managed to give me a clear answer. The next question I asked is 'What is a document?' Everyone I asked gave me a clear answer.

From a user interface perspective, it's clear that a document is a better abstraction than a file. A file is a very convenient abstraction for operating systems; it's basically a virtualised block device with a simple text key (the path/filename) that can be used to uniquely identify it. It is not a good abstraction for users.

Files are used for two things:

• Storing a document.
• Publishing a document.

From a user interface perspective, these are very different tasks. Storing a document is not something that should ever need to be done explicitly. Raskin's first law states:

A program shall not harm a user's data, or through inaction allow the user's data to come to harm.

Everything I do to a document should automatically be stored if possible. In some situations, such as sudden power failure, some data loss is inevitable, but the program should do everything it can to minimise the chance of avoidable data loss. A simple corollary to this is that versioning information should also be stored. If I hit select all, delete, then I don't want the stored form of my document to be overwritten with an empty document. I want an undo feature, and I don't want this to be contingent on keeping the document in memory (select all, delete, {autosave}, power failure, panic).

CoreObject's serialisation function does this. You don't need to explicitly save a document. From the time an application tells CoreObject to manage the object graph representing the document model, you have the ability to replay every single change you've made to it (this actually works in the version in /trunk now, although it needs more testing).

While you don't have to save a document explicitly, you might want to tag it with some metadata. Some of this will be created automatically for all objects (creation dates, modification dates, etc). Some will be created automatically for certain object types (e.g. colour depth, word count, table of contents). Some can be specified manually. This will be indexed by the higher layers of CoreObject. These tags can either be assigned to a specific version, or to the latest version. You might tag a book you are working on with the book title, and also tag the version you sent to the proof readers, so you can jump back to that one to compare with the comments they gave you.

Publishing is a very different problem. When you publish a document, you typically don't want to include revision information, you want a snapshot. A few government agencies have been embarrassed in recent years by forgetting that Word Documents are intended for storing, not publishing, and include a lot of revision information.

How does CoreObject help with the publishing? Well, the current implementation doesn't (yet), but the plan is to integrate something like Apple's UTI (or, more likely, UTI itself). This is a type hierarchy supporting multiple inheritance that is orthogonal to the object hierarchy. Each compliant object will publish a number of types that is inherits from, such as rich text, or image. It will also support exporting its contents as each of these. For complex compound documents, the root document will simply query the enclosed components, and assemble a composite of images, text etc. Each object only needs to be able to export to something one layer up the type hierarchy. For example, a word processor might export as rich text, and the system would then convert this to text using a shared component.

What About My Friends

The other important feature of CoreObject is collaboration, which is central to the Étoilé vision. CoreObject's serialisation of invocations allows these to be sent across any kind of network connection. In 0.3, there will be a XML-over-XMPP system for this. This will stream changes between two (or, in theory, more) users as they are made. Some systems exist for doing this in very specific cases, such as SubEthaEdit for text and a few whiteboarding solutions for images. CoreObject will allow us to do this in the general case. Any document that works with CoreObject will be able to be shared in this way.

Because it only sends the deltas, this approach will scale to relatively large object types. Imagine something like a raw digital photograph. These can easily be several tens of megabytes. The changes made to them, however, are usually of the form 'alter the brightness level by 5%,' or 'apply this filter with these parameters.' These are not very big, and so once the photograph is initially shared, it can be tweaked in a collaborative fashion easily.

This is even true of video editing. Something like Apple's Final Cut does non-destructive editing. While the source footage is often tens of gigabytes, the project file is very small, since all it contains are instructions like 'take insert ten seconds from source file x at y in the timeline,' and 'cross fade for 10 seconds.' With CoreObject, we get this kind of non-destructive editing for free, and we also get the ability to collaborate on documents like this for free. We could have two people editing the same video on their own machines and having the changes automatically kept in sync. Once it's done, they export it as something like MPEG-4, and anyone can view it irrespective of whether they're using Étoilé.

Labels: CoreObject

Étoilé 0.2 is now officially released

Posted on 28 July 2007 by Yen-Ju Chen

Étoilé 0.2 is now officially released. See the full 0.2 Release Announcement for more information. There are a number of screenshots of this release online. A source tarball is available for download. Those preferring to use subversion should check out /tags/Etoile-0.2 from the repository. If you have any questions regarding this release, please post your queries to the Etoile-discuss mailing list or visit the SILC channel Etoile on silc.etoile-project.org.

First Steps into Étoilé and GNUstep Programming

Posted on 20 July 2007 by Quentin Mathé

If you are interested to contribute to Étoilé directly or by writing your own applications, I have compiled below some resources to learn GNUstep programming…

The best resource for Objective-C development with Cocoa/GNUstep frameworks is Apple documentation in my opinion. Cocoa documentation is now excellent, but don't forget GNUstep isn't always in sync with Cocoa and Carbon stuff isn't available on GNUstep platforms.

To begin the best is probably to read the following Apple guide: Cocoa Fundamentals.

On GNUstep website, visit Developer documentation area. From this documentation, you should read gnustep-make documentation. Take note it isn't really exhaustive and doesn't cover the new stuff brought by latest release 2.0. Base Programming Manual can be worth to read. It details a bit some GNUstep specific extensions like Documentation system, logging etc. and includes a very good overview of Distributed Objects with real examples.

GNUstep API references:

• Foundation
• Foundation Additions… This is the place where you can find documentation on XML support and Runtime utility functions.
• Foundation Tools… Important place where GNUstep Documentation system (aka GSDoc) with the related tool autogsdoc is explained.
• AppKit

Cocoa Dev Central is a good place to start with plenty of nice tutorials too.

Here are other very good guides you can read to dig into GNUstep development:

• Application Architecture
• Memory Management
• Document based Application
• Event Handling
• Coding Guidelines
• Objective-C Language
• Drawing

Many good resources are listed on CocoaDev and Stepwise.

If you are interested by books, check:

• Cocoa Programming for Mac OS X, I learnt Objective-C development with the first edition of this book :-)
• Coecoa Programming, this book is outdated but it's still the only one which covers advanced stuff decently

The Road to CoreObject Part 1: EtoileSerialise

Posted on 19 July 2007 by David Chisnall

The Road to CoreObject Part 1: EtoileSerialise

I've now submitted my PhD thesis and more or less wrapped up my Xen Book, so now I'm taking the summer off (apart from the odd article) to work on Étoilé before looking for some kind of job.

I've mainly been focussing on development of CoreObject, specifically the low level components required for CoreObject, this month.

Files Suck

Many of the ideas here come from a discussion between myself and Nicolas a few years back about how computers suck and how we can make them suck less (a recurring theme, that eventually led to the formation of Étoilé when it turned out that we weren't the only people having this conversation).

CoreObject is intended to be one of the foundation pieces of Étoilé. The current roadmap calls for an experimental version in 0.3, a stable interface in 0.4, and a completely stable version in 0.5.

What is CoreObject? Basically, it's a replacement for a filesystem as a programmer and user interface. Files (in the UNIX sense of the word) never were a good abstraction; an untyped series of bytes is no use to anyone. The operating system needs to deal with things like this, but programmers shouldn't have to.

We already have a much nicer abstraction than a file; the object. Unlike files, objects have all of the structure and introspection that we want in order to be able to interact with them programatically. In Étoilé, we want to treat everything as an object, and objects as first-class citizens.

Persistence

At the lowest level, CoreObject provides a persistence layer, and that's what the EtoileSerialise framework is for. It turns out that the Objective-C runtime stores the names and types of all of a class's instance variables, making it almost possible to serialise arbitrary objects without any extra code.

Why Almost? Because Objective-C isn't really a language, it's two languages that you are allowed to mix together in the same source file. One of these languages is a close relative of Smalltalk. The other is C. Anything that just uses the Smalltalk components of the language is easy, since the runtime stores all of the type information. Anything that uses the C component is impossible, because there is no runtime information at all. Most code that we are interested in lies on the boundary.

The EtoileSerialise framework doesn't serialise anything other than objects. Objects, however, have instance variables that can be C types. Some of these are easy. An int, or float, for example, has its type encoded in the object's description, and so we just need some special handlers for the various C types (of which there are not many). The same is true of static arrays. If you say something like 'int foo[20];' in an instance variable description, the runtime encodes the size of the array and we can retrieve that and transparently serialise the array. The same is true of structs, where the runtime stores the type for each field.

Dynamic arrays start to get a bit harder. A dynamic array in C is just a pointer. If you say 'int * foo' then the only information available to the serialiser is that foo is a pointer to an integer. It could be an array, or a pointer to an aliased intrinsic. If it's an array, we have no way of knowing the size. This is not quite true. On Windows and OS X there are extensions to the malloc() family that let us know the size of a block of memory identified by a pointer, but they are non-portable. This doesn't help us at all when we have something like 'int **' though, since we don't know if it's a 2D array, an index array, or something else. At this point, the serialiser just gives up.

Fortunately, Objective-C is a nice dynamic language, so we can fudge this slightly. For objects that use low-level components of C, we introduce an informal protocol that asks the object to manually serialise it. This takes the ivar name and the serialiser back end as arguments, and so anything the serialiser can do, this method can too. On the deserialisation end, two other methods are available for fixups. One is the converse of this, requesting that an object manually deserialise an ivar. The other is invoked with no arguments once an entire object graph has been deserialised. Note that the serialise and deserialise methods don't do any type checking for manually serialised things, so you can serialise an int as a C string if you need to. One reason you might want to do this is for an object wrapping a file, where the ivar would contain a file descriptor (an int). If you just store this and reload it, you will get nonsense, so you might instead store the file name and re-open the file on deserialisation.

There are two other pieces of the puzzle. One is named structures. Some structures need special handling, and it is a bit rubbish to expect the developer of every object that needs to handle them to know about this. Fortunately, the runtime system knows the name of structures that are used. To make use of this, you can register a function that handles serialisation of a named structure. The serialiser will then call this whenever a structure of this form is encountered. The other part is versioning.

OpenStep's NSObject already has a -version and a -setVersion: method. We make use of this with the serialiser by encoding the version with each serialised class (for subclasses, we encode the version of each class in the hierarchy with the instance variables inherited from that class). The manual deserialiser method takes the version as a third argument. If you change the instance variables of a class, it's easy to add support for deserialising the old version by implementing this. It is even possible to do this in reverse, by supplying a category on the old object that loads the new one, or even deserialise an object using an object of a completely different class using the poseAs: mechanism.

Versioning

We can version classes, but what about objects? It would be nice to have the revision history preserved. We do this by turning a model-controller-view trio into a model-(CoreObject proxy)-controller-view system.

With Objective-C, everything is an object, including the messages you send to objects (the equivalent of method calls in C++/Java). The combination of the message name (selector) and arguments is known as an invocation, represented by the NSInvocation class. Our COProxy object, or a subclass, sits between the model and controller and serialises every message that is sent to the model. This stores the complete revision history of the object. To reload any version of the object, you can just reload and replay the invocations. So this doesn't take too long, the COProxy object periodically serialises a copy of the object. Currently this is done every 100 messages. In future it will be configurable.

I mentioned COProxy subclasses. The reason for needing these is that Objective-C doesn't have the concept of 'const' methods, i.e. methods that are guaranteed not to affect the state of the object. We don't want to bother serialising these, so we will use a subclass for each class we might use as the principle class in our model to automatically pass these through.

We thus have saving, restoring, and versioning of arbitrary objects for free. Since we're greedy, we want more. Let's also have branching and merging. Branching is easy; we just define two objects with the same previous version. What about merging? Well, I think we can do this by re-playing the invocations from the two branches in an interleaved way. This will probably be done with a UI allowing the user to select which invocation should be run next.

Anything else? What about collaboration? Since we are serialising invocations, we can pass them over a network to another user, and they can keep their copy of an object in sync with ours. With a simple locking protocol, we can have bi-directional syncing. The serialiser and deserialiser are split into a front and back end, with the back end defining the storage format. At the moment there is a binary file format and a simple human-readable output-only back end for testing. An XML backend will be added too, allowing objects (including invocations) to be passed over XMPP (Jabber).

Oh, one more thing. This also gives us non-destructive editing of any arbitrary object type, from text through images to video, as long as we have an object encapsulating it. Excited yet?

Does it work?

So, what's the current status? Actually, pretty good. The code contains a few 32-bit x86isms that need fixing. Serialisation works for a lot of objects with no modification, and more with a little tweaking. Deserialisation is a bit less finished. Deserialisation of named structures is not finished, and neither is the special code-path for serialising invocations. The COProxy object works, and serialises invocations properly. It does not yet include a mechanism for re-loading them, but the example back end allows you to see that all the required information is saved.

Currently, the build system creates a test app, rather than a framework. This will be changed towards the end of the month, when the first alpha will be ready. The interfaces are still likely to change, however, so don't start integrating it into your code yet. Étoilé 0.3 will have a fairly stable version for everyone to play with and, hopefully, some of the higher-levels of CoreObject, which will deal with metadata, indexing, and type conversions too.

Idiot! Use Smalltalk!

Some people will be reading this and thinking 'this whole thing would be much easier in Smalltalk.' This is true. In Smalltalk it would be possible to write the completely generic version, and use the garbage collector to track any aliasing, etc. So, why don't we use Smalltalk? Smalltalk's a great language (ask Nicolas why, but only if you've got a long time to listen to the answer), and easy to learn; even small children can pick it up quickly. It doesn't, however, play nicely with other languages (or even GUIs), and there are not many people who know the language well.

Nicolas has described Étoilé as a 'pragmatic Smalltalk.' We sacrifice some of the nice features of Smalltalk, but gain the ability to make use of lots of legacy C code. Objective-C isn't quite as nice as Smalltak, although it's close, but we gain a lot more from the Objective-C frameworks than we lose from the language.

Labels: CoreObject, EtoileSerialise, shiny

Font Manager

Posted on 23 June 2007 by Isaiah Beerbower

I've been working on a new app for Étoilé: Font Manager. Font Manager is, as the name suggests, an application with which to manage fonts.

As you can see in the above screenshot, Font Manager already shows you your installed fonts, and presents samples of them. Planned features are: disabling/enabling fonts, previewing and installing fonts, and automatic conversion to the nfont format.

Update 8/7/07: New URL for image & thumbnail.

Jabber Update

Posted on 13 June 2007 by David Chisnall

There have been lots of changes in the Jabber code recently. The GNUstep version has had a bit of extra hacking to work around some GNUstep bugs (I'm going to have to have a serious look at the GNUstep implementation of NSOutlineView over the summer), and Jesse and I have been working on integrating µblogging and Jabber.

First, a quick introduction to the Jabber code. The /Services/User/Jabber directory in trunk contains an application, StepChat, and two frameworks. One of these handles XML parsing, the other handles XMPP. The application is a thin wrapper around the XMPP framework. Currently it works on OS X and GNUstep (although there are some GNUstep-related bugs remaining). Eventually, the code will diverge into two branches, one taking on Étoilé-specific features, the other remaining a stand-alone application for OS X and GNUstep-without-Étoilé users.

The framework uses a 'people' and 'identities' abstraction that is likely to show up in other parts of Étoilé. It understands that a person can have multiple identities (e.g. a work Jabber ID, a home Jabber ID, and an MSN Messenger account) and treats the person as the important one. When you have a conversation, it is with a person. You can swap between talking to any of the person's identities transparently. If you have a conversation active then it will automatically pick the most relevant identity to talk to, so if a person moves from one machine logged in with one account to another logged in with another (or the same account with a different resource) then it will switch over to the correct one automatically. It maintains the same abstraction in the roster, which has a three-tier hierarchy of groups, people and identities. Of course, if you want to talk to a specific identity, you still can.

On to the µblogging stuff, Jesse has been playing with a Jaiku µblog recently. Since µblog entires are very similar to status messages in an IM application, it made sense to try integrating them. A couple of changes were made to StepChat for this. First, a new status message UI was created, adding a text field to the roster so it's easy to set a message (the old UI was a quick hack), and a 'presence log' UI was added to give an aggregated view of messages set by others. Next, the presence code was modified to send a distributed notification whenever the presence was updated.

Jesse and I put together an example tool which makes use of ththis and pushes updated presences out to Jaiku. Now, when you set your presence in StepChat, it will set your presence and push it out to Jaiku. Anyone who wants to bother could add support for Twitter fairly easily too.

Build individual Étoilé component

Posted on 6 June 2007 by Yen-Ju Chen

Étoilé project comes with many components, but it is not necessary to build everything if you do not want to use every component. You can just go to individual subdirectory and build a component with 'make; sudo make install'. Most of applications are under /Etoile/Services. Components in User subdirectory are for regular tasks. They are often launched by users when needed. Components in Private subdirectory are usually staying through the whole session after login.

If you choose to build Étoilé components individually, there are two levels of dependencies. First, some applications depend on other frameworks or libraries. If a building fails, you should be able to find out which framework or library is needed. Second, some applications depend on other applications. GNUstep provides excellent mechanism for inter-process communication and Étoilé takes an advantage of it. In that case, the component will build and run, but may not behave as expected. An example is that AZDock depends on Azalea.

If you met any problem of building or running an Étoilé component, you can ask people on Étoilé SILC server at silc.etoile-project.org. Here is a list of SILC clients to use.

Progress in May

Posted on 3 June 2007 by Yen-Ju Chen

Besides fixing bugs and preparing LiveCD 0.2, we have new applications in -stable:

DEPENDENCIES = RSSKit


-include ../../../etoile.make

RSSKit = Frameworks/RSSKit

- (void) applicationWillFinishLaunching:(NSNotification *)aNotification
{
  Display *dpy = (Display *)[GSCurrentServer() serverDevice];
  int screen = [[NSScreen mainScreen] screenNumber];
  Window root_win = RootWindow(dpy, screen);


/* Listen event */
  NSRunLoop     *loop = [NSRunLoop currentRunLoop];
  int xEventQueueFd = XConnectionNumber(dpy);

[loop addEvent: (void*)(gsaddr)xEventQueueFd
                        type: ET_RDESC
                     watcher: (id)self
                     forMode: NSDefaultRunLoopMode];

}

- (void)receivedEvent:(void *)data
                 type:(RunLoopEventType)type
                extra:(void *)extra
              forMode:(NSString *)mode
{
  XEvent event;


while (XPending(dpy))
  {
    XNextEvent (dpy, &event);
    /* Intercept event here /
    switch (event.type) {
      case Expose:
      case DestroyNotify:
      case PropertyNotify:
      case FocusIn:
      default:
      / Go back to GNUstep */
      [server processEvent: &event];
    }
  }
}

  /* Listen to root window for window closing and opening */
  XSelectInput(dpy, root_win, PropertyChangeMask);

defaults write Camaelon Theme Nesedah

Now, the system is ready to go. Azalea is the window manager and EtoileMenuServer is the menu bar on top of the screen. To start, use 'openapp Azalea.app' and 'openapp EtoileMenuServer.app' from a terminal emulator.

Étoilé: What's going on? (July 2006)

Posted on 8 August 2006 by Nicolas Roard

Another summary of the svn activity :-)

Well, we are already the 8th... so I'll simply sum things up to today :-)

Yen-Ju did some updates on Azalea, as well as updating the Io VM; Saso fixed MenuServer to use the proper defaults key for suppressing the App icons tiles that GNUstep generates. Quentin imported Oniguruma 4.2.1 (released on 31/07/2006), the regular expression engine used in LuceneKit, on the repository (in the Dependencies folder) -- easier for people to get it that way, as it's apparently slightly exotic on some operating systems / distributions.

Günther continued to work on the new RSSKit -- more refactoring, some fixes, multithreading code, and doxygen documentation. Grr (the rss reader) then also had its share of updates and the current version should work with the current RSSKit. Günther also worked on DictionaryReader (the dictionary application), and did some refactoring and a few UI improvements (avoiding unnecessary line breaks, an edit menu item to copy/select all). Finally, he added icons to the Monitor and the Look and Behaviour preference panels.

Quentin fixed a few compilation issues in ServicesBarKit, and fixed the gorm file for the Hardware pref panel (it was apparently corrupted in the cvs to svn transition).

Quentin also worked on System -- the Étoilé main system process, which plays the role of an init process and a main server process (taking care of starting/stopping and monitor Étoilé core processes, possibly restarting them if they die). So System is now a real server and daemon, Distributed Objects support was added, it is now compiled by default when compiling Étoilé, and there is even some early documentation. Also, an "etoile.desktop" file was added -- which means you can now easily start étoilé (azalea, menuserver..) directly from a login panel such as GDM/KDM :-) System is still in early stages, but looks promising.

Finally, Quentin also added two nice postinstall scripts:

• setup.sh: install Étoilé transparently
• setdown.sh: revert the effect of setup.sh

Setup.sh nicely install everything you need, in the proper place: EtoileWildMenu bundle, the Camaelon theme engine, Nesedah (our default GUI theme), System, etc.

Étoilé: What's going on? (June 2006)

Posted on 4 July 2006 by Nicolas Roard

A summary of what happend this last month on the subversion repository...

There wasn't as much activity compared to the previous month -- I'd guess for various reasons (end of academic year, exams, holidays, etc.) ;-)

Most of the activity was concentrated on updating the code, fixing bugs, reorganization, or refactorization.

The only addition to the repository is a new menulet (MenuServer plugin) by Yen-Ju Chen, WorkspaceSwitcher, which, like indicated by its name, allows you to switch between virtual desktops provided by Azalea.

Günther Noack worked on RSSKit -- more unit tests, some refactorization (a new class hierarchy for RSS types -- each class contains code to parse documents of that type). Thus the current version is deemed experimental, but bode well for the future ;-)

He also worked a bit on DictionaryReader, fixing some bugs ('bad dictionary name' bug in particular).

Saso Kiselkov added Solaris OS support code in the MenuServer, as well as fixing some bugs and reorganizing the makefiles.

Quentin Mathé worked on the ServicesBarKit to prepare its future integration with the MenuServer (Basically, the ServicesBarKit let applications provide "tray icons" -- currently displayed in a separate window, and in the near future it will thus be integrated in the MenuServer so those tray icons will appear directly in the menu; similar to OS X). He also fixed a few things in the compilation of process, added some better log statements in MenuServer, and reorganized a few things in EtoileExtensionsKit.

Azalea (our window manager) was updated by Yen-Ju, Quentin also improved the build process to better deal with the XWindowServerKit dependency.

Finally, Yen-Ju continued to work on the Io bridge, adding ObjC constants and importing an example (Developer/Examples/IoExample/example.io) on how to use the Io Bridge to create a simple gui application.

To sum it up, lots of "internal work" this month... :-)

People like screenshots...

Posted on 18 June 2006 by Günther Noack

...so here's a shot of the current version of DictionaryReader.app, which is an application to look up words that also integrates nicely in your personal GNUstep environment by providing a service.

The application queries multiple free dictionaries via the Dict protocol and allows you to easily browse through their content using a web-browser like interface with "Hyperlinks" and a "back"-button.

Étoilé: What's going on? (May 2006)

Posted on 28 May 2006 by Nicolas Roard

While we do not seem to write a lot of news on the blog, étoilé is far from dead: there is a lot of activity going on on the repository and the mailing list. Let's see what happened during last month...

First: the existing stuff...

Yen-Ju did a lot of work on Azalea, our X11 window manager, doing its share of fixes and improvement (like working nicely with borderless windows or compiling on OSX), and he is on top of that working on a branch using directly appkit for drawing...

Günther fixed some issues with DictionaryReader and applied a patch from Chris Vetter to add a service -- so you can now select some text in any application, go in the service menu, click on "look up in dictionary" and voila! a window will open containing the definition of what you selected. Very convenient isn't it ?

Saso improved quite a bit the MenuServer, also applying various patches from Chris Vetter and David Chisnall. The info panel now displays properly informations about the machine/os, and a nice animation was even added in the about panel.

To give you an idea of where we are, here is a screenshot showing Azalea plus the MenuServer and an application (TalkSoup, an irc client), with the Nesedah theme:

On the frameworks front, Quentin is working on the ServicesBarKit to integrate it with the MenuServer; Yen-Ju updated LuceneKit to the 391895 java version, and Günther added unit tests in RSSKit.

In addition to improving what is on the repository, new code also appeared :-)

Yen-Ju imported the first version of XWindowServerKit -- a simple framework that applications can use to "talk" with X11 in order to set things like the available space, the current desktop for a window, etc. He also imported a patched version of the io bridge and added StepTalk support -- so you can now use io to develop apps and/or as a StepTalk language... nice !

Finally... I imported the Nesedah theme on the repository, and fixed a couple of things. Quentin worked on the Documentation, specifically a technical overview of étoilé, an user experience overview, and a core object overview.

So as you see, things are moving fast (special mention to Yen-Ju !) and I will try to regularly post some news about what's going on the repository...

New Azalea based on OpenBox3

Posted on 5 April 2006 by Yen-Ju Chen

A newer version Azalea is ported from OpenBox3. It currently supports GNUstep window style and window level. It is avaialbe under trunk/Etoile/Services/Private/Azalea/.

Étoilé presentation at the Fosdem

Posted on 5 March 2006 by Nicolas Roard

We did a talk in the gnustep devroom at the FOSDEM 2006 presenting Étoilé.

I uploaded the slides and the video. Check also the page about the other talks!

Azalea

Posted on 15 February 2006 by Yen-Ju Chen

Azalea is derived from WMaker released by Enrico Sersale, which is based on WindowMaker 0.92. The goal is to replace WINGs with GNUstep while keeping it work as WindowMaker. It is still experimental and may not work out in the end.

I put Azalea in SVN in hope people can help here and there. It is in Etoile/Service/Private/. You can run 'configure' before 'make', but 'make' should work. It depends on libwraster, but not WINGs. Some parts of WINGs are in the source code already. It should works as WindowMaker. No major function is broken. If you found one, it is a bug to be fixed.

Besides the TODO in README, here is the list of tasks in my mind:

1. Switch Panel (switchpanel.m):

   This is only called from cycling.m and is an eye-candy. Therefore, it should be easy to work on. Probably a NSPanel will do. But you might need to deal with both Xwindow and NSApplication event. It is triggered by key press through Xwindow, then once the switch panel (NSPanel) shows up, it is handled by NSApplication. Currently it does not work because the pixmap files are missing. But the Alt-Tab still work to switch focus among windows.

2. Contextual menu:

   The main menu is a NSMenu already (right-click on the empty space). There are still four menus needed to be rewritten: dock menu (icon) on the right), application icon and miniwindow menu (icon on the bottom), clip menu (icon on the top-left corner), window menu (right-click on the window title or Ctrl+ESC). Some menu will call a dialog for settings (icon chooser is rewritten already). It can be rewritten as NSPanel later. You can check the OpenRootMenu() in rootmenu.m for main menu. It should be better to replace these menus with contextual menu from GNUstep.

3. Array and dictionary:

   WINGs has its own WMArray and WMHashTable. It may be nice to replace them with NSArray and NSDictionary, or NSMapTable and NSHashTable. But it is used everywhere, and may not be worth to rewrite at this stage.

4. Notification:

   Same as above.

5. Dialog:

   Beside the dialog called from menus mentioned above, there are still two dialogs: message and crash dialog. I already partially rewrite the message dialog. But here is the trick: the dialogs may be called outside NSApplication run loop, especially when Azalea starts up and crash. Therefore, it may not be safe to use NSPanel for these situation.

6. Property list:

   Most of the property list are already rewritten. The rest is in winspector, which will be replaced by WMWindowInspector. It is work-in-progress. Once it is done, we should be able to remove property list from WINGs.

7. General clean-up:

   There are some general functions here and there, which can be easily rewritten in ObjC. They can probably be collected into a WMUtilities class.

8. Localization:

   Replace all @"..." with _(@"...") should be fine.

9. Remove most of WINGs:

   Some parts of WINGs may not be able to replaced easily, but others should be clean up.

PreferencesKit Documentation... well, it's ready to taste!

Posted on 14 February 2006 by Quentin Mathé

After several days spent at fighting with autogsdoc (GNUstep documentation system), I have finally managed to output PreferencesKit documentation in html. I'm happy to stop fiddling with it, now I understand how it works and its shortcomings.

Basically in my case it was very important to pass both headers and implementations, to have implementations following headers in autogsdoc paramaters… An another important point was to turn on both -Verbose and -Warn options and to never forget autogsdoc may fail silently without reporting any errors. When autogsdoc is stumbling on constants declared at several places, you may just process your source code files two by two (.h and .m), that's the last trick I used to output the current PreferencesKit Reference.

With some key points highlighted in documentation, I'm pretty sure it could be easier to tackle autogsdoc. I suppose I have to report the weird experiences I encountered with it ;-)

Back to the crux of the matter now… I have updated PreferencesKit page with various links :

• PreferencesKit Reference
• PreferencesKit Class Diagrams

both recently uploaded on Étoilé website.

Preferences Applications Early Screenshots

Posted on 30 January 2006 by Quentin Mathé

Three Étoilé Preferences applications are currently planned:

• Look & Behavior
• Hardware
• Network

Another one to handle user accounts will probably be needed.

Here are some early screenshots (presenting both Hardware and Look & Behavior currently under development).

You can download these applications from Étoilé repository (they are located in trunk/Etoile/Services/User). Take note you will have to compile PreferencesKit first. If you build the whole repository, the build system will take care of that.

These applications are based on PreferencesKit and currently contain no real functionnalities. Their User Interface is still to be discussed and refined. Preferences Utilities are a very important point in my opinion for Étoilé, because both KDE and GNOME lack a simple yet efficient User Experience on this matter.

It is planned to use GNOME system-tools-backend (written in Perl) as Preferences applications backend to handle various OS deployment. More about System Tools Backend here.

PreferencesKit Preview

Posted on 25 January 2006 by Quentin Mathé

I'm happy to announce first PreferencesKit preview I have been working on until recently (with some help from Yen-Ju). It is based on GSSytemPreferences initial code written by Uli Kusterer. Thanks Uli :-)

PreferencesKit is a framework which provides various features to build flexible Preferences-like window in any GNUstep or Cocoa applications.

Key features:

• Generic plugin model/schema
• Plugins registry mechanism (with search paths) that can be specialized
• NSPreferencePane and Backbone PrefsModule support
• Possibility to choose between various presentations like toolbar or table view (where panes are listed)
• Possibility to extend PreferencesKit with new custom presentation
• Facility for Preferences-only applications development (just set up a main nib file, list plugins to load in a plist and link PreferencesKit to have an application ready to use)
• Cocoa compatibility (Xcode project bundled)

Here are PreferencesKitExample screenshots running on both GNUstep and Cocoa.

You can download the framework from Étoilé repository. You can browse the code with this link.

Most of PreferencesKit features are demonstrated in PreferencesKitExample, it is worth to take a look at it.

GNA! Hotspot

Posted on 16 January 2006 by Nicolas Roard

On a regular basis, Gna! people pick and interview one project to highlight the software and personalities that drive Libre Software community. And for the 8th hotspot.. it's us!

Link to the interview: https://gna.org/forum/forum.php?forum_id=1046 (an html version is also available)

New Site Theme

Posted on 16 January 2006 by Jesse Ross

The etoile-project.org site has a new theme -- Flora! The theme should be going live soon, but in the mean time, if you have a wiki account and want to test drive it, follow the instructions below:

• Point your browser here
• Click 'Skin', select 'Flora' and then 'Save Preferences'

You many notice some bugs here and there -- we'll be ironing those out throughout the week.

Build LuceneKit straight

Posted on 22 December 2005 by Yen-Ju Chen

Some people asked about how to build LuceneKit from scratch. Here is the instruction:

1. GNUstep-make:

   ./configure --prefix=/usr/local (or other place like /sw/local) make sudo make install

2. Oniguruma:

   ./configure --prefix=/usr/local/ (or other place like /sw/local) make sudo make install

3. OgreKit in Etoile. make sudo make install

4. LuceneKit in Etoile. make sudo make install

   You probably need to use -lLuceneKit -lOgreKit to link the libraries in your project.

AddressBook with LuceneKit support

Posted on 15 December 2005 by Yen-Ju Chen

A new version of AddressBook with LuceneKit support can be download here. See README.LuceneKit for details. It creates a new group with search result and supports boolean and prefix query.

This is just a showcase for LuceneKit.

WildMenus

Posted on 1 December 2005 by Nicolas Roard

WildMenus is a nice gui bundle to have horizontal menus with GNUstep apps. As it could be an option for Etoile apps (when using with small screens, horizontal menus are nice), and as I wanted to sligthly modify it to be a better Camaelon companion, I just imported it (with michael's green light) in Etoile's repository, in /Etoile/Bundles/WildMenus. That way people can have a simpler way of getting WildMenus, and we can use subversion to improve it.

Track... New Étoilé List!

Posted on 30 November 2005 by Quentin Mathé

A new Étoilé list has been set up, it is called etoile-track. It is configured to receive notifications sent by Bug, Patch and Task trackers each time one of their item is updated. I would have preferred to have such list named etoile-support or eventually etoile-bug, but none of them were accepted by GNA system (strangely, they are reported to be reserved to avoid potential conflicts).

If you are involved in Étoilé or just interested to track Étoilé development, you should subscribe here.

For now, this list isn't mirrored on Gmane.

Last note, it is widely public… It is not needed to be a subscribed member to post on this list. I mean any mails about support issues are welcome on it, by the way why not call it officially Étoilé support list!

Finally we have our Build Guide

Posted on 30 November 2005 by Quentin Mathé

Finally today, I have just added this lengthily promised Étoilé Build Guide in form of two README and INSTALL files to our repository. INSTALL will probably need some adjustements, Nicolas has already improved it a bit.

Étoilé Build Guide is available on Étoilé web site too, take a look here.

Now it should be possible to try building this whole repository to know on which platforms Étoilé is really supported.

Subversion!

Posted on 10 November 2005 by Nicolas Roard

Quentin announced today that the Subversion repository is now set up on gna.org, with the previous cvs history imported.

SVN client or SVK are appropriate. Quentin wrote a small tutorial/introduction to SVK, which brings distributed capacities to SVN.

OgreKit compilation was also fixed yesterday by Quentin, and last Camaelon changes (done after the cvs history was saved..) were committed too.

Here and Tomorrow

Posted on 9 October 2005 by Quentin Mathé

Étoilé News is alive now... Well, you probably want to know more about Étoilé, may be you should move there: http://www.etoile-project.org