Edument blog – Development and mentorship

Nordic Perl Workshop 2011

The Nordic Perl Workshop is an annual event that the Nordic countries take turns hosting. This year’s workshop will take place in Malmö, Sweden. The companies Edument and Informator have graciously agreed to sponsor the event.

Perl is a modern scripting language with a vibrant developer community, active core developers, and just massive amounts of modules. Perl source code drives companies, web pages, IRC bots, and parts of the Internet infrastructure. However, much has happened since the first release of Perl in 1987; as a way to celebrate that, the theme of the workshop this year is “Perl in 2011″.

Edument is a newly started company in the Skåne region in southern Sweden, providing consultancy, education, and development mentorship. Informator is a well-established provider of education in the Nordic countries. Their joint sponsoring of the event means that the attendees will have a lovely venue close to the station, with a nice view of the Malmö harbour area.

The event is arranged by Jonathan Worthington and Carl Mäsak, both of them Edument employees who also happen to be core developers in the Perl 6 community.

May 10, 2011 at 12:19 am carl Leave a comment

Trying out CQRS and Event Sourcing for a day

Last Friday, Jonathan Worthington and I (Carl Mäsak) decided to get our feet wet with CQRS and event sourcing. The toy project we settled on: a simple but realistic web site for two-player board games.

In this post, I summarize how things went.

Architect meets domain expert

Since there were only the two of us, I took the role of the domain expert, and Jonathan took the role of the architect. He expertly teased a model out of me. We arrived at two aggregate roots: Player and Game. Easy enough.

Design: commands and events

Using an inept but sufficient schema tool, we drew up the commands and events we figured we needed. These were as follows:

RegisterPlayerCommand
ActivatePlayerCommand
InvitePlayerCommand
AcceptInvitationCommand
RejectInvitationCommand
(no StartGameCommand)
PlaceStoneCommand
SwapPlayerColorsCommand
ResignGameCommand
TimeOutGameCommand

PlayerRegisteredEvent
PlayerActivatedEvent
PlayerInvitedEvent
GameStartedEvent
InvitationRejectedEvent
StonePlacedEvent
GameWonEvent
PlayerColorsSwappedEvent
GameResignedEvent
GameTimedOutEvent

For each command and event, we took a moment to model through what data we needed to send along. It gave us an appreciation for one of the ways in which commands and events differ: on the inside.

There was a moment of joyful insight as we realized that we had gotten this far into the design of the system and not once talked about state. Quite a
refreshing change.

Being the one with the “domain expert” knowledge, I kept unwillingly slipping back into the role of the client. Otherwise we’d have gotten some things wrong, which wouldn’t have shown up until the next “meeting with the client”. Jonathan remarked: “There’s got to be a lesson in here somewhere.”

(Afterwards, we’ve changed two things in the above model: we eventually realized that we would need an InvitationAcceptedEvent after all. The reason we originally figured we’d be able to do without it is that we noticed that it would fire off a GameStartedEvent, and that would be enough. But no, it needs to fire off both, otherwise the Invitation would still be open. The other thing we realized was that a better name for InvitePlayerCommand and PlayerInvitedEvent would be MakeInvitationCommand and InvitationMadeEvent. That way, all the commands and events contain in their names the aggregate that they are acting on — which makes a lot of sense.)

Commands, events, test framework

We wrote our first test, and the necessary classes to go with it.

Infrastructure

Our goal now was to get enough of the system running for our first test to fail. That took a few hours, partly due to the fact that we were figuring out how to fit everything together.

The wiring is like this: The test contains a ‘given’ list of events, a ‘when’ command, and a ‘then’ list of events or an exception. The test fixture creates an aggregate root to do the testing on, and loads it up with the events from the ‘given’ part. Aggregate roots have a special flag on the apply_event method for applying events without having them register as changes to be committed. That’s all the concession to testing that’s needed. Quite neat.

The test fixture then sends the command to a bus-like thing. This triggers the right command handler, which does the required validation and then calls a method on the aggregate. That’s the command part of things.

Now, the method on the aggregate is just a thin wrapper for applying an event. The event is mapped through a lookup table (our workaround for the lack of method overloading in Perl 5) to an apply-event method. Note that on the way, we visited the same apply_event method as when we prepared the aggregate with the ‘given’ events. This time the generated events are saved, though… and that’s exactly what we’re then using to check against the ‘then’ events. (Or, if we got an exception, the test fixture captures that and compares it with whatever was expected.)

It’s quite a simple system, though it took us a few hours to understand and get running. Still not too bad for our first attempt.

Getting the first test to pass

Trying to get the test we’d written at the beginning of the day to pass, we realized that we were still missing one component: a repository to store the aggregate in while we were testing it. We settled on writing a test repository, with a total capacity of one (1) aggregate.

After that, things fell into place quickly. We got our event wired up, and the test passing. Thus, we entered into the next phase…

Ping-pong pair programming

By the looks of the commit log, that’s where I became unconscious and Jonathan kept on hacking. But what really happened is that we paired up over Jonathan’s keyboard and started hacking in earnest.

Ordinary pair programming has a “driver” and a “navigator”. In ping-pong pair programming, the idea
is for the two people to alternate by taking turns writing a test for the other to implement. This was the first time we tried that, and it went very smoothly. Definitely something to try again. In regular pair programming, the navigator can sometimes doze off. But doing things this way, both of us were engaging with the process of writing code and tests, even when we weren’t in the role of driver.

We got through eight such cycles of ping and pong. At this point, things were really effortless: all the groundwork was already made, and now that we were finally implementing state in our aggregates, there were no longer any obstacles left. A very weird feeling; the aggregate was its own little world, merely responding to commands and events as they came flying by. Coding was effortless, not least because we managed to time it with the Ballmer Peak. Mmm, beer.

We surprised each other a bit by turning what appeared to be quite tricky tests into excessively simple bits of implementation. Things generally required less wiring up than we expected. (Again, because object state wasn’t the driving component, leaving us free to structure the innards of an aggregate any which way we wanted.)

One thing we also discovered is that we generally had to write fewer tests than “usual”. Each new test covered a bit more ground than we expected, and we often didn’t bother to write a test because we already knew it was going to pass. We’re not sure whether that’s (a) a good thing, and we shouldn’t worry, (b) a bad thing that’s going to cost us in the future, or just (c) a sign that we knew too much about the implementation. Guess more practice with this way of testing will tell.

All in all, a happy first day with CQRS and event sourcing in actual practice.

Attribution

go is Copyright © 2007 by luisbg.

The author

Carl Mäsak has a passion for software and software process. He works as an architect and programming mentor at Edument. He likes to work on healing systems in need of an architecture, or to help introduce order into a chaotic domain. On his spare time, he rides a bike, cooks food, and writes music. Not necessarily all at the same time.

Upcoming CQRS events

Edument will arrange a Software Architecture – Community Day in Malmo, Gothenburg and Stockholm.

Edument will also host a CQRS evening event with Swenug.

May 4, 2011 at 2:48 am carl Leave a comment

Fitting AOP into the paradigm jigsaw

As one of the core contributors to a compiler for a new language, I often find myself needing to explain new concepts and ideas to other developers. Over time, I’ve gradually found an approach that tends to work. First, where does the concept fit on the language feature or paradigm map? Second, what is the problem that it is aimed at, and how are other perhaps more familiar solutions weak? And then — with that critical foundation laid — what does the new concept actually do to help? Recently, I was asked to speak about Aspect Oriented Programming at the .Net Community Day. I’ve known about AOP for years and discussed it with developers before, but this was a nice opportunity for me to spend a while thinking about how best to go about explaining it. So I set about coming up with the AOP answers for my three questions. There are a bunch of titles that we tend to put before the word “programming”. “Object Oriented Programming”, “Functional Programming”, “Declarative Programming” and “Generic Programming” are just some examples. They’re all paradigms, but the amount they try to answer differs. The first three I listed — OOP, FP and DP — will happily jostle for pride of place in a language, seeking to be its “core paradigm”, and posing a challenge to language designers who see value in all of them. Of course, some languages do decide to just follow a single paradigm: regexes are happily declarative, and Java has admitted little into the language that isn’t squarely inside the OO box. However, most of the popular and expressive general purpose languages out there today embrace multiple of these “core paradigms”, recognizing that solving every programming task with a single paradigm is like doing every garden task with a lawnmower. At the same time, there are paradigms that simply seek to deal with a very particular set of issues. Generic programming is one of them: it deals with the situations where we want to use the same piece of code with different types, in a strongly typed way. This idea actually cross-cuts the “core paradigms”; generics are simply a case of parametric polymorphism, which one finds in functional languages. This is the kind of place that AOP sits. It’s not pitching itself as a successor to — or even a competitor of — any of our familiar core paradigms. Instead, it takes a problem and proposes an approach for solving it. So what’s the problem? In any complex code base, there are cross-cutting concerns — things that we need to do in many places that are incidental to the core logic. For example, logging and exception handling and reporting are generally things that a piece of code does in addition to the task that it really exists to perform. Of course, we factor as much of the logging and reporting code out as we can, but generally we find ourselves doomed to repeat similar-looking bits of cross-cutting logic in many places. Even if it’s just a call to the logger, or a catch block, it ends up duplicated. Duplicated code is generally a bad sign. For one, it inhibits future refactoring; if we decide to switch logging library, this is now a change in many places. Since we have mentioned the logging library in many places, we have many pieces of our code that are closely coupled to it — something that goes against our general preference for code that is loosely coupled. We also run into challenges when we want to install new cross-cutting concerns into our software — there’s so many places to change! Aspect Oriented Programming is aimed at these kinds of challenges. It introduces the concept of join points — places in our code that we could add some functionality — and advice — the functionality that we want to incorporate. It provides a mechanism to write the cross-cutting functionality in one place and have it “inserted” into the join points. A common reaction is, “wow, magical”, but if we cast our eyes back over programming history, it’s really not so surprising. Once upon a time there was just assemblers. Repeating regular sequences of instructions was tiring, so macro systems were created. These “magically” got substituted for a sequence of instructions, with some parametrization thrown in to make them more powerful. A little further on, named and callable subroutines allowed for real re-use of code. We raised the abstraction bar: “you don’t have to worry about what this bit of code does, just call it like this”. Then came OO. Here things got really magical: invoking one of these method things leaves you with no idea what bit of code is going to be called! It could come from that familiar bit of code you wrote yesterday, or it could come from a subclass that’s going to be written in 10 years time. Over the years, we’ve evolved ways to manage complexity in software, many of them hanging off increasingly non-linear or loosely coupled flow of control. We’ve coped with the increasiging complexity in our paradigms and programming style, because they more than pay themselves back by decreasing complexity in the software we produce using them. AOP’s concept of join points and advice is giving us a little more conceptual overhead in exchange for a more maintainable and less coupled implementation of our cross-cutting concerns. Looking to the future, I think it’s important to remember that where we are with any paradigm today is almost certainly some way from the ideal. We’ve learned a lot about object oriented programming over the decades that it has existed, and continue to do so. Multiple inheritance has generally been marked down as a bad idea, and the emergence of traits in Smalltalk (also known as roles in Perl 6) has made some — myself included — question whether we’d be better off dropping inheritance altogether in favor of flattening composition. AOP has only had 15 years of lessons learned so far, and I’ve no doubt we’ve got a long road to walk from here. However, that being the case shouldn’t stop us embracing AOP today. After all, the ongoing journey of the object oriented paradigm isn’t something that stops us from using it. The critical question is, “does this help me deliver higher quality, more maintainable software” — and I think there are a bunch of use cases where today’s implementations of AOP help us to do so. About The Author Jonathan Worthington is an architect and programming mentor at Edument. He works to help development teams improve their software architecture and work more efficiently, following best practices. He is also one of the core developers of the Rakudo Perl 6 compiler, where his work focuses on the implementation of object oriented language features and the type system. In his free time, he enjoys hiking in the mountains, good beer and curry.

March 2, 2011 at 2:00 pm edument Leave a comment

Product Predictions

Using Product Predictions to increase sales
One of the areas I’ve been working on since joining Edument is product prediction engines. In product prediction, we take a set of historical data that indicates what products a customer was interested in, and use it to suggest products that current customers may be interested in. That may be based on the customer’s own purchase history. Often we also want to use the data to compute a set of similar products to the one that the customer is currently viewing – something that works with new customers who we have no historical sales data about.

How a predictor engine typically integrates with an existing database-driven website

For example, if a customer is looking at a travel guide to Russia, from the historical sales data we may see that other customers who purchased this travel guide also tended to purchase a Russian phrasebook or a guide book about neighbouring Ukraine. “Gee,” says the customer. “I guess a phrase book would come in handy too, so I know my borsch from my babushki.” And so you get an extra sale.

Data Models
The key thing you need to get started with product prediction is a data set. Generally, the larger the better, although extremely large data sets may lead to slow predictions. This is something you will have to assess in your particular use case. To give you an idea of what’s manageable, though, I’ve been comfortably churning out predictions in under 10 milliseconds for data sets containing millions of past purchases. Granted it took me a little effort to achieve this kind of performance, but it’s within reach.

There are two types of data models: boolean models and ratings models. In a boolean model, either a customer has a connection with a product or they don’t. For example:

• They either purchased the product (and have a connection with it) or they didn’t purchase the product (and thus don’t)
• They either wrote a review of the product or they didn’t (note that it could be argued that even if the user wrote a crappy review of a product, they still had an interest in it, so we just treat the presence of a view as a connection)
• They either viewed the item or they didn’t (if you don’t have loads of historical sales data, you could data-mine web logs and try and get some connection data based upon which products a given site visitor viewed, on the basis that visitors to a site will generally tend to view related products; here, site visitors serve the place of customers in the data set).

An alternative is to have a ratings based data model. For example, we may have a large number of product ratings given by customers, on a scale of 1 to 5, where 1 is suckful and 5 is awesome. With a ratings based model, a customer giving a product a rating of 5 implies a stronger connection than a rating of 1. The prediction process then considers these values. Note that ratings are not the only possible source of data for using such a model; you could also consider:

• How many purchases a customer made of a product, if you have a range of products that often lead to repeat orders. For example, if you’re selling beer online, then a customer may have purchased Hobgoblin 10 times, but (understandably) only purchased Carling once.
• If mining web logs, you could consider number of visits to a product page as being your kind of “rating”

Your choice of boolean or rating model will directly influence the prediction algorithm that you use, since some algorithms only work with one model or the other. Which you opt for largely depends on what type of data you have, though as noted even if you have ratings data you may just like to abstract it down to “did the customer rate the product or not”. A boolean model certainly has the benefit of simplicity, and I’ve had very good results using such a model. If you have resources, then trying multiple algorithms and data models and using A-B testing would, of course, give you a far more scientific answer.

User-based and Item-based Predictors
Once you have a data model, there are two general types of predictors that you can use it with: user-based and item-based.

Improving Predictions
In some cases, you may have extra information that you wish to use to influence product prediction. For example:

A user-based predictor works by building a “network” of users with similar interests, and then making recommendations based on that. That is, given our current user, we can work out the other users nearest to them in the network and give recommendations based upon what those other users purchased. This depends on having some current historical data for the current user. We generally need to pick two things when working with a user-based predictor: a user similarity algorithm that computes the similarity of two users, and a user neighbourhood algorithm that builds the network of most related users. A side-effect of this is that we also win a way to compute “users with similar interests”, which for some cases may be useful.

An item-based predictor works based upon item similarity. To work, it also requires you to select an item similarity algorithm, which determines how similar two items are based on the overlap in customers purchasing them (depending on the algorithm, it actually can make transitive reasoning rather than requiring immediate overlap). To produce recommendations for a user, we go from the items they purchased and find most similar items to all of those. However, the big win from this approach is that we can compute most similar products. This is extremely useful because it gives us a way to make predictions for a product that is currently being viewed.

One additional benefit of an item-based predictor is that they allow for more pre-computation. The reason I could compute most similar products in 10 milliseconds was actually because I could, off-line in a batch job, pre-compute tens of thousands of set intersections that would be used in producing the results rather than computing them on the fly. Thus if you have time constraints or a large data set, item predictors are going to scale better.

• You may want to filter out products that a customer has already added to their basket from the prediction results; they already decided to buy it, so it’s best to show them new things they may also wish to buy.
• If you have some products that are not “family friendly” then you may wish to filter them out. Just because a many parents of kids who like Thomas The Tank Engine also happen to have purchased horror movies doesn’t mean they are suitable recommendations.
• You may want to suggest products in a given price range, which will take the customer over some threshold where they get a volume discount, or can use a voucher, or get free shipping.

An additional tweak can involve re-scoring the predicted items. Prediction engines not only find products to recommend, but they rank them by giving each one a score. When we re-score, we twiddle this score a bit to bias certain products to being more likely to appear at the top of the predictions list. Since a site will only have space to show a selection of the predictions, this can make the difference as to whether a product appears as predicted or not. Use cases for re-scoring include:

• Finding a way to factor in some ratings data when we have a boolean model. For example, if you primarily want to go with a boolean data model based on sales history, you may also want to use customer’s ratings of a product somehow. You could bias items with high ratings to be more likely to appear in the results than items with lower ratings. This helps you make the most of both data sets.
• Biasing products that are more profitable for you to sell towards being ranked as better predictions.
• Biasing products that are on special offer to being ranked higher (perhaps you have a lot of a certain item in stock and want to try to increase sales of it)

Tracking and A-B Testing
Just introducing a product prediction system is not the end of the story – or at least, should not be. It’s also important to track how many purchases are made as a result of the predictions. This just means tracking click-throughs to product pages made from the product prediction results, and setting a flag when such products are later purchased. This way, you can get a sense of the return on investment that your product prediction engine is delivering.

Isn’t this expensive?
Getting a basic product prediction solution in place need not be expensive. Naturally, much time can be spent optimizing predictions and trying out different configurations. However, a basic solution can deliver significant ROI. Then, once you’re comfortable that the predictor is delivering value, you can invest in further optimizing it to increase your sales.

Going further
If you’re interested in product prediction solutions, feel free to contact Edument. We’ll be happy to discuss how you can put the data you already have available to good use in making product predictions. We are also experts at integrating with existing systems and data, with specialists in a range of technologies.

Taking things a step further, you may want to make your prediction engine configurable, or even run two parallel versions of it. The first time a visitor hits the site, you select which version of the prediction engine that their visit will fall under. You then store not just that the purchase was the result of a prediction, but which version or configuration of the predictor led to the purchase. By combining that with data on how many predictions were served by each configuration or version, you can get real world data on how well different approaches are working for you, and then pick the configuration that works best.

About the author:

Jonathan Worthington is an architect and programming mentor at Edument. He works to help development teams improve their software architecture and work more efficiently, following best practices. He is also one of the core developers of the Rakudo Perl 6 compiler, where his work focuses on the implementation of object oriented language features and the type system. In his free time, he enjoys hiking in the mountains, good beer and curry.

February 23, 2011 at 11:21 am edument Leave a comment

Välkommen, Carl Mäsak, till Edument!

Vi vill börja med att välkomna vår nya medarbetare Carl Mäsak till Edument och hoppas att han kommer trivas hos oss. Vi har tänkt köra en presentation av Carl så att både våra kunder och medarbetare ska få chansen att lära känna honom!

Vad har du gjort tidigare?

Jag har studerat till civilingenjör i Uppsala. Under studietiden har jag jobbat vid sidan om med programmering, mestadels för universitetet. Jag har en bred erfarenhet som utvecklare av öppen källkod, särskilt som Perl-utvecklare. Jag är en av de mer aktiva personerna i utvecklingen av Perl 6. I jobbet har det mestadels blivit Java och Eclipse, men jag är även naturligt nyfiken på nya språk och paradigm.

Du har hållit på med bioinformatisk mjukvara tidigare?

Ja, min civilingengörsutbildning har namnet ”Bioinformatik”, och under de senste åren har jag ingått i en forskargrupp som utvecklar Bioclipse, en integrerad mjukvaruplattform för kemo- och bioinformatik.

Vad inom systemutveckling brinner du mest för?

Jag tycker att själva utvecklingsprocessen är det häftiga – att utforska ett problem, och skriva något som sedan körs och löser problemet.

J. S. Bach lär ha sagt att det inte är så svårt att spela orgel: man behöver bara trycka på rätt tangenter i rätt ordning. Programmering känns “lätt-svårt” på just det sättet; det är lätt när man kan det. Och det är spännande att lära sig nya tekniker för att förstå det bättre och nå längre.

Varför valde du att jobba för just Edument?

Jag gillar värdena som ligger till grund för Edument. Konsultverksamhet kan se ut på så många olika sätt – hos Edument handlar det om att ge kunden något långsiktigt bra, inte bara plåsterlösningar. Det finns något lockande i att vara med hela vägen med kunden, både i utvecklingsarbetet på plats och i fortbildning och mentorskap.

Hon Edument kommer jag att ägna mig åt mobil utveckling, Android, Meego, samt undervisning/konsulting i diverse språk: Java, C++, Perl…

Har du några speciella highlights i din karriär?

Min “karriär” så här långt återfinns huvudsakligen i Perl-världen. Jag har varit med och utvecklat specifikation och implementationer för Perl 6, jag har skrivit ett antal program och moduler, mina föredrag på konferenser är omtyckta, och min blogg har tusentals besökare i månaden. Under den gångna månaden har jag arrangerat en programmeringstävling på Internet.

Vad har du för fritidsintressen?

Jag tycker om att dansa, simma, springa, sjunga i kör, laga mat. baka, spela piano/orgel, komponera, skriva, och rita. Dock ej alla samtidigt.

Kuriosa: Vilken var din första dator?

En Amstrad med två 5 1/4 tums diskettstationer och utan hårddisk. MS-DOS 3 någonting. Gjorde mitt första “plattformsspel” i Lotus 123, den tidens Excel.

Läs mer: Carls konsultprofil hos Edument.

January 14, 2011 at 9:10 am edument Leave a comment