Perl is a modern scripting language with a vibrant developer community, active core developers, and just massive amounts of modules. Perl source code drives companies, web pages, IRC bots, and parts of the Internet infrastructure. However, much has happened since the first release of Perl in 1987; as a way to celebrate that, the theme of the workshop this year is “Perl in 2011″.

Edument is a newly started company in the Skåne region in southern Sweden, providing consultancy, education, and development mentorship. Informator is a well-established provider of education in the Nordic countries. Their joint sponsoring of the event means that the attendees will have a lovely venue close to the station, with a nice view of the Malmö harbour area.

The event is arranged by Jonathan Worthington and Carl Mäsak, both of them Edument employees who also happen to be core developers in the Perl 6 community.

In this post, I summarize how things went.

Architect meets domain expert

Since there were only the two of us, I took the role of the domain expert, and Jonathan took the role of the architect. He expertly teased a model out of me. We arrived at two aggregate roots: Player and Game. Easy enough.

Design: commands and events

Using an inept but sufficient schema tool, we drew up the commands and events we figured we needed. These were as follows:

RegisterPlayerCommand
ActivatePlayerCommand
InvitePlayerCommand
AcceptInvitationCommand
RejectInvitationCommand
(no StartGameCommand)
PlaceStoneCommand
SwapPlayerColorsCommand
ResignGameCommand
TimeOutGameCommand

PlayerRegisteredEvent
PlayerActivatedEvent
PlayerInvitedEvent
GameStartedEvent
InvitationRejectedEvent
StonePlacedEvent
GameWonEvent
PlayerColorsSwappedEvent
GameResignedEvent
GameTimedOutEvent

For each command and event, we took a moment to model through what data we needed to send along. It gave us an appreciation for one of the ways in which commands and events differ: on the inside.

There was a moment of joyful insight as we realized that we had gotten this far into the design of the system and not once talked about state. Quite a
refreshing change.

Being the one with the “domain expert” knowledge, I kept unwillingly slipping back into the role of the client. Otherwise we’d have gotten some things wrong, which wouldn’t have shown up until the next “meeting with the client”. Jonathan remarked: “There’s got to be a lesson in here somewhere.”

(Afterwards, we’ve changed two things in the above model: we eventually realized that we would need an InvitationAcceptedEvent after all. The reason we originally figured we’d be able to do without it is that we noticed that it would fire off a GameStartedEvent, and that would be enough. But no, it needs to fire off both, otherwise the Invitation would still be open. The other thing we realized was that a better name for InvitePlayerCommand and PlayerInvitedEvent would be MakeInvitationCommand and InvitationMadeEvent. That way, all the commands and events contain in their names the aggregate that they are acting on — which makes a lot of sense.)

Commands, events, test framework

We wrote our first test, and the necessary classes to go with it.

Infrastructure

Our goal now was to get enough of the system running for our first test to fail. That took a few hours, partly due to the fact that we were figuring out how to fit everything together.

The wiring is like this: The test contains a ‘given’ list of events, a ‘when’ command, and a ‘then’ list of events or an exception. The test fixture creates an aggregate root to do the testing on, and loads it up with the events from the ‘given’ part. Aggregate roots have a special flag on the apply_event method for applying events without having them register as changes to be committed. That’s all the concession to testing that’s needed. Quite neat.

The test fixture then sends the command to a bus-like thing. This triggers the right command handler, which does the required validation and then calls a method on the aggregate. That’s the command part of things.

Now, the method on the aggregate is just a thin wrapper for applying an event. The event is mapped through a lookup table (our workaround for the lack of method overloading in Perl 5) to an apply-event method. Note that on the way, we visited the same apply_event method as when we prepared the aggregate with the ‘given’ events. This time the generated events are saved, though… and that’s exactly what we’re then using to check against the ‘then’ events. (Or, if we got an exception, the test fixture captures that and compares it with whatever was expected.)

It’s quite a simple system, though it took us a few hours to understand and get running. Still not too bad for our first attempt.

Getting the first test to pass

Trying to get the test we’d written at the beginning of the day to pass, we realized that we were still missing one component: a repository to store the aggregate in while we were testing it. We settled on writing a test repository, with a total capacity of one (1) aggregate.

After that, things fell into place quickly. We got our event wired up, and the test passing. Thus, we entered into the next phase…

Ping-pong pair programming

By the looks of the commit log, that’s where I became unconscious and Jonathan kept on hacking. But what really happened is that we paired up over Jonathan’s keyboard and started hacking in earnest.

Ordinary pair programming has a “driver” and a “navigator”. In ping-pong pair programming, the idea
is for the two people to alternate by taking turns writing a test for the other to implement. This was the first time we tried that, and it went very smoothly. Definitely something to try again. In regular pair programming, the navigator can sometimes doze off. But doing things this way, both of us were engaging with the process of writing code and tests, even when we weren’t in the role of driver.

We got through eight such cycles of ping and pong. At this point, things were really effortless: all the groundwork was already made, and now that we were finally implementing state in our aggregates, there were no longer any obstacles left. A very weird feeling; the aggregate was its own little world, merely responding to commands and events as they came flying by. Coding was effortless, not least because we managed to time it with the Ballmer Peak. Mmm, beer.

We surprised each other a bit by turning what appeared to be quite tricky tests into excessively simple bits of implementation. Things generally required less wiring up than we expected. (Again, because object state wasn’t the driving component, leaving us free to structure the innards of an aggregate any which way we wanted.)

One thing we also discovered is that we generally had to write fewer tests than “usual”. Each new test covered a bit more ground than we expected, and we often didn’t bother to write a test because we already knew it was going to pass. We’re not sure whether that’s (a) a good thing, and we shouldn’t worry, (b) a bad thing that’s going to cost us in the future, or just (c) a sign that we knew too much about the implementation. Guess more practice with this way of testing will tell.

All in all, a happy first day with CQRS and event sourcing in actual practice.

Attribution

go is Copyright © 2007 by luisbg.

The author

Carl Mäsak has a passion for software and software process. He works as an architect and programming mentor at Edument. He likes to work on healing systems in need of an architecture, or to help introduce order into a chaotic domain. On his spare time, he rides a bike, cooks food, and writes music. Not necessarily all at the same time.

Upcoming CQRS events

Edument will arrange a Software Architecture – Community Day in Malmo, Gothenburg and Stockholm.

Edument will also host a CQRS evening event with Swenug.

Inte nog med det så lanserades även nya kurser och utbildningar som kunde bokas direkt på plats av våra kunder. Exempel på dessa kurser är:
Web Security for Developers
Software Architecture
Using ASP.NET MVC in the real world: From start to finish

Vi tackar alla som deltog denna gång och ser redan fram emot nästa!

(för att se Part: 2,3,4 o.s.v klicka in er på YouTube där resterande finns!)

How a predictor engine typically integrates with an existing database-driven website

For example, if a customer is looking at a travel guide to Russia, from the historical sales data we may see that other customers who purchased this travel guide also tended to purchase a Russian phrasebook or a guide book about neighbouring Ukraine. “Gee,” says the customer. “I guess a phrase book would come in handy too, so I know my borsch from my babushki.” And so you get an extra sale.

Data Models
The key thing you need to get started with product prediction is a data set. Generally, the larger the better, although extremely large data sets may lead to slow predictions. This is something you will have to assess in your particular use case. To give you an idea of what’s manageable, though, I’ve been comfortably churning out predictions in under 10 milliseconds for data sets containing millions of past purchases. Granted it took me a little effort to achieve this kind of performance, but it’s within reach.

There are two types of data models: boolean models and ratings models. In a boolean model, either a customer has a connection with a product or they don’t. For example:

• They either purchased the product (and have a connection with it) or they didn’t purchase the product (and thus don’t)
• They either wrote a review of the product or they didn’t (note that it could be argued that even if the user wrote a crappy review of a product, they still had an interest in it, so we just treat the presence of a view as a connection)
• They either viewed the item or they didn’t (if you don’t have loads of historical sales data, you could data-mine web logs and try and get some connection data based upon which products a given site visitor viewed, on the basis that visitors to a site will generally tend to view related products; here, site visitors serve the place of customers in the data set).

An alternative is to have a ratings based data model. For example, we may have a large number of product ratings given by customers, on a scale of 1 to 5, where 1 is suckful and 5 is awesome. With a ratings based model, a customer giving a product a rating of 5 implies a stronger connection than a rating of 1. The prediction process then considers these values. Note that ratings are not the only possible source of data for using such a model; you could also consider:

• How many purchases a customer made of a product, if you have a range of products that often lead to repeat orders. For example, if you’re selling beer online, then a customer may have purchased Hobgoblin 10 times, but (understandably) only purchased Carling once.
• If mining web logs, you could consider number of visits to a product page as being your kind of “rating”

Your choice of boolean or rating model will directly influence the prediction algorithm that you use, since some algorithms only work with one model or the other. Which you opt for largely depends on what type of data you have, though as noted even if you have ratings data you may just like to abstract it down to “did the customer rate the product or not”. A boolean model certainly has the benefit of simplicity, and I’ve had very good results using such a model. If you have resources, then trying multiple algorithms and data models and using A-B testing would, of course, give you a far more scientific answer.

User-based and Item-based Predictors
Once you have a data model, there are two general types of predictors that you can use it with: user-based and item-based.

Improving Predictions
In some cases, you may have extra information that you wish to use to influence product prediction. For example:

A user-based predictor works by building a “network” of users with similar interests, and then making recommendations based on that. That is, given our current user, we can work out the other users nearest to them in the network and give recommendations based upon what those other users purchased. This depends on having some current historical data for the current user. We generally need to pick two things when working with a user-based predictor: a user similarity algorithm that computes the similarity of two users, and a user neighbourhood algorithm that builds the network of most related users. A side-effect of this is that we also win a way to compute “users with similar interests”, which for some cases may be useful.

An item-based predictor works based upon item similarity. To work, it also requires you to select an item similarity algorithm, which determines how similar two items are based on the overlap in customers purchasing them (depending on the algorithm, it actually can make transitive reasoning rather than requiring immediate overlap). To produce recommendations for a user, we go from the items they purchased and find most similar items to all of those. However, the big win from this approach is that we can compute most similar products. This is extremely useful because it gives us a way to make predictions for a product that is currently being viewed.

One additional benefit of an item-based predictor is that they allow for more pre-computation. The reason I could compute most similar products in 10 milliseconds was actually because I could, off-line in a batch job, pre-compute tens of thousands of set intersections that would be used in producing the results rather than computing them on the fly. Thus if you have time constraints or a large data set, item predictors are going to scale better.

• You may want to filter out products that a customer has already added to their basket from the prediction results; they already decided to buy it, so it’s best to show them new things they may also wish to buy.
• If you have some products that are not “family friendly” then you may wish to filter them out. Just because a many parents of kids who like Thomas The Tank Engine also happen to have purchased horror movies doesn’t mean they are suitable recommendations.
• You may want to suggest products in a given price range, which will take the customer over some threshold where they get a volume discount, or can use a voucher, or get free shipping.

An additional tweak can involve re-scoring the predicted items. Prediction engines not only find products to recommend, but they rank them by giving each one a score. When we re-score, we twiddle this score a bit to bias certain products to being more likely to appear at the top of the predictions list. Since a site will only have space to show a selection of the predictions, this can make the difference as to whether a product appears as predicted or not. Use cases for re-scoring include:

• Finding a way to factor in some ratings data when we have a boolean model. For example, if you primarily want to go with a boolean data model based on sales history, you may also want to use customer’s ratings of a product somehow. You could bias items with high ratings to be more likely to appear in the results than items with lower ratings. This helps you make the most of both data sets.
• Biasing products that are more profitable for you to sell towards being ranked as better predictions.
• Biasing products that are on special offer to being ranked higher (perhaps you have a lot of a certain item in stock and want to try to increase sales of it)

Tracking and A-B Testing
Just introducing a product prediction system is not the end of the story – or at least, should not be. It’s also important to track how many purchases are made as a result of the predictions. This just means tracking click-throughs to product pages made from the product prediction results, and setting a flag when such products are later purchased. This way, you can get a sense of the return on investment that your product prediction engine is delivering.

Isn’t this expensive?
Getting a basic product prediction solution in place need not be expensive. Naturally, much time can be spent optimizing predictions and trying out different configurations. However, a basic solution can deliver significant ROI. Then, once you’re comfortable that the predictor is delivering value, you can invest in further optimizing it to increase your sales.

Going further
If you’re interested in product prediction solutions, feel free to contact Edument. We’ll be happy to discuss how you can put the data you already have available to good use in making product predictions. We are also experts at integrating with existing systems and data, with specialists in a range of technologies.

Taking things a step further, you may want to make your prediction engine configurable, or even run two parallel versions of it. The first time a visitor hits the site, you select which version of the prediction engine that their visit will fall under. You then store not just that the purchase was the result of a prediction, but which version or configuration of the predictor led to the purchase. By combining that with data on how many predictions were served by each configuration or version, you can get real world data on how well different approaches are working for you, and then pick the configuration that works best.

About the author:

Jonathan Worthington is an architect and programming mentor at Edument. He works to help development teams improve their software architecture and work more efficiently, following best practices. He is also one of the core developers of the Rakudo Perl 6 compiler, where his work focuses on the implementation of object oriented language features and the type system. In his free time, he enjoys hiking in the mountains, good beer and curry.

Vad har du gjort tidigare?

Jag har studerat till civilingenjör i Uppsala. Under studietiden har jag jobbat vid sidan om med programmering, mestadels för universitetet. Jag har en bred erfarenhet som utvecklare av öppen källkod, särskilt som Perl-utvecklare. Jag är en av de mer aktiva personerna i utvecklingen av Perl 6. I jobbet har det mestadels blivit Java och Eclipse, men jag är även naturligt nyfiken på nya språk och paradigm.

Du har hållit på med bioinformatisk mjukvara tidigare?

Ja, min civilingengörsutbildning har namnet ”Bioinformatik”, och under de senste åren har jag ingått i en forskargrupp som utvecklar Bioclipse, en integrerad mjukvaruplattform för kemo- och bioinformatik.

Vad inom systemutveckling brinner du mest för?

Jag tycker att själva utvecklingsprocessen är det häftiga – att utforska ett problem, och skriva något som sedan körs och löser problemet.

J. S. Bach lär ha sagt att det inte är så svårt att spela orgel: man behöver bara trycka på rätt tangenter i rätt ordning. Programmering känns “lätt-svårt” på just det sättet; det är lätt när man kan det. Och det är spännande att lära sig nya tekniker för att förstå det bättre och nå längre.

Varför valde du att jobba för just Edument?

Jag gillar värdena som ligger till grund för Edument. Konsultverksamhet kan se ut på så många olika sätt – hos Edument handlar det om att ge kunden något långsiktigt bra, inte bara plåsterlösningar. Det finns något lockande i att vara med hela vägen med kunden, både i utvecklingsarbetet på plats och i fortbildning och mentorskap.

Hon Edument kommer jag att ägna mig åt mobil utveckling, Android, Meego, samt undervisning/konsulting i diverse språk: Java, C++, Perl…

Har du några speciella highlights i din karriär?

Min “karriär” så här långt återfinns huvudsakligen i Perl-världen. Jag har varit med och utvecklat specifikation och implementationer för Perl 6, jag har skrivit ett antal program och moduler, mina föredrag på konferenser är omtyckta, och min blogg har tusentals besökare i månaden. Under den gångna månaden har jag arrangerat en programmeringstävling på Internet.

Vad har du för fritidsintressen?

Jag tycker om att dansa, simma, springa, sjunga i kör, laga mat. baka, spela piano/orgel, komponera, skriva, och rita. Dock ej alla samtidigt.

Kuriosa: Vilken var din första dator?

En Amstrad med två 5 1/4 tums diskettstationer och utan hårddisk. MS-DOS 3 någonting. Gjorde mitt första “plattformsspel” i Lotus 123, den tidens Excel.

Läs mer: Carls konsultprofil hos Edument.

Med över 150 deltagare och positiv feedback så är vi otroligt nöjda och ser redan fram emot nästa gång då vi hoppas få se alla deltagande igen, och kanske några nya ansikten? Det hoppas vi!

Eduments eget föredrag handlade om infallsvinklar till dataåtkomst på dotnet. Vi frågade om det fanns “för många sätt att göra det” för närvarande, och vi drog slutsatsen att idag på dotnet-plattformen har vi många verktyg för många olika uppgifter, snarare än många verktyg som alla passar för samma problemområde. Vi fokuserade på vikten av språket (i lingvistisk mening, inte som programmeringsspråk) när man överväger dataåtkomst-arkitektur.

Bilderna från våra föredrag finns tillgängliga här, om ni är intresserade att se dem.

Keynote: Microsoft introducerar SQL Server 2008 R2
Vanligaste misstagen som utvecklare gör i SQL Server
.Net Data Access: Too many ways to do it?
SQL CLR
Execution Plan Basics
Introduktion till BI för utvecklare
PowerPivot

Vi vill tacka alla som hjälpte till att göra ett lyckat evenemang: organisatörerna, föredragshållarna, våra sponsorer

JetBrains
FinalBuilder
Redgate
Typemock
Microsoft

och sist men inte minst publiken. Vi ses nästa gång!

I föredraget fokuserade han på hur språket är utformat för att vara skalbart från små enradiga engångsprogram, över små verktyg, och vidare upp till att bygga större applikationer. I synnerhet tittade vi på några exempel av specifika språksärdrag som är nyttiga för vart och ett av dessa områden. Det tillhandahåller en bra överblick av något av det mest intressanta nya innehållet i Perl 6.

Lyckligtvis gjorde NUUG en utmärkt videoinspelning av föredraget, som man kan se på:

http://www.nuug.no/aktiviteter/20100914-little-tools-large-apps/

Tack, NUUG, för inbjudan, till alla som anslöt till föredraget, och för den mycket angenäma middagen och drinkarna efteråt. Oslo är en härlig stad, och det var “greit” att hänga med några av dess vänliga hackare också.

Agenda

07.30-08.30 Frukost

08.30-08.40 Introduktion av arrangörerna, Informator, Pass, Edument

08.40-09.25 Keynote: Microsoft introducerar SQL Server 2008 R2.

09.40-10.25 Vanligaste misstagen som utvecklare gör i SQL Server
Peter Larsson (MVP), PASS Scania

Vilka är de vanligaste misstagen utvecklare gör i SQL Server i början? Varför går det inte så snabbt nu längre? Frågorna är många och det finns lyckligtvis några få enkla svar! Vi kommer att gå igenom de vanligaste misstagen och hur man åtgärdar problemen för att få bättre prestanda och säkrare resultat.

• Översikt
• Beräkningar på indexerade kolumner
• Tid- och datumberäkningar
• Felaktiga datatyper
• RBAR (Row-By-Agonizing-Row)

10.40-11.25 .Net Data Access: Too many ways to do it?
Jonathan Worthington

Today’s .Net developers have more choice than ever when it comes to data access. To simply select some data from a database, one could use the SqlConnection, SqlCommand and SqlDataReader objects (with an SQL query or through a stored procedure), Linq, Enity Framework (POCO or not) – and that’s before we even consider non-Microsoft offerings, such as NHibernate, or the possibility of generating data access layers from templates and database schemas.

No one of these approaches is right for every application; indeed, applications may legitimately make use of more than one approach. In this talk we’ll look through the strengths and pitfalls of a range of the options, their performance both in terms of development time and execution time, and see what tasks they lend themselves to.

11.40-12.40 Lunch & Mingel

12.40-13.25 SQL CLR

Matt Whitfield (MVP), Atlantis Interactive

CLR integration within SQL Server allows developers and administrators alike to harness the power of the .NET Framework from directly within their databases. This seminar will cover the basics of implementing all types of CLR objects, including types, aggregates, procedures, triggers and functions, as well as showing some examples of how CLR objects have been put to good use.

SQL CLR Overview

• * Overview of CLR in SQL Server
• * Explore all types of CLR object
• * The differences in CLR permission sets
• * Worked examples of problem-solving with the CLR

13.40-14.25 Execution Plan Basics
Peter Larsson (MVP), PASS Scania

Som utvecklare kan exekveringsplanen i SQL Server vara dötrist, och långtråkig och… och alldeles underbar! I denna session visar vi på hur du kan tyda resultaten i vad som verkar en främmande och krånglig karta och hur du kan dra slutsatser ur resultaten. Vi visar också hur man jämför två queries och ser vilken som är mest effektiv.

14.40-15.25 SQL Azure

15.40-16.25 PowerPivot
Thomas Ivarsson (MVP), PASS Scania

PowerPivot is a new client add in for Excel 2010 and a Sharepoint 2010 Server application that can query extremely large datasets, with millions of rows, with an instant response within a second.

PowerPivot is based on the older Analysis Services archtecture but with a new column based in memory engine called Vertipaq that surpasses all previous limits for Excel clients. In this seminar you will get a general introduction to this new application.

• * PowerPivot for beginners
• * What is PowerPivot and why is it important
• * Data sources supported in PowerPivot
• * Import data into PowerPivot
• * What can you do with the new DAX-expressions language for PowerPivot calculations?

Läs mer om eventet och boka nu!