Using Amazon S3 as a Maven repository

October 15th, 2009

I’ve been creating several small projects that I’ve shared with friends and collegues. An often recurring problem is that as soon as I share a project I receive the same complaint: “it doesn’t build. Maven complains about an artifact that it cannot find”. It works great on my local development laptop since all my Maven dependencies are safely located in my local repository. The solution is to provide my artifacts through a publically available Maven repository.

Among the many Maven Repository Managers, Nexus is a popular choice. I’m quite familiar with it, but in this case I’d rather just drop off my artifacts on a remote server. Reading this blog, I decided to deploy all my artifacts to a bucket on my Amazon S3 account. These are the steps required to make it all happen.

Wagon

First of all I had to checkout and build Wagon-s3 from source. Wagon-S3 is a protocol provider to the Wagon Maven extension. Wagon is used by, for example, the maven-deploy-plugin to upload artifacts to a Maven repository. As the same suggeste, Wagon-S3 enables Wagon to transfer artifacts to an Amazon S3 bucket.

First I needed to checkout and build the main Wagon project:

<code>
  svn checkout http://svn.apache.org/repos/asf/maven/wagon/tags/wagon-1.0-beta-6 wagon
  cd wagon/
  mvn install
</code>

Then I need to checkout the Wagon-S3 project:

<code>
  svn checkout http://svn.apache.org/repos/asf/maven/sandbox/trunk/wagon/wagon-s3 wagon-s3
</code>

Make sure the parent POM of the Wagon-S3 project points to the same versjon as the main Wagon project you just build:

<code lang="XML">
  <parent>
    <artifactId>wagon-providers</artifactId>
    <groupId>org.apache.maven.wagon</groupId>
    <version><strong>1.0-beta-6</strong></version>
  </parent>
</code>

Now you should have a brand new build of Wagon S3 safely in your local Maven repository. Now it’s time to make it possible to deploy your project artifact to S3 using this extension.

Maven Dependency Management

The maven-deploy-plugin makes use of the information provided in your POMs  dependencyManagement section:

<code lang="XML">
  <distributionManagement>
    <repository>
      <id>s3-release</id>
      <name>jForce S3 Release</name>
      <url><b>s3rest://release.jforce.no/</b></url>
    </repository>
    <snapshotRepository>
      <id>s3-snapshot</id>
      <name>jForce S3 Snapshot</name>
      <url><b>s3rest://snapshot.jforce.no/</b></url>
    </snapshotRepository>
  </distributionManagement>
</code>

Trying to deploy your project now will result in nothing but a stacktrace on your console. The next step is to create an Amazon S3 account.

Amazon S3

Howto create an S3 account is covered by several blog articles and tutorials on the Internet, thus I’ll skip this part here. In my case I installed the Firefox S3 plugin S3Fox. Using this tool I created two buckets, namely snapshot.jforce.no and release.jforce.no and edited their ACL to allow read access to everyone.

Uploading artifacts to the buckets requires authorization by Maven. This is done by adding the following snippet to your settings.xml file:

<code lang="XML">
  <servers>
    <server>
      <id><strong>s3-release</strong></id>
      <privateKey>[YOUR AWS ACCESS KEY]</privateKey>
      <passphrase>[YOUR AWS SECRET KEY]</passphrase>
    </server>
    <server>
      <id><strong>s3-snapshot</strong></id>
      <privateKey>[YOUR AWS ACCESS KEY]</privateKey>
      <passphrase>[YOUR AWS SECRET KEY]</passphrase>
    </server>
  </servers>
</code>

Notice how the server id is equal to the repository id in the distribution-management section. Your Amazon Web Services (AWS) credentials are available from the Your Account menu on the Amazon S3 webpage.

Deploy

Now you are ready to deploy your artifacts. This is done using the following Maven command:

<code>
mvn clean deploy
</code>

When done you can browse your bucket with S3Fox and find all your project artifacts in the familiar Maven directory structure.

Downloading from S3

Now that deployment to S3 has been taken care of, other projects that have dependencies to your projects can simply add the following to their POM (or to their repository manager):

<code lang="XML">
  <repositories>
    <repository>
      <id>s3-snapshot</id>
      <name>jForce S3 Snapshot</name>
      <url><strong>http://snapshot.jforce.no.s3.amazonaws.com/</strong></url>
    </repository>
    <repository>
      <id>s3-release</id>
      <name>jForce S3 Release</name>
      <url><strong>http://release.jforce.no.s3.amazonaws.com/</strong></url>
    </repository>
  </repositories>
</code>

This should complete the circle. Now you are able to deploy to S3 and your friends and collegues can resolve/download dependencies from S3. Enjoy!

Scala WTF: Scala and generics

October 12th, 2009

I finally decided to start a project in order to learn the basics of Scala. I already have a personal Java framework with some DDD concepts, and I though it was a good idea to reuse this framework in my new project. Boy did I hit the wall early ;) As a novice Scala developer I’m reaching out for someone to help me out here.

I have the following interface in my Java framework:

public interface Entity<E extends Entity, I> {
    I id();
    boolean equals(E otherEntity);
}

In my first attempt to implements this interface in Scala I came up with the following snippet:

class ShoppingList(val id: Long) extends Entity[ShoppingList, Long] {
    ...
}

This didn’t compile, giving me the following error message:

    error: type arguments [ShoppingList,Long] do not conform to trait Entity's type parameter
    bounds [E <: no.jforce.fundament.domain.Entity, Long].

Through the process of experimentation I was able to achieve partial success by doing the following:

trait _Entity[A <: Entity[A, B], B] extends Entity[A, B] {
    ...
}

class ShoppingList(val id: Long) extends _Entity[ShoppingList, Long] {
    ...
}

Can anyone please explain to me how I can use the Entity Java-interface directly?

Ninety-Nine Scala Problems (11 through 25) - solved using Erlang

July 23rd, 2009

The weather isn’t improving here in Oslo, which is bad. However, it gives me a reason to solve another 15 of the famous Ninety-Nine Scala Problems that I started on in my previous blog post. It is funny to see how, by doing only a few exercises, my style of programming Erlang changes. Learning is a constant process.

  •   P11 - Modified run-length encoding.

    encodeModified([]) -> [];
    encodeModified(L) when is_list(L) -> [ modified_encode_sub_list(SubList) || SubList <- scala:pack(L)].
    modified_encode_sub_list([Head|[]]) -> { Head };
    modified_encode_sub_list([Head|Tail]) -> {1 + scala:length(Tail), Head}.
  •   P12 - Decode a run-length encoded list.

    decode([]) -> [];
    decode([{Element}|Tail]) -> [Element] ++ decode(Tail);
    decode([{0, Element}|Tail]) -> [Element] ++ decode(Tail);
    decode([{Count, Element}|Tail]) -> [Element] ++ decode([{Count - 1, Element}|Tail]).
  •   P13 - Run-length encoding of a list (direct solution).

    encodeDirect([]) -> [];
    encodeDirect([Head|Tail]) -> encodeDirect([{Head}], Tail).
    encodeDirect([{Element}], [Element|Tail]) -> encodeDirect([{2, Element}], Tail);
    encodeDirect([{Element}], L) -> [{Element}] ++ encodeDirect(L);
    encodeDirect([{Count, Element}], [Element|Tail]) -> encodeDirect([{Count + 1, Element}], Tail);
    encodeDirect([{Count, Element}], L) -> [{Count, Element}] ++ encodeDirect(L).
  •   P14 - Duplicate the elements of a list.

    duplicate([]) -> [];
    duplicate([Head|Tail]) -> [Head] ++ [Head] ++ duplicate(Tail).
  •   P15 - Duplicate the elements of a list a given number of times.

    duplicateN(_, []) -> [];
    duplicateN(N, L) -> duplicateN(N, N - 1, L).
    duplicateN(N, 0, [Head|Tail]) -> [Head] ++ duplicateN(N, Tail);
    duplicateN(N, I, [Head|Tail]) -> [Head] ++ duplicateN(N, I - 1, [Head|Tail]).
  •   P16 - Drop every Nth element from a list.

    drop(_, []) -> [];
    drop(N, L) -> drop(N, N - 1, L).
    drop(_, _, []) -> [];
    drop(N, 0, [_|Tail]) -> drop(N, Tail);
    drop(N, I, [Head|Tail]) -> [Head] ++ drop(N, I - 1, Tail).
  •   P17 - Split a list into two parts.

    split(N, L) -> split_index(N, lists:zip(L, lists:seq(0, scala:length(L) - 1))).
    split_index(N, L) -> {[Element || {Element, Index} <- L, N > Index], [Element || {Element, Index} <- L, Index >= N]}.
  •   P18 - Extract a slice from a list.

    slice(From, To, List) -> slice(From, To, 0, List).
    slice(From, To, Index, [Head|Tail]) when (Index >= From), (To > Index) -> [Head] ++ slice(From, To, Index + 1, Tail);
    slice(From, To, Index, [_|Tail]) -> slice(From, To, Index + 1, Tail);
    slice(_,_,_,[]) -> [].
  •   P19 - Rotate a list N places to the left.

    rotate(Offset, List) -> rotate(Offset, 0, List).
    rotate(Offset, Offset, [Head|Tail]) -> [Head] ++ Tail;
    rotate(Offset, Index, [Head|Tail]) -> rotate(Offset, Index + 1, Tail ++ [Head]).
  •   P20 - Remove the Kth element from a list.

    removeAt(N, List) -> removeAt(N, 0, [], List).
    removeAt(N, N, ResultList, [Head|Tail]) -> {ResultList ++ Tail, Head};
    removeAt(N, Index, ResultList, [Head|Tail]) -> removeAt(N, Index + 1, ResultList ++ [Head], Tail).
  •   P21 - Insert an element at a given position into a list.

    insertAt(Element, N, List) -> insertAt(Element, N, 0, List).
    insertAt(Element, N, N, List) -> [Element] ++ List;
    insertAt(Element, N, Index, [Head|Tail]) -> [Head] ++ insertAt(Element, N, Index + 1, Tail).
  •   P22 - Create a list containing all integers within a given range.

    range(From, From) -> [From];
    range(From, To) -> [From] ++ range(From + 1, To).
  •   P23 - Extract a given number of randomly selected elements from a list.

    randomSelect(N, List) -> randomSelect(N, 0, [], List).
    randomSelect(N, N, ResultList, _) -> ResultList;
    randomSelect(N, Index, ResultList, List) ->
    {RemainingList, Element} = removeAt(random:uniform(scala:length(List)) - 1, List),
    randomSelect(N, Index + 1, ResultList ++ [Element], RemainingList).
  •   P24 - Lotto: Draw N different random numbers from the set 1..M.

    lotto(N, M) -> randomSelect(N, scala:range(1, M)).
  •   P25 - Generate a random permutation of the elements of a list.

    randomPermute(List) -> randomSelect(scala:length(List) - 1, List).

Again, any Erlang-Ninjas that arrives at this blog, please leave comments and suggestions concerning the solutions.

Ninety-Nine Scala Problems (1 through 10) - solved using Erlang

July 22nd, 2009

A rainy day during my summer vacation served as a perfect opportunity to brush up on my
Erlang skills. And what better way to brush up than to attack the Ninety-nine Scala Problems using Erlang.

  1.   P01 - Find the last element of a list.

    last([Element]) -> Element;
    last([_|Tail]) -> last(Tail).
  2.   P02 - Find the last but one element of a list.

    penultimate([Head|Tail]) when length(Tail) =:= 1 -> Head;
    penultimate([_|Tail]) -> penultimate(Tail).
  3.   P03 - Find the Kth element of a list.

    nth(Index, L) -> nth(Index, 0, L).
    nth(Index, N, [Head|_]) when Index =:= N -> Head;
    nth(Index, N, [_|Tail]) -> nth(Index, N + 1, Tail).
  4.   P04 - Find the number of elements of a list.

    length([]) -> 0;
    length([_|Tail]) -> 1 + scala:length(Tail).
  5.   P05 - Reverse a list.

    reverse([]) -> [];
    reverse([Head|Tail]) -> reverse(Tail) ++ [Head].
  6.   P06 - Find out wether a list is a palindrome.

    isPalindrome([]) -> true;
    isPalindrome(L) -> L == scala:reverse(L).
  7.   P07 - Flatten a nested list.

    flatten([]) -> [];
    flatten([Head|Tail]) when is_list(Head) -> flatten(Head) ++ flatten(Tail);
    flatten([Head|Tail]) -> [Head] ++ flatten(Tail).
  8.   P08 - Eliminate consecutive duplicates of list elements.

    compress([]) -> [];
    compress([Head|Tail]) -> [Head] ++ compress(Head, Tail).
    compress(_, []) -> [];
    compress(Last, [Last|Tail]) -> compress(Last, Tail);
    compress(_, [Head|Tail]) -> [Head] ++ compress(Head, Tail).
  9.   P09 - Pack consecutive duplicates of list elements into sublists.

    pack([]) -> [];
    pack([Head|Tail]) -> pack([Head], Tail).
    pack([Last|Rest], [Last|Tail]) -> pack([Last] ++ [Last|Rest], Tail);
    pack(SubList, [Head|Tail]) -> [SubList] ++ pack([Head], Tail);
    pack(SubList, []) -> [SubList].
  10.   P10 - Run-length encoding of a list.

    encode([]) -> [];
    encode(L) when is_list(L) -> [ encode_sub_list(SubList) || SubList <- scala:pack(L)].
    encode_sub_list([Head|Tail]) -> {1 + scala:length(Tail), Head}.

Those were the first 10 problems solved using Erlang. I can`t promise I`ll finish all of them, but I`ll most certainly try. I welcome any Erlang-Ninja to give feedback on the proposed solutions. Enjoy!

WidgetFX

June 5th, 2009

I attended a session on WidgetFX by Stephen Chin and Joshua Marinacci, which is a desktop and mobile widget platform written in JavaFX. WidgetFX is a derivative from Glossitope, which Joshua Marinachi worked on, but with JavaFX thrown into the mix.

The presentation was quite good. A proper balance between code, feature and demo. It has to be said - WidgetFX looks really good. Transparency on the desktop, drag a widget into its own window. JavaFX really makes sense for this. They also got their act together and included multi-monitor support right from the beginning.

As part of the demo they showcased a WidgetFX widget for interacting with the Nabaztag bunny. Technically not all that advanced, but very visual and that is what counts in a demo. Both presenters did a good job and it was a good start after yesterdays string of disappointments.

RESTful Transaction Systems

June 4th, 2009

I’m a sucker for integration systems and transaction management will always be important in that regard. This is why I chose to attend the presentation titled “RESTful Transaction Systems”. This is, with some close competitors, the worse presentation ever. This is pretty impressive given that the content is rather interesting.

The presentation started out with some historic background about transaction systems, WS-transaction systems and REST. This part of the presentation was pretty good - mostly because the presenter did a good job at conveying the information. The presentation went downhill when the second presenter took over. Talking in a sleepy-dull voice, half of the time mumbling, I had a really hard time understanding a single word he said. A demo would normally help, but what a nightmare. He fired up a console with a font size that can hardly be visible when sitting right infront of the laptop. It was utterly impossible to see anything. It became slightly better when he brought up a Web-application invoking some web services - he really had put a lot of work into creating a elaborate demo and I have to give him credit for that, but the wayhe talked just sucked the energy out of the audience. The audience, by the way, became smaller and smaller for every minute the presentation went on. What a sad tale.

Yet again, a great example of someone totally underestimating the importance of having a good presenter representing your company. In this case, Red Hat made a grave error.

Design Patterns for Complex Event Processing

June 3rd, 2009

Two Oracle engineers gave this presentation about Design Patterns for complex event processing. In all fairness, this was a well-hidden product presentation for Oracle CP, which is a rebrand of a former BEA product for doing, yes, event processing.

The presentation was dull, but still very informative. A very well layed out presentation where core concepts were presented first with just enough detail, and then this concepts were used to describe 8 design patterns, namely Event Filtering, Event Partitioning, Event Enrichment, Event Aggregation, Event Correlation, Application Time, Missed Event Detection and one more that I don’t remember.

Not one wordt was spent mentioning the work described in Enterprise Integration Patterns, which I found strange. Instead the patterns were used to demonstrate the expressive power of CQL - Complex Query Language. All patterns were implemented using this language, which to me looked pretty powerful.

Again I felt that I could pick up a whitepaper about their product and me much smarter. I miss the live demo - fewer patterns and a live demo would convey the message more clearly.

Restful Web Services using JAX-RS

June 3rd, 2009

I knew I picked the wrong presentation. I’ve been trying out Jersey for quite some time now and this simple walkthrough of the JAX-RS had almost no value to me. That said, the presentation and the demo was pretty good. But,  the only thing that I gained from it was that during the demo they demonstrated a pretty rapid application deployment lifecycle against Glassfish. Again, using NetBeans, which has heavy exposure during JavaONE. It’s pretty clear that If you choose Glassfish you should also choose NetBeans. Their tools are getting really integrated.

Spring 3.0 - new and noteworthy

June 3rd, 2009

Although I’ve read quite a bit about the upcoming major release of Spring, I still wanted to attend this session to hear it directly from Rod Johnsen. I’m glad I did. Not so much because the presentation was all that great, but rather because of all the new “Best Practise” changes that Spring now openly have changed. Rod did also present a new Gem in the Spring family, called Roo, which may be an important step into increasing productivity for projects that are heavy on CRUD operations.

Annotation came with Spring 2.5, and they have now taken it a step further in Spring 3.0. If you are comfortable with using annotations in your service and repository classes, you can get rid of most of your XML configurations. This is great news. A brand new Expression Language (EL) is also available. This makes it possible to do the following:

<bean id="config” class=”my.ConfigurationImpl”></bean>
<bean id=”server” class=”my.ServerImpl”>
<constructor-arg value=”#{config.serverPort}“/>
</bean>

Another really noteworthy change is that Spring 3.0 is now 100% Java 5-based, meaning that generics is now used in the whole Spring API. This means, for example, that you will have to do the following from now on:

Person person = context.getBean("person", Person.class);

Another big change is that REST is now directly supported by Spring. Sadly, in my opinion, they have integrated this support into Spring MVC, and not using JAX-RS constructs. This means that instead of using the @Path annotation, they piggyback REST support on the already present Spring MVC annotations. I find this disappointing, but I can still make use of Jersey’s Spring support.

Towards the end of Ron’s presentation he talked a bit about Spring Roo, which is a Spring + Pure Java based answer to Ruby on Rails. In addition Roo has a built-support feature that does continuous code(re)generation while you are developing. Add a field - and the toString is updated for you. SpringSource now has Grails and Roo available for CRUD-heavy application development. Smart move.

Architecting robust applications on EC2

June 3rd, 2009

Went to a presentation given by Chris Richardson, the man behind CloudFoundry and cloudtools. I was hoping that the presentation should be about different architectures that can put the dynamic availability of hosts on EC2 to good use. This, however, was not part of the presentation content. Chris gave a rather interesting summary about the different services that Amazon now delivers in its Cloud initiative. This included not only the services themselve, but also different tooling alternatives available to users.

I’ve played around with EC2 for quite some time now, thus I know about how to create, expand and launch EC2 instances. The idea behind paying for actual usage and scaling when needed really appeals to me. The other services, however, was unknown to me. Particularly the Amazon Block Storage service was interesting to me. EC2 instance have ephemeral storage, meaning that when an instance is terminated, all its generated storage is also lost. Amazon Block Storage lets you create a virtual block device that can be formatted with your filesystem of choice and attached to an EC2 instance. This is very powerful.

Chris also discussed Elastic IP, Elastic Auto-scaling, Amazon Load-balancing Service etc. which are all services that you can put to good use, and pay for usage. Unlike a lot of other presentation I have attended so far, Chris was very clear on both the positive and negative sides of the different Amazon services - including pricing.

Only problem with this presentation is that I get really inspired by the whole Cloud experience - and as a consultant, I’d probably never get to work on an actual project using it. Well, we’ll see.