Publishing Maven Artifacts to AWS S3

Published 2013-07-07.
Time to read: 2 minutes.

This page is part of the posts collection, categorized under AWS, Git, Open Source.

I wanted a simple, flexible and cheap way of publishing proprietary Maven artifacts created by sbt projects such that they could be securely retrieved by authorized individuals. I liked the idea of versioned artifacts, but did not want to use GitHub or BitBucket to host the artifacts because of the hassle of maintaining ever-larger git repos. Instead, I opted for S3's optional versioning mechanism.

The technique described here relies on the fact that publishing to a local file (using sbt publish) generates all the necessary artifacts, which merely need to be copied to the right directory on the Artifactory server. The server need not be anything special: a normal web server works fine, as does webdav. A variety of other protocols are also supported by sbt.

I created two test projects on GitHub: testPublishLib and testPublishApp. You could clone them if you would like to try this. Each of them has a README that explains what they do.

I used s3cmd to manage the AWS S3 buckets that hold the repository. You can obtain s3cmd for most OSes. I found I needed to install python-magic:

Shell

$ sudo pip install python-magic

Let s3cmd know your s3 keys:
Shell
```
$ s3cmd --configure
```
Create the S3 bucket, which must be unique. If you want to repository to be publicly visible, be sure that the bucket name starts with www. If you already have the S3 bucket then just omit this step.
Shell
```
$ s3cmd mb s3://www.mymavenrepo
```
If you want to repository to be publicly visible, you need to enable the S3 bucket web site option:
Shell
```
$ s3cmd ws-create s3://www.mymavenrepo
```
Publish your library to a local repository. testPublishLib is an example of how to set up a project properly.
Shell
```
$ sbt publish
```
Copy your locally published artifacts to S3. You can either type it out longhand:
Shell
```
$ s3cmd -P sync ~/.ivy2/local/com/micronautics/test_publish_lib \
s3://www.mymavenrepo/snapshots/com/micronautics/test_publish_lib
```
... (note that the -P option makes the files publicly visible), or you can use s3publish:
Shell
```
$ s3publish com/micronautics/test_publish_lib
```
If your repository is not public, you will need to provide authentication in a file which I called ~/.sbt/awsCreds.sbt for convenience:
~/.sbt/awsCreds.sbt
```
credentials +=
  Credentials("AWS Realm", "www.mavenrepo.s3.amazonaws.com",
    "myUserId", "myPassword")
```
Use the published artifact from your sbt project by including a resolver of the form:
Shell
```
"AWS Snapshots" at "https://www.mavenrepo.s3.amazonaws.com/snapshots"
```
The TestPublishApp project is a working example of how to do that.

Following is the script I wrote to upload to S3, which I called s3publish. It assumes you are always publishing a snapshot; and that the files should be public. I leave it to you to extend this script to handle releases and private content if you have the need.

#!/bin/bash if [ $# -eq 0 ]; then echo "Usage: `basename $0` pubpath/name [repo]" echo " Where pubpath might be something like com/micronautics" echo " name is name of artifact to publish" echo " repo is the optional name of the bucket to publish to" echo " Example: `basename $0` com/micronautics/test_publish_lib" exit 1 fi REPO=www.mymavenrepo OPTIONS=-P if [ $# -eq 2 ]; REPO=$2; fi s3cmd $OPTIONS sync ~/.ivy2/local/$1 s3://$REPO/snapshots/$1