Advertisements
RSS

Tag Archives: AWS

How to connect to the CEPH Object Gateway S3 API with Java

We use a CEPH storage solution and specifically want to use the Ceph Object Gateway with the S3 API through a Java client. The API is based on the AWS S3 standard however requires some special tweaking to work. Took me some effort to get a working connection, so here to share:

<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-java-sdk</artifactId>
  <version>1.11.325</version>
</dependency>

We can use either the new AmazonS3ClientBuilder

package nl.rubix.s3;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.SDKGlobalConfiguration;

public class AmazonS3ClientBuilder
{
  public static void main(String[] args)
  {
    String accessKey = "XXXXX";
    String secretKey = "XXXXX";

    // Our firewall on DEV does some weird stuff so we disable SSL cert check
    System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY,"true");
    if (SDKGlobalConfiguration.isCertCheckingDisabled())
    {
      System.out.println("Cert checking is disabled");
    }
		
    // S3 Client configuration
    ClientConfiguration config = new ClientConfiguration();
    // Not the standard "AWS3SignerType", maar expliciet signerTypeV2
    config.setSignerOverride("S3SignerType");
    config.setProtocol(Protocol.HTTPS);
    config.setProxyHost("proxy.rubix.nl");

    config.setProxyPort(8080);
    // S3 Credentials
    BasicAWSCredentials credentials = new BasicAWSCredentials(accessKey,secretKey);
    // S3 Endpoint
    AwsClientBuilder.EndpointConfiguration endpointConfiguration = new
      AwsClientBuilder.EndpointConfiguration("objects.dc1.rubix.nl", "");
    AmazonS3 s3 = com.amazonaws.services.s3.AmazonS3ClientBuilder.standard()
      .withClientConfiguration(config)
      .withCredentials(new AWSStaticCredentialsProvider(credentials))
      .withEndpointConfiguration(endpointConfiguration)
      .build();
    
    System.out.println("===========================================");
    System.out.println(" Connection to the Rubix S3 ");
    System.out.println("===========================================\n");
    try { 
       /*
       * List of buckets and objects in our account
       */
       System.out.println("Listing buckets and objects");
       for (Bucket bucket : s3.listBuckets())
       {
         System.out.println(" - " + bucket.getName() +" "
           + "(owner = " + bucket.getOwner()
           + " "
           + "(creationDate = " + bucket.getCreationDate());
         ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
           .withBucketName(bucket.getName()));
         for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) 
         {
           System.out.println(" --- " + objectSummary.getKey() +" "
           + "(size = " + objectSummary.getSize() + ")" +" "
           + "(eTag = " + objectSummary.getETag() + ")");
           System.out.println();
         }
       }
     }
     catch (AmazonServiceException ase)
     {
       System.out.println("Caught an AmazonServiceException, which means your request made it to S3, but was rejected with an error response for some reason.");
       System.out.println("Error Message:    " + ase.getMessage());
       System.out.println("HTTP Status Code: " + ase.getStatusCode());
       System.out.println("AWS Error Code: " + ase.getErrorCode());
       System.out.println("Error Type: " + ase.getErrorType());
       System.out.println("Request ID: " + ase.getRequestId());
     }
     catch (AmazonClientException ace)
     {
       System.out.println("Caught an AmazonClientException, which means the client encountered "
       + "a serious internal problem while trying to communicate with S3,
       + "such as not being able to access the network.");
       System.out.println("Error Message: " + ace.getMessage());
     }

or make it work with the older and depricated AmazonS3Client

package nl.rubix.s3;

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.SDKGlobalConfiguration;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amaonaws.services.s3.model.Bucket;

public class BasicAWSCredentials
{
    public static void main(String[] args)
    {
        String accessKey = "XXXXXXX";
        String secretKey = "XXXXXXX";
        System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY,"true"
    );

    if (SDKGlobalConfiguration.isCertCheckingDisabled())
    {
        System.out.println("Cert checking is disabled");
    }
    AWSCredentials credentials = new com.amazonaws.auth.BasicAWSCredentials(accessKey,secretKey);

    ClientConfiguration clientConfig = new ClientConfiguration();
    clientConfig.setSignerOverride("S3SignerType");
    clientConfig.setProxyHost("proxy.rubix.nl");
    clientConfig.setProxyPort(8080);

    AmazonS3 conn = new AmazonS3Client(credentials, clientConfig);
    conn.setEndpoint("objects.gn3.rubix.nl");

    for (Bucket bucket : conn.listBuckets())
    {
      System.out.println(" - " + bucket.getName() 
        + " "
        + "(owner = " + bucket.getOwner()
        + " "
        + "(creationDate = " + bucket.getCreationDate());
    }
  }
}

Hope it helps!

Advertisements
 
Leave a comment

Posted by on 03-07-2018 in Uncategorized

 

Tags: , , , , ,

How to push to AWS CodeCommit from Mac OS X

When trying to commit to a AWS CodeCommit GIT repository I receive the following error:

jvzoggel$ git push
 fatal: unable to access 'https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myProject/': The requested URL returned error: 403

The Amazon website states:

If you are using macOS, use HTTPS to connect to an AWS CodeCommit repository. After you connect to an AWS CodeCommit repository with HTTPS for the first time, subsequent access will fail after about fifteen minutes. The default Git version on macOS uses the Keychain Access utility to store credentials. For security measures, the password generated for access to your AWS CodeCommit repository is temporary, so the credentials stored in the keychain will stop working after about 15 minutes. To prevent these expired credentials from being used, you must either:

  • Install a version of Git that does not use the keychain by default.
  • Configure the Keychain Access utility to not provide credentials for AWS CodeCommit repositories.

I used the second option to fix it, so:

  1. Open the Keychain Access utility (use Finder to locate it)
  2. Search for git-codecommit
  3. Select the row, right-click and then choose Get Info.
  4. Choose the Access Control tab.
  5. In Confirm before allowing access, choose git-credential-osxkeychain, and then choose the minus sign to remove it from the list.

screen-shot-2017-02-15-at-19-21-22

After removing git-credential-osxkeychain from the list, you will see a pop-up dialog whenever you run a Git command. Choose Deny to continue. The pop-up is really annoying so I will probably switch over to SSH soon.

References

 

 
Leave a comment

Posted by on 18-02-2017 in Uncategorized

 

Tags: , , , ,

How to install AWS CLI on Mac OS X

The 2nd time last month I had to do it myself / figure it out / explain it so I decided to note it down.

Install AWS CLI on your OSX

jvzoggel$ brew install awscli
jvzoggel$ echo 'complete -C aws_completer aws' >> ~/.bashrc

AWS Identity and Access Management

From the IAM console create your personal access key ID and secret access key. Make sure to note them both down in a safe place !

screen-shot-2017-02-15-at-18-43-45

Configure the aws-cli

Use the generated AWS IAM credentials to configure your AWS CLI  for connection.

jvzoggel$ aws configure
 AWS Access Key ID [None]: xxxxxxxxx
 AWS Secret Access Key [None]: xxxxx
 Default region name [None]: eu-west-1
 Default output format [None]:

jvzoggel$ aws --version
 aws-cli/1.11.48 Python/2.7.10 Darwin/16.4.0 botocore/1.5.11

 

References

 
Leave a comment

Posted by on 16-02-2017 in Uncategorized

 

Tags: , ,

Cloud integration using federation between Microsoft Office 365 Azure Active Directory (AAD) and Amazon Web Service (AWS)

Not an Oracle blog for a change, but when an organization uses both Amazon Web Services (AWS) and Microsoft Office 365 it is possible to allow single sign-on with the internal LDAP Microsoft uses (Azure AD). Since RubiX uses both cloud products since day 1, I decided to look into integration between both products when Microsoft recently allowed SAML federation.

In this blog I will demonstrate how to connect Amazon Web Services (AWS) to the internal Azure Active Directory (AAD) that is used by Microsoft. As a result of this blog your users should be able to login to AWS from the Office 365 menu.

RESULT

1. Configure Microsoft Office 365 / Azure Active Directory (AAD)

Go to your Administration console and select Azure AD from ADMIN

AZURE000_01

In the Azure AD console select “Active Direcory”, click on your Office 365 domain name and the AD menu should open. Click on “Applications” from the top menu

AZURE000

By defaut you will see a lot of Microsoft web applications, so we click on “Add+” on the bottom menu. Then select “Add application from the gallery”

AZURE000-2

The AWS application will be added to your list. Select Configure Single Sign-On next.

AZURE001

We will select the 1st option (MS AAD SSO) to establish federation between AAD & AWS. The Federated Single Sign-On enables the users in your organization to be automatically signed in to a third-party application like AWS by using the AAD user account information. In this scenario, when you have already been logged into Office 365 the federation eliminates the need for you to login again to AWS.

AZURE002

In this case, we don’t need to perform any extra advanced settings. So NEXT

AZURE003

Download the metadata XML and store it for future use and make sure to accept the checkbox

AZURE004

Go to the users tab and assign (bottom button) your users that are allowed to login to AWS

AZURE011

Before we can finalize our SSO from AAD, we first need to setup AWS.

2. Configure Amazon Web Service (AWS)

Login to your AWS account and select the Identity & Access Management

EC2001

First we will create an Identity Provider for AAD.
Select SAML as Provider Type and choose a logical name (I use “Office365” in my example).
Browse to the exported metadata we downloaded from the AAD console earlier.

Important: check your metadata xml file

  • The exported metadata XML file from Azure might be encoded as UTF-8 with byte order mark (BOM). Make sure to convert it to UTF-8 without BOM otherwise the AWS console will not be able to import it.
  • Make sure to remove the <?xml version=”1.0″?> on line 1, otherwise AWS will not be able to parse the file

EC2001_provider

As a result we now have a SAML provider configured, so time to set some roles.
Select Roles in the IAM menu, select “Create New Role” and give your role a logical name (I use “RubixUsers” here)
In the Role Type select “Grant Web Single Sign-On (WebSSO) access to SAML providers“.

EC2004

  • Select the SAML provider we trust, so we use the earlier created “Office365” provider here.
  • Next step we can customize the policy, which we won’t do so next.
  • In the next step we can select the policy you want to attach to your SSO users.
    You can go fine grained with policies, but for now I will use the default PowerUser policy
  • The last screen you will receive a review of the configuration, make sure to note down the Role ARN and Thrusted Entities
    Role ARN = arn:aws:iam::[customerID]:role/[RoleName]
    Trust = arn:aws:iam::[customerID]:saml-provider/[ProviderName]

EC2001_review

 

3. Configure Microsoft Office 365 / Azure Active Directory (AAD) – part 2

Go back to the AAD management console (https://manage.windowsazure.com).
Select applications -> Amazon Web Services (AWS) -> Attributes

Add the following 2 attributes:

EC2_attributes

 

4. Result

With these configuration steps you are now able to login to AWS from your Office 365 apps tile.

RESULT

 
2 Comments

Posted by on 16-10-2015 in Uncategorized

 

Tags: , , , , , , ,