How to Automate Data Sync to Amazon S3 with Hubway Extract

This guide covers the end-to-end workflow, from preparing your data project to verifying files in your AWS environment.

📋 Prerequisites (Before You Start)

To ensure a successful integration, please ensure the following requirements are met in your AWS environment:

• Active AWS Account: You will need access to the AWS Management Console.
• Valid Permissions: Your IAM credentials (Access Key & Secret Key) must have the ‘List objects and Write’ (PutObject) permissions on the S3 bucket.
• Bucket Already Exists: Hubway does not create the bucket for you. The target bucket must already exist in AWS; otherwise, the connection will fail.

Step 1: Create a Project in Hubway Connect

The first step is to define the scope of the data you want to manage.

1. Log in to Hubway Connect and click on “New Project”.
2. Enter a Project Name and select “Create project”.
3. Add the Entities (data tables) you wish to manage.
4. Click Publish.

Status: Your project is created, and the data entities are ready for extraction.

Step 2: Create a New Connection (Hubway Extract)

Now, let’s establish the secure bridge between Hubway and Amazon S3.

1. Navigate to Hubway Extract and click on “New Connection”.
2. Select the connection type: Amazon S3.
3. Fill in the required security credentials and details:
   • Connection Name
   • Access Key & Secret Key
   • Region (e.g., us-east-1)
   • Bucket Name
4. Configure File Path Templates (Crucial for organisation):
   • File Path Template: We recommend using the default suggestion: {Project}/{Entity}_{Timestamp}.json.
   • Metadata Storage Template: Suggested format: {Project}/.
5. Click the Save button.

⚠️ Important Warning on File Naming: If you configure the connection without defining the {Project} or {Timestamp} in the File Path Template, your documents may be overwritten with every new synchronisation.

Best Practice: Always include {Project} and {Timestamp} in the path to differentiate between projects and maintain a history of your files.

Step 3: Define the Extract Plan (Hubway Extract)

With the connection saved, it is time to schedule the automation.

1. Select the “Extract Plans” feature in the menu.
2. Locate the project you created in Step 1 and click on “+Create Plan”.
3. Under Connection Type, select Amazon S3.
4. Select the connection you created in Step 2.
5. Define Schedule: Set how often the data should sync.
6. Define Sync Methods: Choose how each entity should be updated.
7. Click the “Publish and run now” button.

The system will save the plan, publish it, and immediately trigger the first execution.

Step 4: Verify Files in AWS S3

Once the extract plan execution is complete, you can verify the synchronised data directly in your AWS account.

1. Open the AWS Management Console and navigate to the S3 service.
2. Open the Bucket you configured in the connection settings.
3. Locate the project folder.
   • Note: The folder name is automatically derived from the “Service Name” (Project Name) to ensure an organised directory structure.
4. Inside the folder, you will find the synchronised JSON files.
   • These files are automatically named based on the Entity Name (and alias, if applicable) and include a timestamp for full traceability.
     

Troubleshooting (FAQ)

Q: My connection failed to save. 

A: Ensure that the Bucket Name is correct and that the bucket already exists in AWS. Hubway cannot connect to a non-existent bucket. Also, double-check that your Access Keys have “Write” permissions.

Q: My files are being replaced instead of creating a history. 

A: This is likely due to the File Path Template. If you remove the _{Timestamp} variable, the system will use the same filename every time, overwriting the previous version. Edit your connection settings to include the timestamp.

Want to learn more? Visit our FAQs on Hubway Extract – Support Contents – S3