S3 as a FileMaker Container Alternative

FileMaker Container fields are easy and great for a variety of use-cases, but when used to store large files or large numbers of files, they impose a heavy burden on your FileMaker infrastructure. It can be like a ball and chain attached to your FileMaker database, making it more cumbersome to move data between systems or to manage backups.

There is a better way!

Object Storage Basics

S3 and similar services are known as “object storage”. Objects are files. Object storage is unlike a drive volume in the sense that your files aren’t guaranteed to be physically stored on a single device, together on the same file system. In fact they’re almost guaranteed to be spread across many devices, perhaps hundreds of metres apart!

All objects are organized into “buckets”, which exist in a “region” (physical location in the world). You can use one or many buckets to organize your objects. You might decide to make one bucket for each distinct FileMaker app you build. Or you might decide to segment your objects further by adding separate buckets for distinct modules within your app, or perhaps a “public” and “private” bucket with different privileges.

Objects have paths that can look like directory structures, but they aren’t real directory structures – the path is really just an extended object name, or “key”. Note: the S3 console will allow you to navigate the hierarchy as if it were a real directory structure.

The Costs

In AWS S3 Standard, you can store files for $0.023US per GB per month in US East. The same amount of storage on a General Purpose EBS volume costs $0.08US (3.5 times more!). More importantly, you never pay for space you don’t use on S3, whereas in EC2, you provision storage in advance and pay for all the space you don’t use, too!

There are additional costs for requests & retrievals, data transfer, etc, and many ways to reduce your costs for infrequently accessed data, but if you’re reading this blog, I doubt you’re going to need to think about this for a while.

Other Advantages

Durability

Any object stored in S3 is automatically replicated across at least 2 “Availability Zones” by the time each transaction completes. This is one reason Amazon can promise durability of 99.999999999%.

Availability Zones may exist under the same roof, so if you’re concerned about large scale natural disasters and similar, you can configure redundancy across geographic regions also.

Caveat – this type of redundancy doesn’t protect you against accidental deletion. More on that later…

Take a load off

Your FileMaker Server, that is… leave the business of large file transfer, storage, and backups to an external service so your FileMaker Server can focus on the things it does best.

Web Publishing

Need to display an asset in a web interface? No problem! Your FileMaker app can generate a secure “presigned” URL designed to work for seconds or days, so the user’s browser can request the file directly from your S3 bucket. It’s similar to the streaming URL generated in the FileMaker Data API, but more flexible.

Getting Started with AWS and Configuring S3

Sign up for AWS

If you don’t already have an AWS account, go to console.aws.amazon.com and “Create a new AWS account”. It is strongly recommended to add two-factor authentication to this account and protect your credentials carefully.

You can quickly navigate to AWS Services by typing the name in the search bar at the top of the screen and selecting the service from the result.

Configure S3

Go to the S3 service

Click “Create bucket”, choose a bucket name, and select an AWS Region near you.

As you scroll down, you will see that by default, “Block all public access” is enabled. Leave this on. With this option enabled, you ensure that no matter what you do on individual objects, you can’t accidentally make anything in your bucket public. Only an AWS identity with privileges to read the bucket can see files in your bucket. While S3 has a highly granular and complex permission structure, most use cases are possible while applying this strict policy.

On this screen you could enable Bucket Versioning if you wish. When Bucket Versioning is on, if you attempt to upload an object to a path (or “key”) that already exists, the existing object will be retained as a “version”. The same happens if you “delete” an object. To really delete an object, you would need to explicitly delete the version or use a lifecycle rule, for example to automatically delete versions after a certain number of days. Bucket versioning can also be enabled or disabled later.

There are other options on this page that are beyond the scope of this tutorial.

When you’re ready, click “Create Bucket”. You should now see your bucket in your “Buckets” list:

Click on the bucket name, then open the “Permissions” tab. Scroll down to the section: “Cross-origin resource sharing (CORS)”. If you plan to interact with your bucket through a web browser (including a web viewer), click “Edit” and paste the following CORS configuration:

                    


[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD",
            "PUT"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": []
    }
]

Save your changes.

Configure IAM Policy & User

Now you need to configure a User and Policy so that you can authenticate to the bucket and interact with it through an API.

In AWS, users are managed through the Identity and Access Management (IAM) service. Use the search bar to navigate to the IAM service.

First, create a policy with the appropriate permissions. Click “Policies” in the menu at the left, then click “Create Policy”.

Select the JSON tab and replace the existing contents with the following. This policy grants 4 privileges. s3:PutObject (upload), s3:GetObject (download), s3:ListBucket (list bucket contents), and s3:DeleteObject. These privileges apply to the specific resource listed, in this case the bucket and its contents.

                    


{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "VisualEditor0",
             "Effect": "Allow",
             "Action": [
                 "s3:PutObject",
                 "s3:GetObject",
                 "s3:ListBucket",
                 "s3:DeleteObject"
             ],
             "Resource": [
                 "arn:aws:s3:::YOUR-BUCKET-NAME/*",
                 "arn:aws:s3:::YOUR-BUCKET-NAME"
             ]
         }
     ]
 }

Be sure to replace YOUR-BUCKET-NAME with the actual name of your bucket.

You may not need all of these privileges or you may need to add additional privileges for your use case, for example if you want the API user to be able to view and/or delete versions.

Click “Next: tags” then “Next: review”. Give your policy a name like “s3FileMakerApiUser” and add a description if you wish. Click “Create policy”.

Next, create a User. Click “Users” in the menu at the left, then click “Add users”.

Input a user name like “s3FileMakerApiUser” and choose “Programmatic access” as the “Access type”. Click “Next: Permissions”

Click “Attach existing policies directly”. Find the policy you created earlier and select the box, then click “Next: Tags”, then “Next: Review”, and finally “Create user”.

Click “Show” to expose the Secret access key. The Access key ID and Secret access key will be required to connect to S3 from FileMaker. You will not be able to retrieve the secret key again, but you can generate a new one at any time.

You would generally store this information in a preferences table in your FileMaker app. If you do not have a preferences table, you could temporarily paste this information directly into “Set Variable” script steps, but be aware of the security risks. Most importantly, your API credentials will be exposed in plain text in any DDR or XML output of your FileMaker file.

Congratulations! Configuration on the AWS side is done.

The Presigned URL

This little guy does all the heavy lifting.

An AWS presigned URL contains an entire API call including method, headers, authentication, etc. It contains a signature which is generated using the request details and the AWS secret key, so if one changes any aspect of the URL without generating a new signature, it will be invalid.

The presigned URL is especially useful when you want to allow someone to download a resource, such as a user who you have authenticated on your website. You can generate a presigned URL to retrieve an object in S3, and you can make the URL valid for seconds or days. You can then incorporate this URL into a webpage as it is loading, and the image will load just like any other URL to an image file (for example).

You can also use a presigned URL for other actions, like uploading files to S3, deleting files, or getting object metadata. I like this option because I already have the custom function, so why not use it?

The s3presignedUrl() custom function

s3presignedUrl ( method ; bucket ; region ; file ; expireSeconds ; accessKey ; secretKey ; obtionsObj )

Get the function and docs: quarfie / s3presignedUrl_FMFunction on GitHub

FileMaker Usage

Here are some example scripts for working with S3 in FileMaker:

Uploading a file from a container to S3

                    


Set Variable [ $path ; "folder/thefilename.jpg" ]
Set Variable [ $url ; s3presignedUrl ( "PUT" ; $bucket ; $region ; $path ; 60 ; $access_key ; $secret_key ; "" ]
Set Variable [ $container ; Table::Container ]
Set Variable [ $contentType ; "image/jpeg" ]
Set Variable [ $curl ; "-X PUT -H " & Quote ( "Content-Type: " & $contentType) & " --data-binary @$container -D $responseHeaders" ]
Insert From URL [ Select ; With dialog: Off ; Target: $result ; $url ; Verify SSL Certificates ; cURL options: $curl ; Do not automatically encode URL ]
If [ PatternCount ( $responseHeaders ; "200 OK" ) ]
 //File was successfully uploaded
Else
 //Error
End If

NOTE: The above example includes a hard-coded Content-Type header (“image/jpeg”). You can automatically determine the content type (or “MIME type”) using a custom function like GetMIMEType ( fileNameOrExtension ). The Content-Type is important if you wish to render the file in a browser (including a web viewer), as it will be sent back as a header that the browser requires to display it. Without this information, the browser will simply download the file.

Download a file from S3 to a container field

                    


Set Variable [ $path ; "folder/thefilename.jpg" ]
Set Variable [ $url ; s3presignedUrl ( "GET" ; $bucket ; $region ; $path ; 60 ; $access_key ; $secret_key ; "" ]
Set Variable [ $curl ; "--FM-return-container-variable -D $responseHeaders" ]
Insert From URL [ Select ; With dialog: Off ; Target: $result ; $url ; Verify SSL Certificates ; cURL options: $curl ; Do not automatically encode URL ]
If [ PatternCount ( $responseHeaders ; "200 OK" ) ]
 //File is in $result
 Set Field [ Table::Container ; $result ]
Else
 //Error
End If

View a file in a web viewer

This example assumes the file is an image file that can load into an image tag.

                    


Let ([
   ~path = Table::path ;
   ~url = s3presignedUrl ( "GET" ; $bucket ; $region ; ~path ; 60*5 ; $accessKey ; $secretKey ; "" )
];
"data:text/html,<!DOCTYPE html><html><head></head><body style=\"margin:0px;\"><div style=\"width:100wv;height:100vh;\"><img style=\"max-width:100%;max-height:100%;margin-left:auto;margin-right:auto;display:block;\" src=\"" & ~url & "\"></div></body></html>"
)

Tip: Rather than using a calculation in a web viewer object directly, use the Set Web Viewer script step. This can be called OnRecordLoad or whenever the web viewer is exposed. This should solve refresh issues that you may encounter when resizing windows or modifying the underlying record.

Delete File Example

                    


Set Variable [ $path ; "folder/thefilename.jpg" ]
Set Variable [ $url ; s3presignedUrl ( "DELETE" ; $bucket ; $region ; $path ; 60 ; $access_key ; $secret_key ; "" ]
Set Variable [ $curl ; "-X DELETE -D $responseHeaders" ]
Insert From URL [ Select ; With dialog: Off ; Target: $result ; $url ; Verify SSL Certificates ; cURL options: $curl ; Do not automatically encode URL ]
If [ PatternCount ( $responseHeaders ; "204 No Content" ) ]
 //File was successfully deleted or never existed to begin with
Else
 //Error
End If

Get Metadata about the object in S3 (Content-Type and File Size)

                    


Set Variable [ $path ; "folder/thefilename.jpg" ]
Set Variable [ $url ; s3presignedUrl ( "HEAD" ; $bucket ; $region ; $path ; 60 ; $access_key ; $secret_key ; "" ]
Set Variable [ $curl ; "--head" ]
Insert From URL [ Select ; With dialog: Off ; Target: $result ; $url ; Verify SSL Certificates ; cURL options: $curl ; Do not automatically encode URL ]
Set Variable [ $responseCode ; Value: MiddleWords ( $result ; 3 ; 1 ) ]
If [ $responseCode = 200 ]
 //File exists and you can now parse out the headers Content-Type and Content-Length
Else
 //Error
End If

Next up: eliminating container fields and building a great UI

By adding a JavaScript-based uploader like Uppy, and taking advantage of FileMaker 19’s new JavaScript integration features, you can completely eliminate the need for FileMaker containers!

The attached file demonstrates this capability and works in FileMaker Pro, Go, and WebDirect.

Similarly, when it comes time to display the file, conditional logic generates the appropriate html/css to display the file based on its type. If it cannot render the file type or the file type is unfamiliar, it will present an html page with a download link. When clicked, the file is downloaded to the user’s download folder.

The file is intended as a demonstration and not a starter file. I encourage you to build your own file in a way that is consistent with your own coding style.

Closing Thoughts

I’m so glad you made it all the way to the bottom of this very long post! I am looking forward to hearing from you, especially if there is anything I could have done better! If your comment is not posted, please reach out to me, as I employ a very aggressive spam filter!

2 Replies to “S3 as a FileMaker Container Alternative”

  1. Hi Jason,

    Thank you so much for this. I have been trying to get this right and the s3presignedURL function works nicely.

    Regards
    Nathan

Leave a Reply to Nathan Veitch Cancel reply

Your email address will not be published. Required fields are marked *