S3 API Compatibility

The Intercolo Object Storage service uses an S3 compatible API to interact with the storage backend.

This has the advantage of not needing to learn a new API or develop new tooling. In most cases, getting started is as simple as switching over the backend url and credentials to the ones created on the Api Keys page.

Overview

The main differences are:

  • Anonymous access is not allowed, all requests need to be authenticated (signed urls are supported). We do offer a CDN service that works in tandem with our object storage for anonymous access to files.
  • Creating buckets via the S3 API is currently not supported

Authentication

The S3-Compatible API only supports signature version 4. Signature versions 2 and 4A are not supported.

Data Consistency

Interactions with the intercolo object storage are generally consistent and atomic. Once an api call returns, api calls after this one will reflect any change immediately. Any meta-data read will also never be partial or corrupt. This follows the strong read-after-write consistency model of Amazon S3.

The only exception to this consistency is downloading object contents for large multipart objects. We immediately queue data for deletion once an object is deleted or overwritten, so in-progress reads may abort early when they try reading from a part that does not exist anymore. Generally, we have found this to be a non-issue and an acceptable trade-off for data security, if this should be an issue for your use case please do get in touch, and we'll be happy to work on a solution with you.

Supported Actions

The S3-Compatible API currently supports (fully or partially) the following actions:

Differences compared to AWS S3

Concepts

Anonymous access

Anonymous access is not allowed, all requests need to be authenticated (signed urls are supported). We do offer a CDN service that works in tandem with our object storage for anonymous access to files.

Requester Pays

All requests to our S3 compatible api can only be accessed by the owner or via an url signed by the owner. The x-amz-request-payer request header is ignored/does not apply and accordingly there will not be a x-amz-request-charged header in responses.

Encryption

We currently do not support explicitly configuring encryption. This includes Managed Encryption Keys SSE-S3, KMS Keys (SSE-KMS), Dual-layer server-side encryption (DSEE-KMS) and customer-provided encryption keys (SSE-C). This also means any encryption related headers will be ignored/not sent in response

Request headers:

  • x-amz-server-side-encryption-customer-algorithm
  • x-amz-server-side-encryption-customer-key
  • x-amz-server-side-encryption-customer-key-MD5

Response headers:

  • x-amz-server-side-encryption
  • x-amz-server-side-encryption-customer-algorithm
  • x-amz-server-side-encryption-customer-key-MD5
  • x-amz-server-side-encryption-aws-kms-key-id
  • x-amz-server-side-encryption-bucket-key-enabled

Object Locks

Object locks are currently not supported. You can, however, configure an API key to be unable to overwrite or delete objects.

ACLs

Per object ACLs are not supported and permissions can be managed outside the S3 API per bucket per api key.

Lifecycle / Expiring objects

Expiring objects are currently not supported.

Versioning

Versioning is not supported.

Replication

Replication is configured per bucket and happens transparently at upload time.

Storage classes

There is currently only a standard storage class. Higher reliability and durability can be achieved with replication.

Object Tagging

Object tagging is not supported.

Website Hosting

Hosting websites directly on the object storage is not supported. The x-amz-website-redirect-location header will be ignored.

x-amz-expected-bucket-owner Header

The x-amz-expected-bucket-owner and x-amz-source-expected-bucket-owner headers are unsupported and will be ignored.

Per-Action Differences

Any limitations specific to actions other than the general above-mentioned ones

HeadBucket

No known additional differences.

GetBucketLocation

No known additional differences.

GetObjectLockConfiguration

No known additional differences.

GetBucketVersioning

No known additional differences.

ListBuckets

No known additional differences.

ListObjects / ListObjectsV2

Unsupported headers:

  • x-amz-optional-object-attributes is currently not respected and ignored

HeadObject

Unsupported headers:

  • x-amz-checksum-mode Is currently not respected, checksums are always returned
  • If-Match, If-Modified-Since, If-None-Match and If-Unmodified-Since are currently not respected, but are planned to be implemented

GetObject

Unsupported headers:

  • x-amz-checksum-mode Is currently not respected, checksums are always returned
  • If-Match, If-Modified-Since, If-None-Match and If-Unmodified-Since are currently not respected, but are planned to be implemented

Overriding response headers by providing their respective values in the request prefixes with response- is currently not supported. These include:

  • Response-Content-Type
  • Response-Content-Language
  • Response-Expires
  • Response-Cache-Control
  • Response-Content-Disposition
  • Response-Content-Encoding

CopyObject

Unsupported Headers:

  • x-amz-copy-source-if-match is currently unsupported & ignored
  • x-amz-copy-source-if-none-match is currently unsupported & ignored
  • x-amz-copy-source-if-unmodified-since is currently unsupported & ignored
  • x-amz-copy-source-if-modified-since is currently unsupported & ignored
  • x-amz-checksum-algorithm is ignored, all checksums are computed

PutObject

Unsupported Headers (in addition to general limitations, see above):

  • x-amz-sdk-checksum-algorithm is unsupported & ignored
  • Expires is unsupported & ignored

DeleteObject

No known additional differences.

DeleteObjects

Unsupported Headers:

  • x-amz-sdk-checksum-algorithm is unsupported & ignored

ListMultipartUploads

No known additional differences.

CreateMultipartUpload

Unsupported Headers:

  • x-amz-checksum-algorithm is ignored, all checksums are computed

AbortMultipartUpload

No known additional differences.

CompleteMultipartUpload

No known additional differences.

UploadPart

No known additional differences.

UploadPartCopy

Unsupported Headers:

  • x-amz-copy-source-if-match is currently unsupported & ignored
  • x-amz-copy-source-if-none-match is currently unsupported & ignored
  • x-amz-copy-source-if-unmodified-since is currently unsupported & ignored
  • x-amz-copy-source-if-modified-since is currently unsupported & ignored

Mock actions

Some actions may be implemented as a "mock" and reply with empty/dummy data to make some s3 compatible tools work properly.

Currently, the following actions are implemented as a mock:

  • CreateBucket will reply with a BucketAlreadyOwnedByYou error if the bucket exists and is owned by you, or with an Access Denied otherwise.