Skip to main content

Connection Parameters

The connection parameters refer to a set of essential connection details required for establishing a secure link to supported external storage services, like Amazon S3. These parameters are enclosed within parentheses and consists of key-value pairs separated by commas or spaces. It is commonly utilized in operations such as creating a stage, copying data into Databend, and querying staged files from external sources. The provided key-value pairs offer the necessary authentication and configuration information for the connection.

For example, the following statement creates an external stage on Amazon S3 with the connection parameters:

CREATE STAGE my_s3_stage
URL = 's3://load/files/'
CONNECTION = (
ACCESS_KEY_ID = '<your-access-key-id>',
SECRET_ACCESS_KEY = '<your-secret-access-key>'
);

The connection parameters vary for different storage services based on their specific requirements and authentication mechanisms. For more information, please refer to the tables below.

Amazon S3-like Storage Services

The following table lists connection parameters for accessing an Amazon S3-like storage service:

ParameterRequired?Description
endpoint_urlYesEndpoint URL for Amazon S3-like storage service.
access_key_idYesAccess key ID for identifying the requester.
secret_access_keyYesSecret access key for authentication.
allow_anonymousNoWhether anonymous access is allowed. Defaults to false.
enable_virtual_host_styleNoWhether to use virtual host-style URLs. Defaults to false.
master_keyNoOptional master key for advanced data encryption.
regionNoAWS region where the bucket is located.
security_tokenNoSecurity token for temporary credentials.
note
  • If the endpoint_url parameter is not specified in the command, Databend will create the stage on Amazon S3 by default. Therefore, when you create an external stage on an S3-compatible object storage or other object storage solutions, be sure to include the endpoint_url parameter.

  • If you're using S3 storage and your bucket has public read access, you can access and query an external stage associated with the bucket anonymously without providing credentials. To enable this feature, add the allow_anonymous parameter to the [storage.s3] section in the databend-query.toml configuration file and set it to true.

To access your Amazon S3 buckets, you can also specify an AWS IAM role and external ID for authentication. By specifying an AWS IAM role and external ID, you can provide more granular control over which S3 buckets a user can access. This means that if the IAM role has been granted permissions to access only specific S3 buckets, then the user will only be able to access those buckets. An external ID can further enhance security by providing an additional layer of verification. For more information, see https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html

The following table lists connection parameters for accessing Amazon S3 storage service using AWS IAM role authentication:

ParameterRequired?Description
endpoint_urlNoEndpoint URL for Amazon S3.
role_arnYesARN of the AWS IAM role for authorization to S3.
external_idNoExternal ID for enhanced security in role assumption.

Azure Blob Storage

The following table lists connection parameters for accessing Azure Blob Storage:

ParameterRequired?Description
endpoint_urlYesEndpoint URL for Azure Blob Storage.
account_keyYesAzure Blob Storage account key for authentication.
account_nameYesAzure Blob Storage account name for identification.

Google Cloud Storage

The following table lists connection parameters for accessing Google Cloud Storage:

ParameterRequired?Description
endpoint_urlYesEndpoint URL for Google Cloud Storage.
credentialYesGoogle Cloud Storage credential for authentication.

Alibaba Cloud OSS

The following table lists connection parameters for accessing Alibaba Cloud OSS:

ParameterRequired?Description
access_key_idYesAlibaba Cloud OSS access key ID for authentication.
access_key_secretYesAlibaba Cloud OSS access key secret for authentication.
endpoint_urlYesEndpoint URL for Alibaba Cloud OSS.
presign_endpoint_urlNoEndpoint URL for presigning Alibaba Cloud OSS URLs.

Tencent Cloud Object Storage

The following table lists connection parameters for accessing Tencent Cloud Object Storage (COS):

ParameterRequired?Description
endpoint_urlYesEndpoint URL for Tencent Cloud Object Storage.
secret_idYesTencent Cloud Object Storage secret ID for authentication.
secret_keyYesTencent Cloud Object Storage secret key for authentication.

HDFS

The following table lists connection parameters for accessing Hadoop Distributed File System (HDFS):

ParameterRequired?Description
name_nodeYesHDFS NameNode address for connecting to the cluster.

WebHDFS

The following table lists connection parameters for accessing WebHDFS:

ParameterRequired?Description
endpoint_urlYesEndpoint URL for WebHDFS.
delegationNoDelegation token for accessing WebHDFS.