net.snowflake.client.jdbc.cloud.storage

Class SnowflakeS3Client

java.lang.Object

net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client

All Implemented Interfaces:

SnowflakeStorageClient

public class SnowflakeS3Client
extends Object
implements SnowflakeStorageClient

Wrapper around AmazonS3Client.

Author:

ffunke

Constructor Summary

Constructors

Constructor and Description
SnowflakeS3Client(Map<?,?> stageCredentials, com.amazonaws.ClientConfiguration clientConfig, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat, Properties proxyProperties, String stageRegion, String stageEndPoint, boolean isClientSideEncrypted, SFBaseSession session, boolean useS3RegionalUrl)

Method Summary

Modifier and Type	Method and Description
void	addDigestMetadata(StorageObjectMetadata meta, String digest) Adds digest metadata to the StorageObjectMetadata object
void	addEncryptionMetadata(StorageObjectMetadata meta, MatDesc matDesc, byte[] ivData, byte[] encKeK, long contentLength) Adds encryption metadata to the StorageObjectMetadata object
void	addStreamingIngestMetadata(StorageObjectMetadata meta, String clientName, String clientKey) Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation
void	download(SFSession session, String command, String localLocation, String destFileName, int parallelism, String remoteStorageLocation, String stageFilePath, String stageRegion, String presignedUrl) Download a file from S3.
InputStream	downloadToStream(SFSession session, String command, int parallelism, String remoteStorageLocation, String stageFilePath, String stageRegion, String presignedUrl) Download a file from remote storage
String	getDigestMetadata(StorageObjectMetadata meta) Gets digest metadata to the StorageObjectMetadata object
int	getEncryptionKeySize()
String	getMatdescKey() Returns the material descriptor key
int	getMaxRetries()
StorageObjectMetadata	getObjectMetadata(String remoteStorageLocation, String prefix) Returns the metadata properties for a remote storage object
int	getRetryBackoffMaxExponent() Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry
int	getRetryBackoffMin()
String	getStreamingIngestClientKey(StorageObjectMetadata meta) Gets streaming ingest client key to the StorageObjectMetadata object
String	getStreamingIngestClientName(StorageObjectMetadata meta) Gets streaming ingest client name to the StorageObjectMetadata object
void	handleStorageException(Exception ex, int retryCount, String operation, SFSession session, String command) Handles exceptions thrown by the remote storage provider
boolean	isEncrypting()
StorageObjectSummaryCollection	listObjects(String remoteStorageLocation, String prefix) For a set of remote storage objects under a remote location and a given prefix/path returns their properties wrapped in ObjectSummary objects
void	renew(Map<?,?> stageCredentials) Renew the S3 client with fresh AWS credentials/access token
void	shutdown() shuts down the client
void	upload(SFSession session, String command, int parallelism, boolean uploadFromStream, String remoteStorageLocation, File srcFile, String destFileName, InputStream inputStream, FileBackedOutputStream fileBackedOutputStream, StorageObjectMetadata meta, String stageRegion, String presignedUrl) Upload a file (-stream) to S3.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.snowflake.client.jdbc.cloud.storage.SnowflakeStorageClient

requirePresignedUrl, uploadWithPresignedUrlWithoutConnection

Constructor Detail

SnowflakeS3Client

public SnowflakeS3Client(Map<?,?> stageCredentials,
                         com.amazonaws.ClientConfiguration clientConfig,
                         net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                         Properties proxyProperties,
                         String stageRegion,
                         String stageEndPoint,
                         boolean isClientSideEncrypted,
                         SFBaseSession session,
                         boolean useS3RegionalUrl)
                  throws SnowflakeSQLException

Throws:

SnowflakeSQLException

Method Detail

getMaxRetries

public int getMaxRetries()

Specified by:

getMaxRetries in interface SnowflakeStorageClient

Returns:

Returns the Max number of retry attempts

getRetryBackoffMaxExponent

public int getRetryBackoffMaxExponent()

Description copied from interface: SnowflakeStorageClient

Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry

Specified by:

getRetryBackoffMaxExponent in interface SnowflakeStorageClient

Returns:

Returns the exponent

getRetryBackoffMin

public int getRetryBackoffMin()

Specified by:

getRetryBackoffMin in interface SnowflakeStorageClient

Returns:

Returns the min number of milliseconds to sleep before retry

isEncrypting

public boolean isEncrypting()

Specified by:

isEncrypting in interface SnowflakeStorageClient

Returns:

Returns true if encryption is enabled

getEncryptionKeySize

public int getEncryptionKeySize()

Specified by:

getEncryptionKeySize in interface SnowflakeStorageClient

Returns:

Returns the size of the encryption key

renew

public void renew(Map<?,?> stageCredentials)
           throws SnowflakeSQLException

Renew the S3 client with fresh AWS credentials/access token

Specified by:

renew in interface SnowflakeStorageClient

Parameters:

stageCredentials - a Map of new AWS credential properties, to refresh the client with (as returned by GS)

Throws:

SnowflakeSQLException - if any error occurs

shutdown

public void shutdown()

Description copied from interface: SnowflakeStorageClient

shuts down the client

Specified by:

shutdown in interface SnowflakeStorageClient

listObjects

public StorageObjectSummaryCollection listObjects(String remoteStorageLocation,
                                                  String prefix)
                                           throws StorageProviderException

Description copied from interface: SnowflakeStorageClient

For a set of remote storage objects under a remote location and a given prefix/path returns their properties wrapped in ObjectSummary objects

Specified by:

listObjects in interface SnowflakeStorageClient

Parameters:

remoteStorageLocation - location, i.e. bucket for S3

prefix - the prefix to list

Returns:

a collection of storage summary objects

Throws:

StorageProviderException - cloud storage provider error

getObjectMetadata

public StorageObjectMetadata getObjectMetadata(String remoteStorageLocation,
                                               String prefix)
                                        throws StorageProviderException

Description copied from interface: SnowflakeStorageClient

Returns the metadata properties for a remote storage object

Specified by:

getObjectMetadata in interface SnowflakeStorageClient

Parameters:

remoteStorageLocation - location, i.e. bucket for S3

prefix - the prefix/path of the object to retrieve

Returns:

storage metadata object

Throws:

StorageProviderException - cloud storage provider error

download

public void download(SFSession session,
                     String command,
                     String localLocation,
                     String destFileName,
                     int parallelism,
                     String remoteStorageLocation,
                     String stageFilePath,
                     String stageRegion,
                     String presignedUrl)
              throws SnowflakeSQLException

Download a file from S3.

Specified by:

download in interface SnowflakeStorageClient

Parameters:

session - session object

command - command to download file

localLocation - local file path

destFileName - destination file name

parallelism - number of threads for parallel downloading

remoteStorageLocation - s3 bucket name

stageFilePath - stage file path

stageRegion - region name where the stage persists

presignedUrl - Not used in S3

Throws:

SnowflakeSQLException - if download failed without an exception

SnowflakeSQLException - if failed to decrypt downloaded file

SnowflakeSQLException - if file metadata is incomplete

downloadToStream

public InputStream downloadToStream(SFSession session,
                                    String command,
                                    int parallelism,
                                    String remoteStorageLocation,
                                    String stageFilePath,
                                    String stageRegion,
                                    String presignedUrl)
                             throws SnowflakeSQLException

Download a file from remote storage

Specified by:

downloadToStream in interface SnowflakeStorageClient

Parameters:

session - session object

command - command to download file

parallelism - number of threads for parallel downloading

remoteStorageLocation - remote storage location, i.e. bucket for s3

stageFilePath - stage file path

stageRegion - region name where the stage persists

presignedUrl - Not used in S3

Returns:

input file stream

Throws:

SnowflakeSQLException - when download failure

upload

public void upload(SFSession session,
                   String command,
                   int parallelism,
                   boolean uploadFromStream,
                   String remoteStorageLocation,
                   File srcFile,
                   String destFileName,
                   InputStream inputStream,
                   FileBackedOutputStream fileBackedOutputStream,
                   StorageObjectMetadata meta,
                   String stageRegion,
                   String presignedUrl)
            throws SnowflakeSQLException

Upload a file (-stream) to S3.

Specified by:

upload in interface SnowflakeStorageClient

Parameters:

session - session object

command - upload command

parallelism - number of threads do parallel uploading

uploadFromStream - true if upload source is stream

remoteStorageLocation - s3 bucket name

srcFile - source file if not uploading from a stream

destFileName - file name on s3 after upload

inputStream - stream used for uploading if fileBackedOutputStream is null

fileBackedOutputStream - stream used for uploading if not null

meta - object meta data

stageRegion - region name where the stage persists

presignedUrl - Not used in S3

Throws:

SnowflakeSQLException - if upload failed even after retry

handleStorageException

public void handleStorageException(Exception ex,
                                   int retryCount,
                                   String operation,
                                   SFSession session,
                                   String command)
                            throws SnowflakeSQLException

Description copied from interface: SnowflakeStorageClient

Handles exceptions thrown by the remote storage provider

Specified by:

handleStorageException in interface SnowflakeStorageClient

Parameters:

ex - the exception to handle

retryCount - current number of retries, incremented by the caller before each call

operation - string that indicates the function/operation that was taking place, when the exception was raised, for example "upload"

session - the current SFSession object used by the client

command - the command attempted at the time of the exception

Throws:

SnowflakeSQLException - exceptions that were not handled, or retried past what the retry policy allows, are propagated

getMatdescKey

public String getMatdescKey()

Returns the material descriptor key

Specified by:

getMatdescKey in interface SnowflakeStorageClient

Returns:

the material descriptor key

addEncryptionMetadata

public void addEncryptionMetadata(StorageObjectMetadata meta,
                                  MatDesc matDesc,
                                  byte[] ivData,
                                  byte[] encKeK,
                                  long contentLength)

Adds encryption metadata to the StorageObjectMetadata object

Specified by:

addEncryptionMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the encyption info to

matDesc - the material decriptor

ivData - the initialization vector

encKeK - the key encryption key

contentLength - the length of the encrypted content

addDigestMetadata

public void addDigestMetadata(StorageObjectMetadata meta,
                              String digest)

Adds digest metadata to the StorageObjectMetadata object

Specified by:

addDigestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the digest to

digest - the digest metadata to add

getDigestMetadata

public String getDigestMetadata(StorageObjectMetadata meta)

Gets digest metadata to the StorageObjectMetadata object

Specified by:

getDigestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the metadata object to extract the digest metadata from

Returns:

the digest metadata value

addStreamingIngestMetadata

public void addStreamingIngestMetadata(StorageObjectMetadata meta,
                                       String clientName,
                                       String clientKey)

Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation

Specified by:

addStreamingIngestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the digest to

clientName - streaming ingest client name

clientKey - streaming ingest client key, provided by Snowflake

getStreamingIngestClientName

public String getStreamingIngestClientName(StorageObjectMetadata meta)

Gets streaming ingest client name to the StorageObjectMetadata object

Specified by:

getStreamingIngestClientName in interface SnowflakeStorageClient

getStreamingIngestClientKey

public String getStreamingIngestClientKey(StorageObjectMetadata meta)

Gets streaming ingest client key to the StorageObjectMetadata object

Specified by:

getStreamingIngestClientKey in interface SnowflakeStorageClient