net.snowflake.client.jdbc

Class SnowflakeFileTransferAgent

java.lang.Object

net.snowflake.client.jdbc.SFBaseFileTransferAgent

net.snowflake.client.jdbc.SnowflakeFileTransferAgent

All Implemented Interfaces:

SnowflakeFixedView

public class SnowflakeFileTransferAgent
extends SFBaseFileTransferAgent

Class for uploading/downloading files

Author:

jhuang

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SnowflakeFileTransferAgent.ResultStatus Result status enum

Nested classes/interfaces inherited from class net.snowflake.client.jdbc.SFBaseFileTransferAgent

SFBaseFileTransferAgent.CommandType, SFBaseFileTransferAgent.DownloadCommandEncryptionFacade, SFBaseFileTransferAgent.DownloadCommandFacade, SFBaseFileTransferAgent.UploadColumns, SFBaseFileTransferAgent.UploadCommandEncryptionFacade, SFBaseFileTransferAgent.UploadCommandFacade

Field Summary

Fields

Modifier and Type	Field and Description
static String	SRC_FILE_NAME_FOR_STREAM

Fields inherited from class net.snowflake.client.jdbc.SFBaseFileTransferAgent

commandType, compressSourceFromStream, destFileNameForStreamSource, destStagePath, showEncryptionParameter, sourceFromStream, sourceStream, statusRows

Constructor Summary

Constructors

Constructor and Description
SnowflakeFileTransferAgent(String command, SFSession session, SFStatement statement)

Method Summary

Modifier and Type	Method and Description
void	cancel()
InputStream	downloadStream(String fileName) Download a file from remote, and return an input stream
boolean	execute() Run the PUT/GET command, if a command has been set.
static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.remoteLocation	extractLocationAndPath(String stageLocationPath) A small helper for extracting location name and path from full location path
SFBaseFileTransferAgent.CommandType	getCommandType()
static Callable<Void>	getDownloadFileCallable(StageInfo stage, String srcFilePath, String localLocation, Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap, SnowflakeStorageClient client, SFSession session, String command, int parallel, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat, String presignedUrl) A callable that can be executed in a separate thread using executor service.
List<net.snowflake.common.core.RemoteStoreFileEncryptionMaterial>	getEncryptionMaterial()
List<SnowflakeFileTransferMetadata>	getFileTransferMetadatas() This is API function to retrieve the File Transfer Metadatas.
static List<SnowflakeFileTransferMetadata>	getFileTransferMetadatas(com.fasterxml.jackson.databind.JsonNode jsonNode) This is API function to parse the File Transfer Metadatas from a supplied PUT call response.
Object	getResultSet()
Map<String,net.snowflake.common.core.RemoteStoreFileEncryptionMaterial>	getSrcToMaterialsMap()
Map<String,String>	getSrcToPresignedUrlMap()
Map<?,?>	getStageCredentials()
StageInfo	getStageInfo()
String	getStageLocation()
static Callable<Void>	getUploadFileCallable(StageInfo stage, String srcFilePath, net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata metadata, SnowflakeStorageClient client, SFSession session, String command, InputStream inputStream, boolean sourceFromStream, int parallel, File srcFile, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat) A callable that can be executed in a separate thread using exeuctor service.
static void	renewExpiredToken(SFSession session, String command, SnowflakeStorageClient client) This static method is called when we are handling an expired token exception It retrieves a fresh token from GS and then calls .renew() on the storage client to refresh itself with the new token
static void	throwJCEMissingError(String operation, Exception ex)
static void	uploadWithoutConnection(SnowflakeFileTransferConfig config) Static API function to upload data without JDBC session.

Methods inherited from class net.snowflake.client.jdbc.SFBaseFileTransferAgent

describeColumns, getNextRow, getTotalRows, setCompressSourceFromStream, setDestFileNameForStreamSource, setDestStagePath, setSourceStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SRC_FILE_NAME_FOR_STREAM

public static final String SRC_FILE_NAME_FOR_STREAM

See Also:

Constant Field Values

Constructor Detail

SnowflakeFileTransferAgent

public SnowflakeFileTransferAgent(String command,
                                  SFSession session,
                                  SFStatement statement)
                           throws SnowflakeSQLException

Throws:

SnowflakeSQLException

Method Detail

getStageInfo

public StageInfo getStageInfo()

getStageCredentials

public Map<?,?> getStageCredentials()

getEncryptionMaterial

public List<net.snowflake.common.core.RemoteStoreFileEncryptionMaterial> getEncryptionMaterial()

getSrcToMaterialsMap

public Map<String,net.snowflake.common.core.RemoteStoreFileEncryptionMaterial> getSrcToMaterialsMap()

getSrcToPresignedUrlMap

public Map<String,String> getSrcToPresignedUrlMap()

getStageLocation

public String getStageLocation()

getUploadFileCallable

public static Callable<Void> getUploadFileCallable(StageInfo stage,
                                                   String srcFilePath,
                                                   net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata metadata,
                                                   SnowflakeStorageClient client,
                                                   SFSession session,
                                                   String command,
                                                   InputStream inputStream,
                                                   boolean sourceFromStream,
                                                   int parallel,
                                                   File srcFile,
                                                   net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat)

A callable that can be executed in a separate thread using exeuctor service.

The callable does compression if needed and upload the result to the table's staging area.

Parameters:

stage - information about the stage

srcFilePath - source file path

metadata - file metadata

client - client object used to communicate with c3

session - session object

command - command string

inputStream - null if upload source is file

sourceFromStream - whether upload source is file or stream

parallel - number of threads for parallel uploading

srcFile - source file name

encMat - not null if encryption is required

Returns:

a callable that uploading file to the remote store

getDownloadFileCallable

public static Callable<Void> getDownloadFileCallable(StageInfo stage,
                                                     String srcFilePath,
                                                     String localLocation,
                                                     Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap,
                                                     SnowflakeStorageClient client,
                                                     SFSession session,
                                                     String command,
                                                     int parallel,
                                                     net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                                                     String presignedUrl)

A callable that can be executed in a separate thread using executor service.

The callable download files from a stage location to a local location

Parameters:

stage - stage information

srcFilePath - path that stores the downloaded file

localLocation - local location

fileMetadataMap - file metadata map

client - remote store client

session - session object

command - command string

encMat - remote store encryption material

parallel - number of parallel threads for downloading

presignedUrl - Presigned URL for file download

Returns:

a callable responsible for downloading files

getFileTransferMetadatas

public List<SnowflakeFileTransferMetadata> getFileTransferMetadatas()
                                                             throws SnowflakeSQLException

This is API function to retrieve the File Transfer Metadatas.

NOTE: It only supports PUT on S3/AZURE/GCS

Returns:

The file transfer metadatas for to-be-transferred files.

Throws:

SnowflakeSQLException - if any error occurs

getFileTransferMetadatas

public static List<SnowflakeFileTransferMetadata> getFileTransferMetadatas(com.fasterxml.jackson.databind.JsonNode jsonNode)
                                                                    throws SnowflakeSQLException

This is API function to parse the File Transfer Metadatas from a supplied PUT call response.

NOTE: It only supports PUT on S3/AZURE/GCS (i.e. NOT LOCAL_FS)

Parameters:

jsonNode - JSON doc returned by GS from PUT call

Returns:

The file transfer metadatas for to-be-transferred files.

Throws:

SnowflakeSQLException - if any error occurs

execute

public boolean execute()
                throws SQLException

Description copied from class: SFBaseFileTransferAgent

Run the PUT/GET command, if a command has been set.

Specified by:

execute in class SFBaseFileTransferAgent

Returns:

Whether the operation was completed successfully, and completely.

Throws:

SQLException - for SQL or upload errors

downloadStream

public InputStream downloadStream(String fileName)
                           throws SnowflakeSQLException

Download a file from remote, and return an input stream

Specified by:

downloadStream in class SFBaseFileTransferAgent

Parameters:

fileName - A file on a stage to download.

Returns:

An InputStream for the requested file.

Throws:

SnowflakeSQLException - If the file does not exist, or if an error occurred during transport.

cancel

public void cancel()

uploadWithoutConnection

public static void uploadWithoutConnection(SnowflakeFileTransferConfig config)
                                    throws Exception

Static API function to upload data without JDBC session.

NOTE: This function is developed based on getUploadFileCallable().

Parameters:

config - Configuration to upload a file to cloud storage

Throws:

Exception - if error occurs while data upload.

renewExpiredToken

public static void renewExpiredToken(SFSession session,
                                     String command,
                                     SnowflakeStorageClient client)
                              throws SnowflakeSQLException

This static method is called when we are handling an expired token exception It retrieves a fresh token from GS and then calls .renew() on the storage client to refresh itself with the new token

Parameters:

session - a session object

command - a command to be retried

client - a Snowflake Storage client object

Throws:

SnowflakeSQLException - if any error occurs

extractLocationAndPath

public static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.remoteLocation extractLocationAndPath(String stageLocationPath)

A small helper for extracting location name and path from full location path

Parameters:

stageLocationPath - stage location

Returns:

remoteLocation object

getResultSet

public Object getResultSet()
                    throws SnowflakeSQLException

Throws:

SnowflakeSQLException

getCommandType

public SFBaseFileTransferAgent.CommandType getCommandType()

throwJCEMissingError

public static void throwJCEMissingError(String operation,
                                        Exception ex)
                                 throws SnowflakeSQLException

Throws:

SnowflakeSQLException