Package net.snowflake.client.jdbc

Class SnowflakeFileTransferAgent

    • Method Detail

      • getStageInfo

        public StageInfo getStageInfo()

      • getStageCredentials

        public Map<?,​?> getStageCredentials()

      • getEncryptionMaterial

        public List<net.snowflake.common.core.RemoteStoreFileEncryptionMaterial> getEncryptionMaterial()

      • getSrcToMaterialsMap

        public Map<String,​net.snowflake.common.core.RemoteStoreFileEncryptionMaterial> getSrcToMaterialsMap()

      • getSrcToPresignedUrlMap

        public Map<String,​String> getSrcToPresignedUrlMap()

      • getStageLocation

        public String getStageLocation()

      • getUploadFileCallable

        public static Callable<Void> getUploadFileCallable​(StageInfo stage,
                                                   String srcFilePath,
                                                   net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata metadata,
                                                   SnowflakeStorageClient client,
                                                   SFSession session,
                                                   String command,
                                                   InputStream inputStream,
                                                   boolean sourceFromStream,
                                                   int parallel,
                                                   File srcFile,
                                                   net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                                                   String queryId)
        A callable that can be executed in a separate thread using executor service.

        The callable does compression if needed and upload the result to the table's staging area.

        Parameters:
        stage - information about the stage
        srcFilePath - source file path
        metadata - file metadata
        client - client object used to communicate with c3
        session - session object
        command - command string
        inputStream - null if upload source is file
        sourceFromStream - whether upload source is file or stream
        parallel - number of threads for parallel uploading
        srcFile - source file name
        encMat - not null if encryption is required
        queryId - last executed query id (for forwarding in possible exceptions)
        Returns:
        a callable that uploading file to the remote store

      • getDownloadFileCallable

        public static Callable<Void> getDownloadFileCallable​(StageInfo stage,
                                                     String srcFilePath,
                                                     String localLocation,
                                                     Map<String,​net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap,
                                                     SnowflakeStorageClient client,
                                                     SFSession session,
                                                     String command,
                                                     int parallel,
                                                     net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                                                     String presignedUrl,
                                                     String queryId)
        A callable that can be executed in a separate thread using executor service.

        The callable download files from a stage location to a local location

        Parameters:
        stage - stage information
        srcFilePath - path that stores the downloaded file
        localLocation - local location
        fileMetadataMap - file metadata map
        client - remote store client
        session - session object
        command - command string
        encMat - remote store encryption material
        parallel - number of parallel threads for downloading
        presignedUrl - Presigned URL for file download
        queryId - the query ID
        Returns:
        a callable responsible for downloading files

      • getFileTransferMetadatas

        public static List<SnowflakeFileTransferMetadata> getFileTransferMetadatas​(com.fasterxml.jackson.databind.JsonNode jsonNode)
                                                                    throws SnowflakeSQLException
        This is API function to parse the File Transfer Metadatas from a supplied PUT call response.

        NOTE: It only supports PUT on S3/AZURE/GCS (i.e. NOT LOCAL_FS)

        It also assumes there is no active SFSession

        Parameters:
        jsonNode - JSON doc returned by GS from PUT call
        Returns:
        The file transfer metadatas for to-be-transferred files.
        Throws:
        SnowflakeSQLException - if any error occurs

      • getFileTransferMetadatas

        public static List<SnowflakeFileTransferMetadata> getFileTransferMetadatas​(com.fasterxml.jackson.databind.JsonNode jsonNode,
                                                                           String queryId)
                                                                    throws SnowflakeSQLException
        This is API function to parse the File Transfer Metadatas from a supplied PUT call response.

        NOTE: It only supports PUT on S3/AZURE/GCS (i.e. NOT LOCAL_FS)

        It also assumes there is no active SFSession

        Parameters:
        jsonNode - JSON doc returned by GS from PUT call
        queryId - String last executed query id if available
        Returns:
        The file transfer metadatas for to-be-transferred files.
        Throws:
        SnowflakeSQLException - if any error occurs

      • cancel

        public void cancel()

      • uploadWithoutConnection

        @SnowflakeOrgInternalApi
public static void uploadWithoutConnection​(SnowflakeFileTransferConfig config)
                                    throws Exception
        Static API function to upload data without JDBC session.

        NOTE: This function is developed based on getUploadFileCallable().

        Parameters:
        config - Configuration to upload a file to cloud storage
        Throws:
        Exception - if error occurs while data upload.

      • renewExpiredToken

        public static void renewExpiredToken​(SFSession session,
                                     String command,
                                     SnowflakeStorageClient client)
                              throws SnowflakeSQLException
        This static method is called when we are handling an expired token exception It retrieves a fresh token from GS and then calls .renew() on the storage client to refresh itself with the new token
        Parameters:
        session - a session object
        command - a command to be retried
        client - a Snowflake Storage client object
        Throws:
        SnowflakeSQLException - if any error occurs

      • extractLocationAndPath

        public static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.remoteLocation extractLocationAndPath​(String stageLocationPath)
        A small helper for extracting location name and path from full location path
        Parameters:
        stageLocationPath - stage location
        Returns:
        remoteLocation object

      • throwJCEMissingError

        public static void throwJCEMissingError​(String operation,
                                        Exception ex,
                                        String queryId)
                                 throws SnowflakeSQLException
        Handles an InvalidKeyException which indicates that the JCE component is not installed properly
        Parameters:
        operation - a string indicating the operation type, e.g. upload/download
        ex - The exception to be handled
        queryId - last query id if available
        Throws:
        SnowflakeSQLException - throws the error as a SnowflakeSQLException

      • throwNoSpaceLeftError

        public static void throwNoSpaceLeftError​(SFSession session,
                                         String operation,
                                         Exception ex,
                                         String queryId)
                                  throws SnowflakeSQLLoggedException
        For handling IOException: No space left on device when attempting to download a file to a location where there is not enough space. We don't want to retry on this exception.
        Parameters:
        session - the current session
        operation - the operation i.e. GET
        ex - the exception caught
        queryId - the query ID
        Throws:
        SnowflakeSQLLoggedException - if not enough space left on device to download file.