Package net.snowflake.ingest

Class SimpleIngestManager

  • All Implemented Interfaces:
    AutoCloseable
    public class SimpleIngestManager
extends Object
implements AutoCloseable
    This class provides a basic, low-level abstraction over the Snowflake Ingest Service REST api

    Usage of this class delegates all exception and state handling to the developer

    Author:
    obabarinsa

    • Constructor Detail

      • SimpleIngestManager

        @Deprecated
public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           KeyPair keyPair)
        Deprecated.
        Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid KeyPair object registered with Snowflake DB

        This method is deprecated, please use the constructor that only requires PrivateKey instead of KeyPair.

        Parameters:
        account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        keyPair - the KeyPair we'll use to sign JWT tokens

      • SimpleIngestManager

        @Deprecated
public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           KeyPair keyPair,
                           String schemeName,
                           String hostName,
                           int port)
        Deprecated.
        Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid KeyPair object registered with Snowflake DB

        This method is deprecated, please use the constructor that only requires PrivateKey instead of KeyPair.

        Parameters:
        account - the account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com account should be testaccount If this is the case, you should use the constructor that accepts hostname as argument
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        keyPair - the KeyPair we'll use to sign JWT tokens
        schemeName - http or https
        hostName - the hostname
        port - the port number

      • SimpleIngestManager

        public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey)
                    throws InvalidKeySpecException,
                           NoSuchAlgorithmException
        Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB

        Note: this method only takes in account parameter and derive the hostname, i.e. testaccount.snowfakecomputing.com. If your deployment is not aws us-west, please use the constructor that accept hostname as argument

        Parameters:
        account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount. If this is the case, you should use the constructor that accepts hostname as argument
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        privateKey - the private key we'll use to sign JWT tokens
        Throws:
        NoSuchAlgorithmException - if can't create key factory by using RSA algorithm
        InvalidKeySpecException - if private key or public key is invalid

      • SimpleIngestManager

        public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           String hostName,
                           KeyPair keyPair,
                           String userAgentSuffix)
        Using this constructor for Builder pattern. KeyPair can be passed in now since we have made @see Utils.createKeyPairFromPrivateKey(PrivateKey) public
        Parameters:
        account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount. If this is the case, you should use the constructor that accepts hostname as argument
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        hostName - the hostname
        keyPair - keyPair associated with the private key used for authentication. See @see Utils.createKeyPairFromPrivateKey(java.security.PrivateKey) to generate KP from p8Key
        userAgentSuffix - user agent suffix we want to add.

      • SimpleIngestManager

        public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey,
                           String schemeName,
                           String hostName,
                           int port)
                    throws NoSuchAlgorithmException,
                           InvalidKeySpecException
        Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB
        Parameters:
        account - the account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        privateKey - the private key we'll use to sign JWT tokens
        schemeName - http or https
        hostName - the hostname i.e. testaccount.us-east-1.azure .snowflakecomputing.com
        port - the port number
        Throws:
        NoSuchAlgorithmException - if can't create key factory by using RSA algorithm
        InvalidKeySpecException - if private key or public key is invalid

    • Method Detail

      • getAccount

        public String getAccount()
        getAccount - Gives back the name of the account that this IngestManager is targeting
        Returns:
        the name of the account

      • getUser

        public String getUser()
        getUser - gives back the user on behalf of which this ingest manager is loading
        Returns:
        the user name

      • getPipe

        public String getPipe()
        getPipe - gives back the pipe which we are using
        Returns:
        the pipe name

      • wrapFilepaths

        public static List<StagedFileWrapper> wrapFilepaths​(Set<String> filenames)
        wrapFilepaths - convenience method to take a list of filenames and produce a list of FileWrappers with unset size
        Parameters:
        filenames - the filenames you want to wrap up
        Returns:
        a corresponding list of StagedFileWrapper objects

      • ingestFiles

        public IngestResponse ingestFiles​(List<StagedFileWrapper> files,
                                  UUID requestId,
                                  boolean showSkippedFiles,
                                  InsertFilesClientInfo clientInfo)
                           throws URISyntaxException,
                                  IOException,
                                  IngestResponseException,
                                  BackOffException
        ingestFiles With Client Info - synchronously sends a request to the ingest service to enqueue these files along with clientSequencer and offSetToken.

        OffsetToken will be atomically persisted on server(Snowflake) side along with files if the clientSequencer added in this request matches with what Snowflake currently has.

        If clientSequencers doesnt match, 400 response code is sent back and no files will be added.

        Parameters:
        files - - list of wrappers around filenames and sizes
        requestId - - a requestId that we'll use to label - if null, we generate one for the user
        showSkippedFiles - - a flag which returns the files that were skipped when set to true.
        clientInfo - - clientSequencer and offsetToken to pass along with files. Can be null.
        Returns:
        an insert response from the server
        Throws:
        URISyntaxException - - if the provided account name was illegal and caused a URI construction failure
        IOException - - if we have some other network failure
        IngestResponseException - - if snowflake encountered error during ingest
        BackOffException - - if we have a 503 response

      • getHistoryRange

        public HistoryRangeResponse getHistoryRange​(UUID requestId,
                                            String startTimeInclusive,
                                            String endTimeExclusive)
                                     throws URISyntaxException,
                                            IOException,
                                            IngestResponseException,
                                            BackOffException
        Pings the service to see the current ingest history for this table
        Parameters:
        requestId - a UUID we use to label the request, if null, one is generated for the user
        startTimeInclusive - Start time inclusive of scan range, in ISO-8601 format. Missing millisecond part in string will lead to a zero milliseconds. This is a required query parameter, and a 400 will be returned if this query parameter is missing
        endTimeExclusive - End time exclusive of scan range. If this query parameter is missing or user provided value is later than current millis, then current millis is used.
        Returns:
        a response showing the available ingest history from the service
        Throws:
        URISyntaxException - if the provided account name was illegal and caused a URI construction failure
        IOException - - if we have some other network failure
        IngestResponseException - - if snowflake encountered error during ingest
        BackOffException - if we have a 503 response

      • close

        public void close()
        Closes the resources associated with this object. Resources cannot be reopened, initialize new instance of this class SimpleIngestManager to reopen and start ingesting/monitoring new data.
        Specified by:
        close in interface AutoCloseable