Package net.snowflake.ingest

Class SimpleIngestManager

  • All Implemented Interfaces:
    AutoCloseable
    public class SimpleIngestManager
extends Object
implements AutoCloseable
    This class provides a basic, low-level abstraction over the Snowflake Ingest Service REST api

    Usage of this class delegates all exception and state handling to the developer

    Author:
    obabarinsa

    • Constructor Detail

      • SimpleIngestManager

        @Deprecated
public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           KeyPair keyPair)
        Deprecated.
        Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid KeyPair object registered with Snowflake DB

        This method is deprecated, please use the constructor that only requires PrivateKey instead of KeyPair.

        Parameters:
        account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        keyPair - the KeyPair we'll use to sign JWT tokens

      • SimpleIngestManager

        @Deprecated
public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           KeyPair keyPair,
                           String schemeName,
                           String hostName,
                           int port)
        Deprecated.
        Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid KeyPair object registered with Snowflake DB

        This method is deprecated, please use the constructor that only requires PrivateKey instead of KeyPair.

        Parameters:
        account - the account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com account should be testaccount If this is the case, you should use the constructor that accepts hostname as argument
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        keyPair - the KeyPair we'll use to sign JWT tokens
        schemeName - http or https
        hostName - the hostname
        port - the port number

      • SimpleIngestManager

        public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey)
                    throws InvalidKeySpecException,
                           NoSuchAlgorithmException
        Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB

        Note: this method only takes in account parameter and derive the hostname, i.e. testaccount.snowfakecomputing.com. If your deployment is not aws us-west, please use the constructor that accept hostname as argument

        Parameters:
        account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount. If this is the case, you should use the constructor that accepts hostname as argument
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        privateKey - the private key we'll use to sign JWT tokens
        Throws:
        NoSuchAlgorithmException - if can't create key factory by using RSA algorithm
        InvalidKeySpecException - if private key or public key is invalid

      • SimpleIngestManager

        public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           String hostName,
                           KeyPair keyPair,
                           String userAgentSuffix)
        Using this constructor for Builder pattern. KeyPair can be passed in now since we have made @see Utils.createKeyPairFromPrivateKey(PrivateKey) public
        Parameters:
        account - The account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount. If this is the case, you should use the constructor that accepts hostname as argument
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        hostName - the hostname
        keyPair - keyPair associated with the private key used for authentication. See @see Utils.createKeyPairFromPrivateKey(java.security.PrivateKey) to generate KP from p8Key
        userAgentSuffix - user agent suffix we want to add.

      • SimpleIngestManager

        public SimpleIngestManager​(String account,
                           String user,
                           String pipe,
                           PrivateKey privateKey,
                           String schemeName,
                           String hostName,
                           int port)
                    throws NoSuchAlgorithmException,
                           InvalidKeySpecException
        Constructs a SimpleIngestManager for a given user in a specific account In addition, this also takes takes the target table and source stage Finally, it also requires a valid private key registered with Snowflake DB
        Parameters:
        account - the account into which we're loading Note: account should not include region or cloud provider info. e.g. if host is testaccount.us-east-1.azure .snowflakecomputing.com, account should be testaccount
        user - the user performing this load
        pipe - the fully qualified name of the pipe
        privateKey - the private key we'll use to sign JWT tokens
        schemeName - http or https
        hostName - the hostname i.e. testaccount.us-east-1.azure .snowflakecomputing.com
        port - the port number
        Throws:
        NoSuchAlgorithmException - if can't create key factory by using RSA algorithm
        InvalidKeySpecException - if private key or public key is invalid

    • Method Detail

      • getAccount

        public String getAccount()
        getAccount - Gives back the name of the account that this IngestManager is targeting
        Returns:
        the name of the account

      • getUser

        public String getUser()
        getUser - gives back the user on behalf of which this ingest manager is loading
        Returns:
        the user name

      • getPipe

        public String getPipe()
        getPipe - gives back the pipe which we are using
        Returns:
        the pipe name

      • wrapFilepaths

        public static List<StagedFileWrapper> wrapFilepaths​(Set<String> filenames)
        wrapFilepaths - convenience method to take a list of filenames and produce a list of FileWrappers with unset size
        Parameters:
        filenames - the filenames you want to wrap up
        Returns:
        a corresponding list of StagedFileWrapper objects

      • getHistoryRange

        public HistoryRangeResponse getHistoryRange​(UUID requestId,
                                            String startTimeInclusive,
                                            String endTimeExclusive)
                                     throws URISyntaxException,
                                            IOException,
                                            IngestResponseException,
                                            BackOffException
        Pings the service to see the current ingest history for this table
        Parameters:
        requestId - a UUID we use to label the request, if null, one is generated for the user
        startTimeInclusive - Start time inclusive of scan range, in ISO-8601 format. Missing millisecond part in string will lead to a zero milliseconds. This is a required query parameter, and a 400 will be returned if this query parameter is missing
        endTimeExclusive - End time exclusive of scan range. If this query parameter is missing or user provided value is later than current millis, then current millis is used.
        Returns:
        a response showing the available ingest history from the service
        Throws:
        URISyntaxException - if the provided account name was illegal and caused a URI construction failure
        IOException - - if we have some other network failure
        IngestResponseException - - if snowflake encountered error during ingest
        BackOffException - if we have a 503 response

      • close

        public void close()
        Closes the resources associated with this object. Resources cannot be reopened, initialize new instance of this class SimpleIngestManager to reopen and start ingesting/monitoring new data.
        Specified by:
        close in interface AutoCloseable