Package net.snowflake.ingest.connection

Class RequestBuilder

  • public class RequestBuilder
extends Object
    This class handles constructing the URIs for our requests as well as putting together the payloads we'll be sending
    Author:
    obabarinsa

    • Constructor Detail

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      KeyPair keyPair)
        RequestBuilder - general usage constructor
        Parameters:
        accountName - - the name of the Snowflake account to which we're connecting
        userName - - the username of the entity loading files
        keyPair - - the Public/Private key pair we'll use to authenticate

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      String hostName,
                      KeyPair keyPair,
                      String userAgentSuffix)
        RequestBuilder constructor which uses default schemes, host and port.
        Parameters:
        accountName - - the name of the Snowflake account to which we're connecting
        userName - - the username of the entity loading files
        keyPair - - the Public/Private key pair we'll use to authenticate
        userAgentSuffix - - The suffix part of HTTP Header User-Agent

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      KeyPair keyPair,
                      String schemeName,
                      String hostName,
                      int portNum)
        Constructor to use if not intended to use userAgentSuffix. i.e. User-Agent HTTP header suffix part is null, (The default one is still valid, check out #defaultUserAgent)
        Parameters:
        accountName - - the account name to which we're connecting
        userName - - for whom are we connecting?
        keyPair - - our auth credentials
        schemeName - - are we HTTP or HTTPS?
        hostName - - the host for this snowflake instance
        portNum - - the port number

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      KeyPair keyPair,
                      String schemeName,
                      String hostName,
                      int portNum,
                      String userAgentSuffix)
        RequestBuilder - this constructor is for testing purposes only
        Parameters:
        accountName - - the account name to which we're connecting
        userName - - for whom are we connecting?
        keyPair - - our auth credentials
        schemeName - - are we HTTP or HTTPS?
        hostName - - the host for this snowflake instance
        portNum - - the port number
        userAgentSuffix - - The suffix part of HTTP Header User-Agent

      • RequestBuilder

        public RequestBuilder​(SnowflakeURL url,
                      String userName,
                      KeyPair keyPair,
                      net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient httpClient,
                      String clientName)
        RequestBuilder - constructor used by streaming ingest
        Parameters:
        url - - the Snowflake account to which we're connecting
        userName - - the username of the entity loading files
        keyPair - - the Public/Private key pair we'll use to authenticate
        httpClient - - reference to the http client
        clientName - - name of the client, used to uniquely identify a client if used

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      KeyPair keyPair,
                      String schemeName,
                      String hostName,
                      int portNum,
                      String userAgentSuffix,
                      net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient httpClient,
                      String clientName)
        RequestBuilder - this constructor is for testing purposes only
        Parameters:
        accountName - - the account name to which we're connecting
        userName - - for whom are we connecting?
        keyPair - - our auth credentials
        schemeName - - are we HTTP or HTTPS?
        hostName - - the host for this snowflake instance
        portNum - - the port number
        userAgentSuffix - - The suffix part of HTTP Header User-Agent
        httpClient - - reference to the http client
        clientName - - name of the client, used to uniquely identify a client if used

    • Method Detail

      • generateInsertRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpPost generateInsertRequest​(UUID requestId,
                                                                                                    String pipe,
                                                                                                    List<StagedFileWrapper> files,
                                                                                                    boolean showSkippedFiles)
                                                                                             throws URISyntaxException
        generateInsertRequest - given a table, stage and list of files, make a request for the insert endpoint
        Parameters:
        requestId - a UUID we will use to label this request
        pipe - a fully qualified pipe name
        files - a list of files
        showSkippedFiles - a boolean which returns skipped files when set to true
        Returns:
        a post request with all the data we need
        Throws:
        URISyntaxException - if the URI components provided are improper

      • generateInsertRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpPost generateInsertRequest​(UUID requestId,
                                                                                                    String pipe,
                                                                                                    List<StagedFileWrapper> files,
                                                                                                    boolean showSkippedFiles,
                                                                                                    Optional<InsertFilesClientInfo> clientInfo)
                                                                                             throws URISyntaxException
        generateInsertRequest - given a pipe, list of files and clientInfo, make a request for the insert endpoint
        Parameters:
        requestId - a UUID we will use to label this request
        pipe - a fully qualified pipe name
        files - a list of files
        showSkippedFiles - a boolean which returns skipped files when set to true
        clientInfo -
        Returns:
        a post request with all the data we need
        Throws:
        URISyntaxException - if the URI components provided are improper

      • generateHistoryRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpGet generateHistoryRequest​(UUID requestId,
                                                                                                    String pipe,
                                                                                                    Integer recentSeconds,
                                                                                                    String beginMark)
                                                                                             throws URISyntaxException
        generateHistoryRequest - given a requestId and a pipe, make a history request
        Parameters:
        requestId - a UUID we will use to label this request
        pipe - a fully qualified pipe name
        recentSeconds - history only for items in the recentSeconds window
        beginMark - mark from which history should be fetched
        Throws:
        URISyntaxException - - If the URI components provided are improper

      • generateHistoryRangeRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpGet generateHistoryRangeRequest​(UUID requestId,
                                                                                                         String pipe,
                                                                                                         String startTimeInclusive,
                                                                                                         String endTimeExclusive)
                                                                                                  throws URISyntaxException
        generateHistoryRangeRequest - given a requestId and a pipe, get history for all ingests between time ranges start-end
        Parameters:
        requestId - a UUID we will use to label this request
        pipe - a fully qualified pipe name
        startTimeInclusive - Start time inclusive of scan range, in ISO-8601 format. Missing millisecond part in string will lead to a zero milliseconds. This is a required query parameter, and a 400 will be returned if this query parameter is missing
        endTimeExclusive - End time exclusive of scan range. If this query parameter is missing or user provided value is later than current millis, then current millis is used.
        Returns:
        URI for the insert request
        Throws:
        URISyntaxException

      • generateStreamingIngestPostRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpPost generateStreamingIngestPostRequest​(String payload,
                                                                                                                 String endPoint,
                                                                                                                 String message)
        Generate post request for streaming ingest related APIs
        Parameters:
        payload - POST request payload as string
        endPoint - REST API endpoint
        message - error message if there are failures during HTTP building
        Returns:
        URI for the POST request

      • generateConfigureClientRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpPost generateConfigureClientRequest​(UUID requestID,
                                                                                                             String pipe)
                                                                                                      throws URISyntaxException
        Given a requestId and a pipe, make a configure client request
        Parameters:
        requestID - a UUID we will use to label this request
        pipe - a fully qualified pipe name
        Returns:
        configure client request
        Throws:
        URISyntaxException

      • generateStreamingIngestPostRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpPost generateStreamingIngestPostRequest​(Map<Object,​Object> payload,
                                                                                                                 String endPoint,
                                                                                                                 String message)
        Generate post request for streaming ingest related APIs
        Parameters:
        payload - POST request payload
        endPoint - REST API endpoint
        message - error message if there are failures during HTTP building
        Returns:
        URI for the POST request

      • generateGetClientStatusRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpGet generateGetClientStatusRequest​(UUID requestID,
                                                                                                            String pipe)
                                                                                                     throws URISyntaxException
        Given a requestId and a pipe, make a get client status request
        Parameters:
        requestID - UUID
        pipe - a fully qualified pipe name
        Returns:
        get client status request
        Throws:
        URISyntaxException

      • closeResources

        public void closeResources()
        Closes the resources being used by RequestBuilder object. SecurityManager is one such resource which uses a threadpool which needs to be shutdown once SimpleIngestManager is done interacting with Snowpipe Service (Rest APIs)
        Throws:
        Exception

      • getTelemetryService

        public TelemetryService getTelemetryService()
        Get the telemetry service