Package net.snowflake.ingest.connection

Class RequestBuilder

  • public class RequestBuilder
extends Object
    This class handles constructing the URIs for our requests as well as putting together the payloads we'll be sending
    Author:
    obabarinsa

    • Constructor Detail

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      Object credential)
        RequestBuilder - general usage constructor
        Parameters:
        accountName - - the name of the Snowflake account to which we're connecting
        userName - - the username of the entity loading files
        credential - - the credential we'll use to authenticate

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      String hostName,
                      Object credential,
                      String userAgentSuffix)
        RequestBuilder constructor which uses default schemes, host and port.
        Parameters:
        accountName - - the name of the Snowflake account to which we're connecting
        userName - - the username of the entity loading files
        credential - - the credential we'll use to authenticate
        userAgentSuffix - - The suffix part of HTTP Header User-Agent

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      Object credential,
                      String schemeName,
                      String hostName,
                      int portNum)
        Constructor to use if not intended to use userAgentSuffix. i.e. User-Agent HTTP header suffix part is null, (The default one is still valid, check out #defaultUserAgent)
        Parameters:
        accountName - - the account name to which we're connecting
        userName - - for whom are we connecting?
        credential - - the credential we'll use to authenticate
        schemeName - - are we HTTP or HTTPS?
        hostName - - the host for this snowflake instance
        portNum - - the port number

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      Object credential,
                      String schemeName,
                      String hostName,
                      int portNum,
                      String userAgentSuffix)
        RequestBuilder - this constructor is for testing purposes only
        Parameters:
        accountName - - the account name to which we're connecting
        userName - - for whom are we connecting?
        credential - - the credential we'll use to authenticate
        schemeName - - are we HTTP or HTTPS?
        hostName - - the host for this snowflake instance
        portNum - - the port number
        userAgentSuffix - - The suffix part of HTTP Header User-Agent

      • RequestBuilder

        public RequestBuilder​(SnowflakeURL url,
                      String userName,
                      Object credential,
                      net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient httpClient,
                      String clientName)
        RequestBuilder - constructor used by streaming ingest
        Parameters:
        url - - the Snowflake account to which we're connecting
        userName - - the username of the entity loading files
        credential - - the credential we'll use to authenticate
        httpClient - - reference to the http client
        clientName - - name of the client, used to uniquely identify a client if used

      • RequestBuilder

        public RequestBuilder​(String accountName,
                      String userName,
                      Object credential,
                      String schemeName,
                      String hostName,
                      int portNum,
                      String userAgentSuffix,
                      net.snowflake.ingest.connection.SecurityManager securityManager,
                      net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient httpClient,
                      String clientName)
        RequestBuilder - this constructor is for testing purposes only
        Parameters:
        accountName - - the account name to which we're connecting
        userName - - for whom are we connecting?
        credential - - our auth credentials, either JWT key pair or OAuth credential
        schemeName - - are we HTTP or HTTPS?
        hostName - - the host for this snowflake instance
        portNum - - the port number
        userAgentSuffix - - The suffix part of HTTP Header User-Agent
        securityManager - - The security manager for authentication
        httpClient - - reference to the http client
        clientName - - name of the client, used to uniquely identify a client if used

    • Method Detail

      • addToken

        public void addToken​(net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpUriRequest request)
        addToken - adds a token to a request
        Parameters:
        request - the URI request

      • generateInsertRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpPost generateInsertRequest​(UUID requestId,
                                                                                                    String pipe,
                                                                                                    List<StagedFileWrapper> files,
                                                                                                    boolean showSkippedFiles)
                                                                                             throws URISyntaxException
        generateInsertRequest - given a table, stage and list of files, make a request for the insert endpoint
        Parameters:
        requestId - a UUID we will use to label this request
        pipe - a fully qualified pipe name
        files - a list of files
        showSkippedFiles - a boolean which returns skipped files when set to true
        Returns:
        a post request with all the data we need
        Throws:
        URISyntaxException - if the URI components provided are improper

      • generateHistoryRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpGet generateHistoryRequest​(UUID requestId,
                                                                                                    String pipe,
                                                                                                    Integer recentSeconds,
                                                                                                    String beginMark)
                                                                                             throws URISyntaxException
        generateHistoryRequest - given a requestId and a pipe, make a history request
        Parameters:
        requestId - a UUID we will use to label this request
        pipe - a fully qualified pipe name
        recentSeconds - history only for items in the recentSeconds window
        beginMark - mark from which history should be fetched
        Throws:
        URISyntaxException - - If the URI components provided are improper

      • generateHistoryRangeRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpGet generateHistoryRangeRequest​(UUID requestId,
                                                                                                         String pipe,
                                                                                                         String startTimeInclusive,
                                                                                                         String endTimeExclusive)
                                                                                                  throws URISyntaxException
        generateHistoryRangeRequest - given a requestId and a pipe, get history for all ingests between time ranges start-end
        Parameters:
        requestId - a UUID we will use to label this request
        pipe - a fully qualified pipe name
        startTimeInclusive - Start time inclusive of scan range, in ISO-8601 format. Missing millisecond part in string will lead to a zero milliseconds. This is a required query parameter, and a 400 will be returned if this query parameter is missing
        endTimeExclusive - End time exclusive of scan range. If this query parameter is missing or user provided value is later than current millis, then current millis is used.
        Returns:
        URI for the insert request
        Throws:
        URISyntaxException

      • generateStreamingIngestPostRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpPost generateStreamingIngestPostRequest​(String payload,
                                                                                                                 String endPoint,
                                                                                                                 String message)
        Generate post request for streaming ingest related APIs
        Parameters:
        payload - POST request payload as string
        endPoint - REST API endpoint
        message - error message if there are failures during HTTP building
        Returns:
        URI for the POST request

      • generateStreamingIngestPostRequest

        public net.snowflake.client.jdbc.internal.apache.http.client.methods.HttpPost generateStreamingIngestPostRequest​(Map<Object,​Object> payload,
                                                                                                                 String endPoint,
                                                                                                                 String message)
        Generate post request for streaming ingest related APIs
        Parameters:
        payload - POST request payload
        endPoint - REST API endpoint
        message - error message if there are failures during HTTP building
        Returns:
        URI for the POST request

      • setRefreshToken

        public void setRefreshToken​(String refreshToken)
        Set refresh token, this method is for refresh token renewal without requiring to restart client. This method only works when the authorization type is OAuth
        Parameters:
        refreshToken - the new refresh token

      • getAuthType

        public String getAuthType()
        Get authorization type

      • refreshToken

        public void refreshToken()
        Refresh token manually

      • closeResources

        public void closeResources()
        Closes the resources being used by RequestBuilder object. SecurityManager is one such resource which uses a threadpool which needs to be shutdown once SimpleIngestManager is done interacting with Snowpipe Service (Rest APIs)
        Throws:
        Exception

      • getTelemetryService

        public TelemetryService getTelemetryService()
        Get the telemetry service