Package com.cedarsoftware.util

Class UrlUtilities

java.lang.Object
com.cedarsoftware.util.UrlUtilities
public final class UrlUtilities extends Object
Useful utilities for working with UrlConnections and IO.

Proxy Configuration

Anyone using the deprecated api calls for proxying to urls should update to use the new suggested calls. To let the jvm proxy for you automatically, use the following -D parameters:

  • http.proxyHost
  • http.proxyPort (default: 80)
  • http.nonProxyHosts (should always include localhost)
  • https.proxyHost
  • https.proxyPort

Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Security Configuration

UrlUtilities provides configurable security controls to prevent various attack vectors including SSRF (Server-Side Request Forgery), resource exhaustion, and cookie injection attacks. All security features are disabled by default for backward compatibility.

Security controls can be enabled via system properties:

  • urlutilities.security.enabled=false — Master switch for all security features
  • urlutilities.max.download.size=0 — Max download size in bytes (0=disabled, default=100MB when enabled)
  • urlutilities.max.content.length=0 — Max Content-Length header value (0=disabled, default=500MB when enabled)
  • urlutilities.allow.internal.hosts=true — Allow access to internal/local hosts (default=true)
  • urlutilities.allowed.protocols=http,https,ftp — Comma-separated list of allowed protocols
  • urlutilities.strict.cookie.domain=false — Enable strict cookie domain validation (default=false)

Security Features

  • SSRF Protection: Validates protocols and optionally blocks internal host access
  • Resource Exhaustion Protection: Limits download sizes and content lengths
  • Cookie Security: Validates cookie domains to prevent hijacking
  • Protocol Restriction: Configurable allowed protocols list

Usage Example


 // Enable security with custom limits
 System.setProperty("urlutilities.security.enabled", "true");
 System.setProperty("urlutilities.max.download.size", "50000000"); // 50MB
 System.setProperty("urlutilities.allow.internal.hosts", "false");
 System.setProperty("urlutilities.allowed.protocols", "https");

 // These will now enforce security controls
 byte[] data = UrlUtilities.readBytesFromUrl(url);
Author:
Ken Partlow, John DeRegnaucourt ([email protected])
Copyright (c) Cedar Software LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

License

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

  • Field Details

    • userAgent

      public static final ThreadLocal<String> userAgent

    • referrer

      public static final ThreadLocal<String> referrer

    • PATH

      public static final String PATH
      See Also:

    • EXPIRES

      public static final String EXPIRES
      See Also:

    • DATE_FORMAT

      public static final SafeSimpleDateFormat DATE_FORMAT

    • NAME_VALUE_SEPARATOR

      public static final char NAME_VALUE_SEPARATOR
      See Also:

    • DOT

      public static final char DOT
      See Also:

    • NAIVE_TRUST_MANAGER

      @Deprecated public static final TrustManager[] NAIVE_TRUST_MANAGER
      Deprecated.
      This creates a serious security vulnerability. Use proper certificate validation.
      ⚠️ SECURITY WARNING ⚠️ This TrustManager accepts ALL SSL certificates without verification, including self-signed, expired, or certificates from unauthorized Certificate Authorities. This completely disables SSL/TLS certificate validation and makes connections vulnerable to man-in-the-middle attacks. DO NOT USE IN PRODUCTION - Only suitable for development/testing against known safe endpoints. For production use, consider: 1. Use proper CA-signed certificates 2. Import self-signed certificates into a custom TrustStore 3. Use certificate pinning for additional security 4. Implement custom TrustManager with proper validation logic

    • NAIVE_VERIFIER

      @Deprecated public static final HostnameVerifier NAIVE_VERIFIER
      Deprecated.
      This creates a serious security vulnerability. Use proper hostname verification.
      ⚠️ SECURITY WARNING ⚠️ This HostnameVerifier accepts ALL hostnames without verification, completely disabling hostname verification for SSL/TLS connections. This makes connections vulnerable to man-in-the-middle attacks where an attacker presents a valid certificate for a different domain. DO NOT USE IN PRODUCTION - Only suitable for development/testing against known safe endpoints.

    • naiveSSLSocketFactory

      protected static SSLSocketFactory naiveSSLSocketFactory

  • Method Details

    • clearGlobalUserAgent

      public static void clearGlobalUserAgent()

    • clearGlobalReferrer

      public static void clearGlobalReferrer()

    • setReferrer

      public static void setReferrer(String referer)

    • getReferrer

      public static String getReferrer()

    • setUserAgent

      public static void setUserAgent(String agent)

    • getUserAgent

      public static String getUserAgent()

    • setDefaultConnectTimeout

      public static void setDefaultConnectTimeout(int millis)

    • setDefaultReadTimeout

      public static void setDefaultReadTimeout(int millis)

    • getDefaultConnectTimeout

      public static int getDefaultConnectTimeout()

    • getDefaultReadTimeout

      public static int getDefaultReadTimeout()

    • setMaxDownloadSize

      public static void setMaxDownloadSize(long maxSizeBytes)
      Set the maximum download size limit for URL content fetching operations. This prevents memory exhaustion attacks from maliciously large downloads.
      Parameters:
      maxSizeBytes - Maximum download size in bytes (default: 100MB)

    • getMaxDownloadSize

      public static long getMaxDownloadSize()
      Get the current maximum download size limit. Returns the configured system property value if available, otherwise the programmatically set value.
      Returns:
      Maximum download size in bytes

    • setMaxContentLength

      public static void setMaxContentLength(int maxLengthBytes)
      Set the maximum Content-Length header value that will be accepted. This prevents acceptance of responses claiming to be larger than reasonable limits.
      Parameters:
      maxLengthBytes - Maximum Content-Length in bytes (default: 500MB)

    • getMaxContentLength

      public static int getMaxContentLength()
      Get the current maximum Content-Length header limit. Returns the configured system property value if available, otherwise the programmatically set value.
      Returns:
      Maximum Content-Length in bytes

    • readErrorResponse

      public static void readErrorResponse(URLConnection c)

    • disconnect

      public static void disconnect(HttpURLConnection c)

    • getCookies

      public static void getCookies(URLConnection conn, Map<String,Map<String,Map<String,String>>> store)
      Retrieves and stores cookies returned by the host on the other side of the open java.net.URLConnection.

      The connection MUST have been opened using the connect() method or a IOException will be thrown.

      Parameters:
      conn - a java.net.URLConnection - must be open, or IOException will be thrown

    • setCookies

      public static void setCookies(URLConnection conn, Map<String,Map<String,Map<String,String>>> store)
      Prior to opening a URLConnection, calling this method will set all unexpired cookies that match the path or subpaths for thi underlying URL

      The connection MUST NOT have been opened method or an IOException will be thrown.

      Parameters:
      conn - a java.net.URLConnection - must NOT be open, or IOException will be thrown
      Throws:
      IOException - if the connection has already been opened (thrown as unchecked)

    • getCookieDomainFromHost

      public static String getCookieDomainFromHost(String host)

    • getContentFromUrlAsString

      public static String getContentFromUrlAsString(String url)
      Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.
      Parameters:
      url - URL to hit
      Returns:
      UTF-8 String read from URL or null in the case of error.

    • getContentFromUrlAsString

      public static String getContentFromUrlAsString(URL url, boolean allowAllCerts)
      Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.
      Parameters:
      url - URL to hit
      allowAllCerts - true to not verify certificates
      Returns:
      UTF-8 String read from URL or null in the case of error.

    • getContentFromUrlAsString

      public static String getContentFromUrlAsString(String url, Map inCookies, Map outCookies, boolean trustAllCerts)
      Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.
      Parameters:
      url - URL to hit
      inCookies - Map of session cookies (or null if not needed)
      outCookies - Map of session cookies (or null if not needed)
      trustAllCerts - if true, SSL connection will always be trusted.
      Returns:
      String of content fetched from URL.

    • getContentFromUrlAsString

      public static String getContentFromUrlAsString(URL url, Map inCookies, Map outCookies, boolean trustAllCerts)
      Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.
      Parameters:
      url - URL to hit
      inCookies - Map of session cookies (or null if not needed)
      outCookies - Map of session cookies (or null if not needed)
      trustAllCerts - if true, SSL connection will always be trusted.
      Returns:
      String of content fetched from URL.

    • getContentFromUrl

      public static byte[] getContentFromUrl(String url)
      Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
      Parameters:
      url - URL to hit
      Returns:
      byte[] read from URL or null in the case of error.

    • getContentFromUrl

      public static byte[] getContentFromUrl(URL url, boolean allowAllCerts)
      Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
      Parameters:
      url - URL to hit
      Returns:
      byte[] read from URL or null in the case of error.

    • getContentFromUrl

      public static byte[] getContentFromUrl(String url, Map inCookies, Map outCookies, boolean allowAllCerts)

    • getContentFromUrl

      public static byte[] getContentFromUrl(URL url, Map inCookies, Map outCookies, boolean allowAllCerts)
      Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
      Parameters:
      url - URL to hit
      inCookies - Map of session cookies (or null if not needed)
      outCookies - Map of session cookies (or null if not needed)
      allowAllCerts - override certificate validation?
      Returns:
      byte[] of content fetched from URL.

    • copyContentFromUrl

      public static void copyContentFromUrl(String url, OutputStream out)
      Convenience method to copy content from a String URL to an output stream.

    • copyContentFromUrl

      public static void copyContentFromUrl(URL url, OutputStream out, Map<String,Map<String,Map<String,String>>> inCookies, Map<String,Map<String,Map<String,String>>> outCookies, boolean allowAllCerts)
      Copy content from a URL to an output stream.

    • getContentFromUrl

      public static byte[] getContentFromUrl(String url, Map inCookies, Map outCookies)
      Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
      Parameters:
      url - URL to hit
      inCookies - Map of session cookies (or null if not needed)
      outCookies - Map of session cookies (or null if not needed)
      Returns:
      byte[] of content fetched from URL.

    • getConnection

      public static URLConnection getConnection(String url, boolean input, boolean output, boolean cache)
      Parameters:
      input - boolean indicating whether this connection will be used for input
      output - boolean indicating whether this connection will be used for output
      cache - boolean allow caching (be careful setting this to true for non-static retrievals).
      Returns:
      URLConnection established URL connection.
      Throws:
      IOException - if an I/O error occurs (thrown as unchecked)
      MalformedURLException - if the URL is invalid (thrown as unchecked)

    • getConnection

      public static URLConnection getConnection(URL url, boolean input, boolean output, boolean cache)
      Parameters:
      input - boolean indicating whether this connection will be used for input
      output - boolean indicating whether this connection will be used for output
      cache - boolean allow caching (be careful setting this to true for non-static retrievals).
      Returns:
      URLConnection established URL connection.

    • getConnection

      public static URLConnection getConnection(URL url, Map inCookies, boolean input, boolean output, boolean cache, boolean allowAllCerts)
      Gets a connection from a url. All getConnection calls should go through this code.
      Parameters:
      inCookies - Supply cookie Map (received from prior setCookies calls from server)
      input - boolean indicating whether this connection will be used for input
      output - boolean indicating whether this connection will be used for output
      cache - boolean allow caching (be careful setting this to true for non-static retrievals).
      Returns:
      URLConnection established URL connection.
      Throws:
      IOException - if an I/O error occurs (thrown as unchecked)

    • getActualUrl

      public static URL getActualUrl(String url)