com.cedarsoftware.util

Class UrlUtilities

java.lang.Object

com.cedarsoftware.util.UrlUtilities

public final class UrlUtilities
extends Object

Useful utilities for working with UrlConnections and IO. Anyone using the deprecated api calls for proxying to urls should update to use the new suggested calls. To let the jvm proxy for you automatically, use the following -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Author:

John DeRegnaucourt ([email protected]) & Ken Partlow
Copyright (c) Cedar Software LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Field Summary

Fields

Modifier and Type	Field and Description
static String	COOKIE
static String	COOKIE_VALUE_DELIMITER
static SafeSimpleDateFormat	DATE_FORMAT
static char	DOT
static String	EXPIRES
static TrustManager[]	NAIVE_TRUST_MANAGER
static HostnameVerifier	NAIVE_VERIFIER
static SSLSocketFactory	naiveSSLSocketFactory
static char	NAME_VALUE_SEPARATOR
static String	PATH
static String	SET_COOKIE
static String	SET_COOKIE_SEPARATOR

Method Summary

Methods

Modifier and Type	Method and Description
static void	disconnect(HttpURLConnection c)
static URL	getActualUrl(String url)
static URLConnection	getConnection(String url, boolean input, boolean output, boolean cache)
static URLConnection	getConnection(URL url, boolean input, boolean output, boolean cache)
static URLConnection	getConnection(URL url, Map inCookies, boolean input, boolean output, boolean cache, boolean allowAllCerts) Gets a connection from a url.
static URLConnection	getConnection(URL url, Map inCookies, boolean input, boolean output, boolean cache, Proxy proxy, boolean allowAllCerts) Deprecated. As of release 1.13.0, replaced by getConnection(java.net.URL, java.util.Map, boolean, boolean, boolean, boolean)
static URLConnection	getConnection(URL url, Map inCookies, boolean input, boolean output, boolean cache, Proxy proxy, SSLSocketFactory factory, HostnameVerifier verifier) Deprecated. As of release 1.13.0, replaced by getConnection(java.net.URL, java.util.Map, boolean, boolean, boolean, boolean)
static URLConnection	getConnection(URL url, String server, int port, Map inCookies, boolean input, boolean output, boolean cache, boolean allowAllCerts) Deprecated. As of release 1.13.0, replaced by getConnection(java.net.URL, java.util.Map, boolean, boolean, boolean, boolean)
static byte[]	getContentFromUrl(String url) Get content from the passed in URL.
static byte[]	getContentFromUrl(String url, Map inCookies, Map outCookies) Get content from the passed in URL.
static byte[]	getContentFromUrl(String url, Map inCookies, Map outCookies, boolean allowAllCerts)
static byte[]	getContentFromUrl(String url, Map inCookies, Map outCookies, Proxy proxy, boolean allowAllCerts) Deprecated. As of release 1.13.0, replaced by getConnection(String, boolean, boolean, boolean)
static byte[]	getContentFromUrl(String url, Map inCookies, Map outCookies, Proxy proxy, SSLSocketFactory factory, HostnameVerifier verifier) Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String)
static byte[]	getContentFromUrl(String url, Proxy proxy) Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String)
static byte[]	getContentFromUrl(String url, Proxy proxy, SSLSocketFactory factory, HostnameVerifier verifier) Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String)
static byte[]	getContentFromUrl(String url, String proxyServer, int port, Map inCookies, Map outCookies, boolean allowAllCerts) Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String, java.util.Map, java.util.Map, boolean)
static byte[]	getContentFromUrl(URL url, boolean allowAllCerts) Get content from the passed in URL.
static byte[]	getContentFromUrl(URL url, Map inCookies, Map outCookies, boolean allowAllCerts) Get content from the passed in URL.
static byte[]	getContentFromUrl(URL url, Map inCookies, Map outCookies, Proxy proxy, boolean allowAllCerts) Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String, java.util.Map, java.util.Map, boolean)
static String	getContentFromUrlAsString(String url) Get content from the passed in URL.
static String	getContentFromUrlAsString(String url, Map inCookies, Map outCookies, boolean trustAllCerts) Get content from the passed in URL.
static String	getContentFromUrlAsString(String url, Proxy proxy) Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String)
static String	getContentFromUrlAsString(String url, String proxyServer, int port, Map inCookies, Map outCookies, boolean ignoreSec) Deprecated. As of release 1.13.0, replaced by getContentFromUrlAsString(String, java.util.Map, java.util.Map, boolean)
static String	getContentFromUrlAsString(URL url, boolean allowAllCerts) Get content from the passed in URL.
static String	getContentFromUrlAsString(URL url, Map inCookies, Map outCookies, boolean trustAllCerts) Get content from the passed in URL.
static String	getCookieDomainFromHost(String host)
static void	getCookies(URLConnection conn, Map store) Retrieves and stores cookies returned by the host on the other side of the open java.net.URLConnection.
static String	getHostName() Deprecated. As of release 1.13.0, replaced by InetAddressUtilities.getHostName()
static String	getUserAgent()
static void	readErrorResponse(URLConnection c)
static void	setCookies(URLConnection conn, Map store) Prior to opening a URLConnection, calling this method will set all unexpired cookies that match the path or subpaths for thi underlying URL The connection MUST NOT have been opened method or an IOException will be thrown.
static void	setReferrer(String referrer)
static void	setUserAgent(String userAgent)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SET_COOKIE

public static final String SET_COOKIE

See Also:

Constant Field Values

COOKIE_VALUE_DELIMITER

public static final String COOKIE_VALUE_DELIMITER

See Also:

Constant Field Values

PATH

public static final String PATH

See Also:

Constant Field Values

EXPIRES

public static final String EXPIRES

See Also:

Constant Field Values

DATE_FORMAT

public static final SafeSimpleDateFormat DATE_FORMAT

SET_COOKIE_SEPARATOR

public static final String SET_COOKIE_SEPARATOR

See Also:

Constant Field Values

COOKIE

public static final String COOKIE

See Also:

Constant Field Values

NAME_VALUE_SEPARATOR

public static final char NAME_VALUE_SEPARATOR

See Also:

Constant Field Values

DOT

public static final char DOT

See Also:

Constant Field Values

NAIVE_TRUST_MANAGER

public static final TrustManager[] NAIVE_TRUST_MANAGER

NAIVE_VERIFIER

public static final HostnameVerifier NAIVE_VERIFIER

naiveSSLSocketFactory

public static SSLSocketFactory naiveSSLSocketFactory

Method Detail

setReferrer

public static void setReferrer(String referrer)

setUserAgent

public static void setUserAgent(String userAgent)

getUserAgent

public static String getUserAgent()

readErrorResponse

public static void readErrorResponse(URLConnection c)

disconnect

public static void disconnect(HttpURLConnection c)

getCookies

public static void getCookies(URLConnection conn,
              Map store)

Retrieves and stores cookies returned by the host on the other side of the open java.net.URLConnection.

The connection MUST have been opened using the connect() method or a IOException will be thrown.

Parameters:

conn - a java.net.URLConnection - must be open, or IOException will be thrown

setCookies

public static void setCookies(URLConnection conn,
              Map store)
                       throws IOException

Prior to opening a URLConnection, calling this method will set all unexpired cookies that match the path or subpaths for thi underlying URL

The connection MUST NOT have been opened method or an IOException will be thrown.

Parameters:

conn - a java.net.URLConnection - must NOT be open, or IOException will be thrown

Throws:

IOException - Thrown if conn has already been opened.

getCookieDomainFromHost

public static String getCookieDomainFromHost(String host)

getContentFromUrlAsString

public static String getContentFromUrlAsString(String url)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.

Parameters:

url - URL to hit

Returns:

UTF-8 String read from URL or null in the case of error.

getContentFromUrlAsString

public static String getContentFromUrlAsString(URL url,
                               boolean allowAllCerts)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.

Parameters:

url - URL to hit

allowAllCerts - true to not verify certificates

Returns:

UTF-8 String read from URL or null in the case of error.

getContentFromUrlAsString

public static String getContentFromUrlAsString(String url,
                               Map inCookies,
                               Map outCookies,
                               boolean trustAllCerts)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.

Parameters:

url - URL to hit

inCookies - Map of session cookies (or null if not needed)

outCookies - Map of session cookies (or null if not needed)

trustAllCerts - if true, SSL connection will always be trusted.

Returns:

String of content fetched from URL.

getContentFromUrlAsString

public static String getContentFromUrlAsString(URL url,
                               Map inCookies,
                               Map outCookies,
                               boolean trustAllCerts)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.

Parameters:

url - URL to hit

inCookies - Map of session cookies (or null if not needed)

outCookies - Map of session cookies (or null if not needed)

trustAllCerts - if true, SSL connection will always be trusted.

Returns:

String of content fetched from URL.

getContentFromUrl

public static byte[] getContentFromUrl(String url)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].

Parameters:

url - URL to hit

Returns:

byte[] read from URL or null in the case of error.

getContentFromUrl

public static byte[] getContentFromUrl(URL url,
                       boolean allowAllCerts)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].

Parameters:

url - URL to hit

Returns:

byte[] read from URL or null in the case of error.

getContentFromUrl

public static byte[] getContentFromUrl(String url,
                       Map inCookies,
                       Map outCookies,
                       boolean allowAllCerts)

getContentFromUrl

public static byte[] getContentFromUrl(URL url,
                       Map inCookies,
                       Map outCookies,
                       boolean allowAllCerts)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].

Parameters:

url - URL to hit

inCookies - Map of session cookies (or null if not needed)

outCookies - Map of session cookies (or null if not needed)

allowAllCerts - override certificate validation?

Returns:

byte[] of content fetched from URL.

getContentFromUrl

public static byte[] getContentFromUrl(String url,
                       Map inCookies,
                       Map outCookies)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].

Parameters:

url - URL to hit

inCookies - Map of session cookies (or null if not needed)

outCookies - Map of session cookies (or null if not needed)

Returns:

byte[] of content fetched from URL.

getConnection

public static URLConnection getConnection(String url,
                          boolean input,
                          boolean output,
                          boolean cache)
                                   throws IOException

Parameters:

url -

input -

output -

cache -

Returns:

Throws:

IOException

getConnection

public static URLConnection getConnection(URL url,
                          boolean input,
                          boolean output,
                          boolean cache)
                                   throws IOException

Parameters:

url -

input -

output -

cache -

Returns:

UrlConnection

Throws:

IOException

getConnection

public static URLConnection getConnection(URL url,
                          Map inCookies,
                          boolean input,
                          boolean output,
                          boolean cache,
                          boolean allowAllCerts)
                                   throws IOException

Gets a connection from a url. All getConnection calls should go through this code.

Parameters:

url -

inCookies -

input -

output -

cache -

Returns:

URLConnection

Throws:

IOException

getActualUrl

public static URL getActualUrl(String url)
                        throws MalformedURLException

Throws:

MalformedURLException

getHostName

@Deprecated
public static String getHostName()

Deprecated. As of release 1.13.0, replaced by InetAddressUtilities.getHostName()

Returns:

getConnection

@Deprecated
public static URLConnection getConnection(URL url,
                                     Map inCookies,
                                     boolean input,
                                     boolean output,
                                     boolean cache,
                                     Proxy proxy,
                                     boolean allowAllCerts)
                                   throws IOException

Deprecated. As of release 1.13.0, replaced by getConnection(java.net.URL, java.util.Map, boolean, boolean, boolean, boolean)

Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url -

inCookies -

input -

output -

cache -

proxy -

Returns:

URLConnection

Throws:

IOException

getConnection

@Deprecated
public static URLConnection getConnection(URL url,
                                     String server,
                                     int port,
                                     Map inCookies,
                                     boolean input,
                                     boolean output,
                                     boolean cache,
                                     boolean allowAllCerts)
                                   throws IOException

Deprecated. As of release 1.13.0, replaced by getConnection(java.net.URL, java.util.Map, boolean, boolean, boolean, boolean)

Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url -

server -

port -

inCookies -

input -

output -

cache -

allowAllCerts -

Returns:

URLConnection

Throws:

IOException

getConnection

@Deprecated
public static URLConnection getConnection(URL url,
                                     Map inCookies,
                                     boolean input,
                                     boolean output,
                                     boolean cache,
                                     Proxy proxy,
                                     SSLSocketFactory factory,
                                     HostnameVerifier verifier)
                                   throws IOException

Deprecated. As of release 1.13.0, replaced by getConnection(java.net.URL, java.util.Map, boolean, boolean, boolean, boolean)

Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url -

inCookies -

input -

output -

cache -

proxy -

factory -

verifier -

Returns:

URLConnection

Throws:

IOException

getContentFromUrl

@Deprecated
public static byte[] getContentFromUrl(String url,
                                  Proxy proxy)

Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String)

Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].

Parameters:

url - URL to hit

proxy - proxy to use to create connection

Returns:

byte[] read from URL or null in the case of error.

getContentFromUrl

@Deprecated
public static byte[] getContentFromUrl(String url,
                                  Proxy proxy,
                                  SSLSocketFactory factory,
                                  HostnameVerifier verifier)

Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String)

Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].

Parameters:

url - URL to hit

proxy - Proxy server to create connection (or null if not needed)

factory - custom SSLSocket factory (or null if not needed)

verifier - custom Hostnameverifier (or null if not needed)

Returns:

byte[] of content fetched from URL.

getContentFromUrl

@Deprecated
public static byte[] getContentFromUrl(String url,
                                  Map inCookies,
                                  Map outCookies,
                                  Proxy proxy,
                                  SSLSocketFactory factory,
                                  HostnameVerifier verifier)

Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url - URL to hit

inCookies -

outCookies -

proxy - Proxy server to create connection (or null if not needed)

factory - custom SSLSocket factory (or null if not needed)

verifier - custom Hostnameverifier (or null if not needed)

Returns:

byte[] of content fetched from URL.

getContentFromUrlAsString

@Deprecated
public static String getContentFromUrlAsString(String url,
                                          Proxy proxy)

Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url - URL to hit

proxy - proxy to use to create connection

Returns:

String read from URL or null in the case of error.

getContentFromUrl

@Deprecated
public static byte[] getContentFromUrl(URL url,
                                  Map inCookies,
                                  Map outCookies,
                                  Proxy proxy,
                                  boolean allowAllCerts)

Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String, java.util.Map, java.util.Map, boolean)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url - URL to hit

inCookies - Map of session cookies (or null if not needed)

outCookies - Map of session cookies (or null if not needed)

proxy - Proxy server to create connection (or null if not needed)

Returns:

byte[] of content fetched from URL.

getContentFromUrl

@Deprecated
public static byte[] getContentFromUrl(String url,
                                  Map inCookies,
                                  Map outCookies,
                                  Proxy proxy,
                                  boolean allowAllCerts)

Deprecated. As of release 1.13.0, replaced by getConnection(String, boolean, boolean, boolean)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url - URL to hit

inCookies - Map of session cookies (or null if not needed)

outCookies - Map of session cookies (or null if not needed)

proxy - Proxy server to create connection (or null if not needed)

Returns:

byte[] of content fetched from URL.

getContentFromUrl

@Deprecated
public static byte[] getContentFromUrl(String url,
                                  String proxyServer,
                                  int port,
                                  Map inCookies,
                                  Map outCookies,
                                  boolean allowAllCerts)

Deprecated. As of release 1.13.0, replaced by getContentFromUrl(String, java.util.Map, java.util.Map, boolean)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url - URL to hit

proxyServer - String named of proxy server

port - port to access proxy server

inCookies - Map of session cookies (or null if not needed)

outCookies - Map of session cookies (or null if not needed)

allowAllCerts - if true, SSL connection will always be trusted.

Returns:

byte[] of content fetched from URL.

getContentFromUrlAsString

@Deprecated
public static String getContentFromUrlAsString(String url,
                                          String proxyServer,
                                          int port,
                                          Map inCookies,
                                          Map outCookies,
                                          boolean ignoreSec)

Deprecated. As of release 1.13.0, replaced by getContentFromUrlAsString(String, java.util.Map, java.util.Map, boolean)

Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should alwasy include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg

Parameters:

url - URL to hit

proxyServer - String named of proxy server

port - port to access proxy server

inCookies - Map of session cookies (or null if not needed)

outCookies - Map of session cookies (or null if not needed)

ignoreSec - if true, SSL connection will always be trusted.

Returns:

String of content fetched from URL.