This is Part 2 of an ongoing series of blogs explaining the use cases of Apache HttpClient(Part-1). This blog is going to take you a step ahead and help you know more about the various use cases such as Authentication, Connection Pooling, Cookie Management, GZIP Compression, Multithreading, Content-Encoding, Redirection, and Retries
1. Authentication
It’s important to use secure connections (HTTPS) when sending sensitive information over the network. Apache HttpClient provides several options for authentication, including Basic, Digest, NTLM, and OAuth.
Basic Authentication:
To use Basic authentication, you can set the username and password using the setHeader method of the HttpRequest class. For example:
HttpGet request = new HttpGet("http://www.cleartrip.com/");
String credentials = "username:password";
String encodedCredentials = Base64.encodeBase64String(credentials.getBytes(StandardCharsets.UTF_8));
request.setHeader("Authorization", "Basic " + encodedCredentials);
HttpResponse response = httpClient.execute(request);This sets the Authorization header with the encoded credentials using the Basic authentication scheme. Note that the Base64 class is used to encode the credentials in Base64 format.
Digest Authentication:
To use Digest authentication, you can use the HttpClientBuilder class to configure a DigestScheme and set the credentials using the setCredentials method of the CredentialsProvider class. For example:
HttpClientBuilder builder = HttpClientBuilder.create();
CredentialsProvider provider = new BasicCredentialsProvider();
DigestScheme digestScheme = new DigestScheme();
digestScheme.overrideParamter("realm", "cleartrip.com");
digestScheme.overrideParamter("nonce", "nonceValue");
digestScheme.overrideParamter("algorithm", "MD5");
provider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("username", "password"));
builder.setDefaultCredentialsProvider(provider);
builder.setDefaultAuthSchemeRegistry(
RegistryBuilder.<AuthSchemeProvider>create()
.register(AuthSchemes.DIGEST, new DigestSchemeFactory())
.build()
);
CloseableHttpClient httpClient = builder.build();
HttpGet request = new HttpGet("http://www.cleartrip.com/");
HttpResponse response = httpClient.execute(request);This sets the realm, nonce, and algorithm parameters for the DigestScheme, and sets the credentials using the CredentialsProvider. The setDefaultAuthSchemeRegistry() method is used to register the DigestSchemeFactory for the DIGEST authentication scheme.
NTLM Authentication:
To use NTLM authentication, you can use the HttpClientBuilder class to configure an NTCredentials and set the credentials using the setCredentials() method of the CredentialsProvider class. For example:
HttpClientBuilder builder = HttpClientBuilder.create();
CredentialsProvider provider = new BasicCredentialsProvider();
NTCredentials credentials = new NTCredentials("username", "password", "workstation", "domain");
provider.setCredentials(AuthScope.ANY, credentials);
builder.setDefaultCredentialsProvider(provider);
builder.setDefaultAuthSchemeRegistry(
RegistryBuilder.<AuthSchemeProvider>create()
.register(AuthSchemes.NTLM, new NTLMSchemeFactory())
.build()
);
CloseableHttpClient httpClient = builder.build();
HttpGet request = new HttpGet("http://www.cleartrip.com/");
HttpResponse response = httpClient.execute(request);This sets the NTCredentials using the CredentialsProvider, and registers the NTLMSchemeFactory for the NTLM authentication scheme.
OAuth Authentication:
To use OAuth authentication, you can use a library such as Apache OAuth Client or ScribeJava, which provides OAuth support for Apache HttpClient. These libraries provide OAuth authentication using OAuth 1.0a or OAuth 2.0.
In general, experts recommend using a secure authentication mechanism like OAuth, particularly for APIs and web services that need authentication.
2. Connection Pooling
Connection pooling is an important feature in Apache HttpClient that can help improve performance by reusing existing connections rather than creating new ones for each request. This can save time and resources, particularly in high-traffic scenarios where multiple requests are made to the same server.
To use connection pooling in Apache HttpClient, you can use the PoolingHttpClientConnectionManager class. This class manages a pool of HTTP connections and allows you to configure various parameters such as the maximum number of connections allowed per route, the maximum total number of connections, and the connection timeout.
Here’s an example of how to use connection pooling with Apache HttpClient:
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(200); // maximum number of connections allowed
connectionManager.setDefaultMaxPerRoute(20); // maximum number of connections allowed per route
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.build();
HttpGet request = new HttpGet("http://www.cleartrip.com/");
HttpResponse response = httpClient.execute(request);In this example, we create a PoolingHttpClientConnectionManager object and set the maximum total number of connections to 200 and the maximum number of connections per route to 20. We then create an HttpClient object using the HttpClients.custom() method and set the connection manager using the setConnectionManager() method.
When a request is made using this HttpClient object, Apache HttpClient will attempt to reuse an existing connection from the pool if one is available. If not, it will create a new connection and add it to the pool. This can improve performance by reducing the overhead of creating new connections for each request.
Note that when using connection pooling, it’s important to properly manage and release resources to avoid issues such as connection leaks. You can use the CloseableHttpClient interface to ensure that resources are properly released when they are no longer needed.
You should always release the connections back to the pool after you’re done using them when using connection pooling. This can be done by calling the close() method on the HttpResponse object, which releases the connection back to the pool. Alternatively, you can use the HttpClientUtils.closeQuietly() method to close the response and release the connection back to the pool.
3. Cookies Management
Cookie management is an important feature in Apache HttpClient that allows you to manage cookies sent by servers in response to HTTP requests, and also to send cookies with subsequent requests to the same server.
To manage cookies in Apache HttpClient, you can use the CookieStore and CookieSpec interfaces. The CookieStore interface provides a way to store and retrieve cookies, while the CookieSpec interface provides a way to parse and format cookies according to various cookie specifications such as the Netscape cookie specification or the RFC 6265 cookie specification.
Here’s an example of how to use cookie management with Apache HttpClient:
CookieStore cookieStore = new BasicCookieStore();
CloseableHttpClient httpClient = HttpClients.custom()
.setDefaultCookieStore(cookieStore)
.build();
HttpGet request = new HttpGet("http://www.cleartrip.com/");
HttpResponse response = httpClient.execute(request);
List<Cookie> cookies = cookieStore.getCookies();
for (Cookie cookie : cookies) {
System.out.println("Cookie: " + cookie.getName() + "=" + cookie.getValue());
}In this example, we create a BasicCookieStore object and set it as the default cookie store for the HttpClient object using the setDefaultCookieStore() method. To perform the task, we create and execute an HttpGet object using the HttpClient object. We retrieve the response from the server and add any cookies sent by the server to the cookie store.
We can then retrieve the cookies from the cookie store using the getCookies() method and iterate over them to print out their names and values.
You can also add new cookies to the CookieStore instance using the addCookie() method:
Cookie cookie = new BasicClientCookie("name", "value");
cookieStore.addCookie(cookie);In this example, we create a new Cookie instance and add it to the CookieStore instance using the addCookie() method.
Overall, Apache HttpClient provides robust support for managing cookies in your HTTP requests and responses. By using the CookieStore interface, you can easily manage cookies and maintain stateful information between requests and responses.
4. GZIP Compression
Apache HttpClient supports GZIP compression of HTTP responses. We can reduce the amount of data that needs to be transferred over the network, resulting in faster response times and reduced bandwidth usage.
GZIP compression works by compressing the response body on the server side and sending it to the client in a compressed format. The client then decompresses the response body before processing it.
HttpClient supports GZIP compression in two ways:
1. Automatic decompression: HttpClient can automatically decompress GZIP-encoded responses. HttpClient automatically decompresses the response body and returns it to the application when it receives a response with a “Content-Encoding: gzip” header.
2. Automatic compression: HttpClient can also automatically compress request bodies using GZIP compression. To enable automatic compression, you need to set the “Content-Encoding” header to “gzip” on the request entity. HttpClient will then compress the request body before sending it to the server.
To enable GZIP compression in Apache HttpClient, you can use the HttpClientBuilder class and set the content encoding interceptor using the addInterceptorLast() method. For example:
HttpClientBuilder builder = HttpClientBuilder.create();
builder.addInterceptorLast(new ContentEncodingInterceptor(ContentEncoding.GZIP));
CloseableHttpClient httpClient = builder.build();
HttpGet request = new HttpGet("http://www.cleartrip.com/");
HttpResponse response = httpClient.execute(request);This sets the content encoding interceptor to use GZIP compression. The content encoding interceptor automatically decompresses the GZIP-compressed response before returning it to the client.
Note that not all servers support GZIP compression, and enabling it may increase CPU usage on both client and server. Testing and optimizing your application is important to find the right balance between compression and performance. Some servers may also use other compression algorithms, such as deflate or brotli.
5. Multithreading
Apache HttpClient provides multithreading support that allows you to make multiple HTTP requests concurrently from multiple threads. This can help improve performance by utilizing multiple CPU cores and network connections, and can also reduce the overall response time for the application.
The main class responsible for multithreading support in Apache HttpClient is CloseableHttpClient. CloseableHttpClient is thread-safe and can be shared across multiple threads, allowing multiple requests to be made concurrently.
Here’s an example of how to use multithreading support with Apache HttpClient:
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.build();
ExecutorService executor = Executors.newFixedThreadPool(10);
List<Callable<String>> tasks = new ArrayList<>();
for (int i = 0; i < 10; i++) {
tasks.add(() -> {
HttpGet request = new HttpGet("http://www.cleartrip.com/");
HttpResponse response = httpClient.execute(request);
String responseString = EntityUtils.toString(response.getEntity());
return responseString;
});
}
List<Future<String>> results = executor.invokeAll(tasks);
for (Future<String> result : results) {
System.out.println(result.get());
}
executor.shutdown();
In this example, we create a PoolingHttpClientConnectionManager object and use it to create a CloseableHttpClient object. Then, we create an ExecutorService with 10 threads using Executors.newFixedThreadPool(10). After that, we create a list of Callable tasks that make HTTP requests using the CloseableHttpClient object and return the response as a String. We added ten tasks to the list, and each task made a request to “http://www.cleartrip.com/“.
We then use the invokeAll() method the ExecutorService to execute all the tasks concurrently. This returns a list of Future objects that can be used to retrieve the results of the tasks.
Finally, we iterate over the list of Future objects and call the get() method to retrieve the results. The get() method will block until the result is available. We then print out the result.
When using multithreading with Apache HttpClient, proper resource management, and release is crucial to avoid connection leaks. In this example, we shut down the ExecutorService after all tasks have been completed ensuring that resources are properly released.
Apache HttpClient supports the non-blocking execution of multiple HTTP requests simultaneously, without blocking the current thread. This can significantly improve the performance of your application, especially in high-concurrency environments.
Overall, Apache HttpClient provides robust support for multithreaded execution, allowing you to execute HTTP requests asynchronously and efficiently in a multithreaded environment.
6. Content-Encoding
HTTP responses have a header field called Content-Encoding, which shows what type of compression applied to the response body. Apache HttpClient supports the automatic decoding of compressed response bodies by default. This means that if the server returns a response with a Content-Encoding header, HttpClient will automatically decode the response body based on the specified encoding.
Here’s an example of how to use Apache HttpClient to request a compressed response:
CloseableHttpClient httpClient = HttpClients.createDefault();
HttpGet request = new HttpGet("https://cleartrip.com/compressed-resource");
HttpResponse response = httpClient.execute(request);
Header contentEncodingHeader = response.getEntity().getContentEncoding();
if (contentEncodingHeader != null) {
String encoding = contentEncodingHeader.getValue();
if (encoding.equalsIgnoreCase("gzip")) {
response.setEntity(new GzipDecompressingEntity(response.getEntity()));
} else if (encoding.equalsIgnoreCase("deflate")) {
response.setEntity(new DeflateDecompressingEntity(response.getEntity()));
}
}
String responseBody = EntityUtils.toString(response.getEntity());
In this example, we create an CloseableHttpClient object and use it to execute an HTTP GET request to a URL that returns a compressed response.
After receiving the response, we check the Content-Encoding header to determine the type of compression used. If the header indicates that the response was compressed with gzip or deflate, we use the corresponding GzipDecompressingEntity or DeflateDecompressingEntity class to automatically decompress the response body.
Finally, we convert the response body to a string using EntityUtils.toString.
Note that if the server returns a response with an unsupported Content-Encoding header, HttpClient will throw an exception. You can customize the list of supported encodings using the HttpClientBuilder.setContentDecoderRegistry() method. For example, to add support for the “br” (Brotli) encoding:
HttpClientBuilder builder = HttpClients.custom();
builder.setContentDecoderRegistry(
RegistryBuilder.<ContentDecoder>create()
.register("gzip", new GzipDecompressor())
.register("deflate", new DeflateDecompressor())
.register("br", new BrotliDecoder())
.build()
);
CloseableHttpClient httpClient = builder.build();
In this example, we create a HttpClientBuilder object and use it to set the ContentDecoderRegistry to a new instance of Registry that includes a BrotliDecoder. We then create a CloseableHttpClient object using the HttpClientBuilder.
Overall, Apache HttpClient provides robust support for content encoding, allowing you to automatically decode compressed responses and encode request data in the desired format.
7. Redirection and Retries
In the context of Apache HttpClient, redirection and retries refer to two different concepts:
Redirection
When a server responds to a client request with a redirection status code (such as 301, 302, or 307), it is instructing the client to make a new request to a different URL. Apache HttpClient provides support for handling redirections automatically by following the redirections and sending new requests to the new URL. This behavior can be configured through the HttpClient instance, which has a setRedirectStrategy() method that takes a RedirectStrategy object. The RedirectStrategy interface defines how redirections should be handled, and Apache HttpClient provides several implementations of this interface that can be used out of the box.
Retries
When a client sends a request to a server and the server responds with an error status code (such as 500 or 503), the client may attempt to retry the request. The Apache HttpClient supports automatic retries by resending the request multiple times until it receives a successful response. This behavior can be configured through the HttpClient instance, which has a setRetryHandler() method that takes a HttpRequestRetryHandler object. The HttpRequestRetryHandler interface defines how retries should be handled, and Apache HttpClient provides several implementations of this interface that can be used out of the box.
Both redirection and retries can be useful in certain scenarios, but they can also be problematic if not used carefully. Automatic retries can send duplicate requests to the server and cause issues if the server is not designed to handle them. Similarly, automatic redirection can cause unexpected behavior if the client is not aware of the redirection and does not handle it properly. Therefore, it is important to understand how these features work and to configure them appropriately for your specific use case.
Note that retries and redirections can have a significant impact on the performance and stability of your application, so you should configure them carefully based on your specific use case.
Conclusion
In conclusion, this blog provided valuable insights into the multiple use cases of Apache HttpClient. We have discussed various important features such as authentication, connection pooling, cookie management, GZIP compression, multithreading, content encoding, redirection, and retries. By understanding these use cases, developers can make the most of Apache HttpClient to create efficient and effective HTTP clients that cater to their specific needs. Stay tuned for a more informative blog on this ongoing series as we delve deeper into the world of Apache HttpClient.
View the Session on Apache HttpClient for more knowledge.
Meanwhile, visit Knoldus Blogs to gain more information about different technologies.
