Go Web Scraping Quick Start Guide
上QQ阅读APP看书,第一时间看更新

Query parameters

For some HTTP requests, extra information needs to be provided by the client in order to refine the request. This is usually done in two different ways. For HTTP GET requests, there is a defined way to include extra information in a request using the URL. Placing a ? at the end of a URL defines the end of the URL resource, and the next section defines query parameters. These parameters are key-value pairs defining the extra information sent to the server. The key value pairs are written as follows:

key1=value1&key2=value2&key3 ...

You will see this quite commonly when you are performing searches. As a hypothetical example, if you were on a site searching for shoes, you might encounter a paginated results page and the URL might look something like this:

https://buystuff.com/product_search?keyword=shoes&page=1

Notice that the resource is product_search, which is followed by the query parameters for the keyword and the page. This way, you can collect the products from all pages by adjusting the query.

Query parameters are defined by the website. There are no standard parameters that all websites must have, so it will take some investigation on your part depending on the site you are scraping.