API Documentation

Pagination

Pagination is a common use-case in ScrapeIN API when requesting results across multiple pages. For certain endpoints, ScrapIN API allows you to automatically retrieve data from multiple pages and concatenate the results into one response.

Single Pages

To request a single page of results, use the page parameter. For example, a request with page=4 will return data from page 4. To find the total number of pages and determine whether a next page is available, inspect the pagination object in the result JSON.

🚧

Infinite Scrolling Pagination

Request types that implement infinite-scrolling pagination use the next_page_token parameter to request the next page of results. In these cases, starting pagination from an explicit page number is not possible.

Multiple Pages

ScrapeIN API provides the max_page parameter to retrieve multiple pages of results and concatenate them into one response, automating workloads such as "get the first 5 pages of results, if they exist." When running a request with max_page set, the main array property of the response contains a concatenation of the array values from all requested pages.

🚧

API Credits and max_page

Each successfully retrieved page incurs an API credit. If the request yields fewer pages than specified, only credits for the actual number of returned pages are charged. For instance, if a request with max_page=5 yields only 3 pages because only 3 pages are available, then only 3 API credits are charged.

Setting a Max Page

When using max_page, ScrapeIN API retrieves the number of pages (if available) specified in the max_page parameter. Results from subsequent pages are concatenated into the main array property of the response. For example, the following request yields data from pages 1, 2, 3, 4, and 5: max_page=5.

Position Parameters

The following parameters provide information about the position of results:

  • position (number): The position of the current result on its page.
  • page (number): The page the current result is taken from.
  • position_overall(number): The position of the current result within all requested pages set in the max_page parameter.

For instance, assuming 10 results per page and a request made with max_page=2, the first result on the page would be position_overall=1, and the last result on page 2 would be position_overall=20.

Setting the Start Page

To set the page to start the multiple-page concatenation from, use the page and max_page parameters together. For example, the following request yields data from pages 2, 3, and 4: page=2&max_page=4.

📘

Limits when using max_page

Note that there are limits when using max_page. For instance, the maximum value of max_page is 5.

📘

Note on output=html

When using max_page in combination with output=html, the resultant HTML returned will consist of each page of HTML, delimited by a horizontal line break (HTML tag).

📘

Note on include_html=true

When using max_page in combination with include_html=true, the html field is returned as an array containing the HTML of each returned page.