A very simple but useful trick !
##The multipage lists
On the web, many websites publish lists over multiple pages. They allow for better browsing and quicker loading. But that makes copying them a hassle.
For this tutorial we will use the website Allflicks US. You will find on the homepage 7,365 movies spread over 295 pages (at the time of writing). 25 movies are shown on each page. Have fun copying that! Scraping them would not be that hard, but there is an easier way.
###Word of warning
- The trick only works with lists which have a menu to select the number of items being shown, along with previous and next buttons.
-
This trick works very well with average-sized lists: around 20,000 items. Over that number, your computer may freeze, as did mine when trying to load a 40,000 item list. A workaround can be found at the end of this tutorial.
###’Inspect Element’
The idea is to display all the 7,365 listed movies on a single page. To do so, right click the selector for the number of displayed items, and choose ‘Inspect Element’.
Inspect Element
Once the code editor of your browser has opened, click on the small arrow at the right of the highlighted line. You should see the screen below:
What we want to change is the ‘value=100’. Edit it with a double click and replace 100 by 7365. If you feel it necessary, you can change the text of the button itself by modifying the other ‘100’, between the option tags. Any text put there will appear directly on the page::
You now only have to select this button to make all the movies appear on a single page !
/!\ Be sure not to have this button selected before modifying it: if you modify the ‘value=100’ button, make sure that you were originally on the ‘value=25’ button or any other.
/!\ It might take a few seconds to load.
Copy/paste
When you have the whole list on a single page, just copy and paste it where you want (Excel, Google Spreadsheet). It might take some time here as well.
We’re done!
###How not to freeze your computer
For bigger lists, you can split the work by limiting the number of items shown on the page: previous and next buttons still work! So a 80,000 list can be copied in 4 chucks of 20,000 using the ‘next’ button.
Leave a Comment