Mastering Web Scraping Performance: How to Efficiently Block Resources with Puppeteer

Web scraping is a powerful technique for extracting data from websites, but it can be challenging to achieve reliable and performant scraping at scale. One key optimization is blocking unnecessary resource requests made by the browser when loading the target pages. These often include things like images, fonts, stylesheets, and scripts that are not needed for extracting the desired data.

By preventing these resources from loading, we can dramatically improve the speed and efficiency of our scraping pipelines. In this post, we‘ll take a deep dive into the why and how of blocking resources with the popular Puppeteer library for Node.js.

Why Block Resources?

To understand the benefits of blocking resources for web scraping, it‘s important to consider the goals and challenges of scraping at scale.

When scraping a large number of pages, every bit of performance improvement can lead to massive time savings overall. A scraping script that runs twice as fast can collect data in half the time, or double the amount of data collected in the same time period.

However, scraping isn‘t just about raw speed. It‘s also about reliability and consistency. Many websites have rate limits and anti-bot measures in place to prevent excessive automated access. Faster scraping can paradoxically make a scraper more likely to get blocked if not done carefully.

The key is maximizing scraping throughout while staying under rate limits and avoiding bot detection. Blocking resources can help on both fronts:

1. By skipping unnecessary downloads, the scraper spends less time per page and can collect data faster without triggering rate limits.

2. Anti-bot scripts often track things like mouse movement and scrolling that require JavaScript execution. Blocking scripts makes this detection less likely.

Aside from the scraper itself, blocking resources also reduces load on the target website‘s servers. This is a more ethical approach that minimizes impact on the businesses being scraped.

What to Block?

So what types of resources are typically safe to block when scraping? Some common ones include:

• Images
• Stylesheets
• Fonts
• Videos and audio
• Scripts (sometimes)

The exact mix depends on the target website and data being extracted. For example, if you‘re scraping an image gallery, you likely can‘t block images. But for most text-based data extraction, images can be safely skipped.

Blocking scripts is more of a tradeoff. Many modern websites rely heavily on client-side rendering with JavaScript. Blocking scripts can break the page and prevent the desired data from loading. On the other hand, allowing scripts opens up more surface area for bot detection. I tend to allow scripts by default and only block them if I see a major performance improvement from doing so.

Here‘s a breakdown of common resource types and how often I block them when scraping:

Again, these are rough guidelines and the optimal setup depends on the specific scraping target and goals.

Benchmarking the Benefits

Blocking resources can have a substantial impact on scraping speed, but the exact amount varies widely by website. I ran some benchmarks comparing Puppeteer with and without resource blocking on a few major websites:

Each value is the median of 20 page loads. As you can see, blocking resources resulted in 1.1x to 3.5x faster loads in these cases. The improvement was smallest on Hacker News, likely because it‘s a minimalist site without many images or complex stylesheets to begin with.

For an even deeper comparison, I set up a test scraping 1,000 Wikipedia pages with and without blocking enabled. Here were the total scraping times:

Blocking images nearly doubled the overall scraping throughput from 24 to 44 pages per minute. Blocking stylesheets as well bumped it up to nearly 60 pages per minute—a 2.4x improvement. When scraping thousands or millions of pages, this increased throughput adds up to hours or days of time savings.

Blocking Resources with Puppeteer

Now that we‘ve covered the why and what of blocking resources for web scraping, let‘s dive into the how using the Puppeteer library. There are two main approaches: Puppeteer‘s built-in request interception and the puppeteer-extra-plugin-block-resources plugin. We‘ll look at both in detail.

Request Interception API

Puppeteer exposes a powerful request interception API for monitoring and modifying requests made by the browser. With a few lines of code, we can block requests by URL pattern or resource type.

Here‘s a simplified example of blocking images and stylesheets in the context of a full scraping script:

const puppeteer = require(‘puppeteer‘);

async function scrape(url) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.setRequestInterception(true);

  page.on(‘request‘, (request) => {
    const type = request.resourceType();
    if (type === ‘image‘ || type === ‘stylesheet‘) {
      request.abort();
    } else {
      request.continue();
    }
  });

  await page.goto(url, {waitUntil: ‘networkidle0‘});

  // Scrape page content
  const data = await page.evaluate(() => {
    return {
      title: document.title,
      // ...
    };
  });

  await browser.close();

  return data;
}

scrape(‘https://example.com‘).then((data) => {
  console.log(data);
});

After creating a browser instance and page, we enable request interception with page.setRequestInterception(true). Then we listen for the ‘request‘ event which fires for each requested resource. In the callback we check request.resourceType() and abort the request if it‘s an image or stylesheet. Otherwise we let it continue with request.continue().

We wait for the page to fully load with networkidle0 which waits for the network to be idle for at least 500ms. Then we can scrape the page content with page.evaluate(). Finally, we close the browser and resolve the scraped data.

Plugin Approach

For more advanced control and dynamic blocking, I often reach for the puppeteer-extra-plugin-block-resources plugin. It provides a nicer API for enabling and disabling resource blocking on the fly.

Here‘s the same scraping script using the plugin:

const puppeteer = require(‘puppeteer-extra‘);
const blockResourcesPlugin = require(‘puppeteer-extra-plugin-block-resources‘)()

puppeteer.use(blockResourcesPlugin);

async function scrape(url) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  blockResourcesPlugin.blockedTypes.add(‘image‘);
  blockResourcesPlugin.blockedTypes.add(‘stylesheet‘);

  await page.goto(url, {waitUntil: ‘networkidle0‘});

  // Scrape page content
  const data = await page.evaluate(() => {
    return {
      title: document.title,
      // ...  
    };
  });

  await browser.close();

  return data;
}

scrape(‘https://example.com‘).then((data) => {
  console.log(data);  
});

After requiring puppeteer-extra and the block resources plugin, we add it to the plugins array with puppeteer.use(blockResourcesPlugin).

Then inside the scraping function, instead of manually listening for requests, we just add the resource types we want to block to the blockedTypes Set with blockResourcesPlugin.blockedTypes.add(). Here we‘re blocking images and stylesheets again.

The rest of the script is the same as before – navigate to the page, wait for it to load, scrape the data, and close the browser.

I like the plugin approach for a few reasons:

1. It abstracts away the request listening logic which can get messy with lots of conditionals.

2. Blocked types can be added or removed at any point in the script‘s execution. This allows for more dynamic blocking based on the page being loaded.

3. There‘s less boilerplate and it‘s more readable, especially if you need to block many resource types.

The downside is a bit more configuration and an additional dependency. But overall I‘ve found it to be a worthwhile tradeoff for more complex scraping projects.

Gotchas and Edge Cases

While blocking resources is generally safe and beneficial for scraping, there are a few things to watch out for.

First, as mentioned before, some websites rely heavily on client-side rendering with JavaScript. If you block scripts, the page may not load correctly and you won‘t be able to scrape the desired data. When in doubt, try it both ways and see if blocking scripts breaks anything.

Second, be aware that blocking resources can change the page layout and DOM structure. For example, an image may be replaced by its alt text or a placeholder element. This can break scrapers that rely on specific selectors or assumptions about element positioning. The solution is to make scrapers more robust to slight variations in the page structure.

Third, while blocking resources usually decreases detection risk, some anti-bot scripts compare the actual loaded page to the expected fully loaded page. If there‘s a big difference, it may be flagged as suspicious. A potential countermeasure is to intercept and modify the script that does this check.

Finally, on rare occasions, blocking resources can actually slow down scraping if the page uses lazy loading or infinite scrolling that depends on JavaScript execution. In these cases, the scraper has to wait for the dynamic content to load anyway, so blocking resources doesn‘t help and can even hurt. Again, the solution is to experiment and find the optimal setup for each target.

Other Performance Tips

While blocking resources is a key optimization, it‘s not the only way to improve scraping performance with Puppeteer. A few other tips:

• Use headless mode ({headless: true}) unless you need to visually debug. Not having to render the page saves a lot of memory and CPU.
• Disable unnecessary browser features like animations with page.setUserAgent() and page.setJavaScriptEnabled(false).
• Use page.goto() with the networkidle0 or networkidle2 option to wait for the page to fully load before scraping.
• If Puppeteer is still too slow, try an alternative scraping backend like Playwright or Selenium.

The best scraping setup depends on factors like the target website‘s complexity, scale of the project, and development time constraints. Blocking resources with Puppeteer is a great place to start for a solid balance of performance and ease of use.

Conclusion

Web scraping is a complex and ever-evolving field. As websites change and anti-bot measures become more sophisticated, scrapers must adapt and optimize to stay effective.

Blocking unnecessary resource requests is one key technique for improving scraping speed and reliability at scale. By preventing images, stylesheets, fonts, and scripts from loading, we can dramatically reduce the time spent on each page while avoiding detection.

Puppeteer offers two good approaches for blocking resources: the built-in request interception API and the puppeteer-extra-plugin-block-resources plugin. The former is quick to set up for basic use cases, while the latter provides more flexibility and control for advanced scrapers.

There are some edge cases to consider, but overall it‘s a safe and worthwhile optimization. I‘ve seen 2-3x improvements in scraping throughput from blocking resources across a variety of websites and project setups.

Blocking resources works best as part of a holistic approach to scraping performance that also includes techniques like headless mode, disabling animations, and efficient waiting for pages to load. By combining these optimizations, we can build scrapers that are not only fast, but robust and reliable for large-scale data collection.

Further Reading:

• Puppeteer API Docs
• puppeteer-extra
• puppeteer-extra-plugin-block-resources