{"id":6566,"date":"2025-06-12T13:03:29","date_gmt":"2025-06-12T11:03:29","guid":{"rendered":"https:\/\/chartud.io\/blog\/how-to-schedule-crawls-and-automate-screaming-frog-reports-in-looker-studio\/"},"modified":"2025-09-16T17:00:44","modified_gmt":"2025-09-16T15:00:44","slug":"automating-screaming-frog-looker-crawls","status":"publish","type":"post","link":"https:\/\/chartud.io\/en\/blog\/automating-screaming-frog-looker-crawls\/","title":{"rendered":"How to Schedule Crawls and Automate Screaming Frog Reports in Looker Studio"},"content":{"rendered":"\n

Scheduling crawls in Screaming Frog is very useful, but the magic happens when that automatic data is dumped into a Looker Studio SEO dashboard<\/a>.<\/p>\n\n\n\n

To have a Looker Studio report with updated data from Screaming Frog crawls, you must take a series of steps.<\/p>\n\n\n\n

What are you going to need?<\/strong><\/p>\n\n\n\n

    \n
  1. Have a licensed version of Screaming Frog. <\/li>\n\n\n\n
  2. A Google account for Looker Studio<\/li>\n\n\n\n
  3. A Google Drive account<\/li>\n<\/ol>\n\n\n\n
    <\/div>\n\n\n\n

    You can view all the data from the scheduled Screaming Frog crawls in a Looker Studio template like this one.<\/a><\/strong><\/p>\n\n\n\n

    If you don’t have this template, don’t worry, you can still follow the tutorial.<\/p>\n\n\n\n

    For the template to work correctly, you must have Screaming Frog configured in English, which is the default language.<\/p>\n\n\n\n

    You can schedule daily, weekly, or monthly crawls, and analyze the evolution in a very visual way.<\/p>\n\n\n\n

    Setting all this up may seem complicated, but it isn’t. Just follow the steps. <\/p>\n\n\n\n

    It is worth automating these reports to save hours of time and be able to analyze much better.<\/p>\n\n\n\n

    Let’s look in detail at the steps you have to take:<\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Step 1: Create a Configuration File in Screaming Frog<\/h2>\n\n\n\n

    This involves configuring the crawl with everything you need. I’m going to tell you what I recommend you activate. <\/p>\n\n\n\n

    Go to Configuration > Crawl Config<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    In the Spider – Crawl <\/strong>section, configure the Sitemaps section. Activate the options and add the sitemaps you want to analyze in the crawl. <\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    In Spider – Extraction <\/strong>activate JSON-LS, Schema.org validation, and Google Rich Result Feature Validation.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Now in Content – Duplicates <\/strong>activate \u201cEnable Near Duplicates\u201d and in Spelling & Grammar, activate the options and choose the language or leave it on automatic.<\/p>\n\n\n\n

    Now we jump to User-agent<\/strong>, where I recommend you choose Googlebot (Smartphone).<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    <\/div>\n\n\n\n

    The next step is to connect the APIs <\/strong>that we are going to use: Google Search Console and Page Speed Insights.<\/p>\n\n\n\n

    (You can connect others if you need to, but for the template we use linked above, these are the ones we use).<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    <\/div>\n\n\n\n

    \ud83d\udc49Check out our selection of Looker Studio templates<\/strong><\/a><\/p>\n\n\n\n

    Google Search Console API<\/strong><\/p>\n\n\n\n

    Once you connect it, choose the last 12 months in the date range and in the URL Inspection<\/strong> tab, activate the \u201cEnable URL Inspection\u201d option.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    <\/div>\n\n\n\n

    Page Speed Insights API<\/strong><\/p>\n\n\n\n

    If you have never activated it, you can do it here.<\/a><\/p>\n\n\n\n

    Once activated and the API Key has been entered, check the \u201cAuto Connect on Start\u201d option.<\/p>\n\n\n\n

    In the Metrics<\/strong> tab, you can check all the options (although not all will be used, but you’ll finish sooner).<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    <\/div>\n\n\n\n

    The last thing to configure in this part is within Crawl Analysis<\/strong>.<\/p>\n\n\n\n

    Check that everything is checked and that the \u201cAuto-analyse at End of Crawl\u201d box is checked. <\/p>\n\n\n\n

    This way it will do the crawl analysis automatically.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    <\/div>\n\n\n\n

    Step 2: save the Configuration File<\/h2>\n\n\n\n

    Once you have everything configured, you must save the configuration file.<\/strong><\/p>\n\n\n\n

    In Configuration > Profiles > Save As.<\/p>\n\n\n\n

    Save it in a place where you can find it easily<\/strong> because you are going to need it in step 3.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    <\/div>\n\n\n\n

    Step 3: Schedule the Crawl<\/h2>\n\n\n\n

    Now you are going to schedule the crawl to run when you indicate.<\/p>\n\n\n\n

    Go to File > Scheduling<\/strong><\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Click on +Add<\/em><\/strong> and configure the scheduled crawl:<\/p>\n\n\n\n

    Name the task and the project.<\/p>\n\n\n\n

    Set the frequency for the crawl. The date of the first crawl and when it will be repeated: once, daily, weekly, or monthly. <\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    In Start Options <\/strong>you must leave Spider as Crawler Mode.<\/p>\n\n\n\n

    In Crawl Seed<\/strong> include the domain of the website you want to analyze.<\/p>\n\n\n\n

    In Crawl Config<\/strong> include the file you configured in step 1 and saved in step 2.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    In API<\/strong>, check Google Search Console and PageSpeed Insights, which are the APIs you configured in step 1.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    In Export:<\/strong><\/p>\n\n\n\n

    Check the \u201cHeadless\u201d option.<\/p>\n\n\n\n

    In Google Drive Account connect your Google Drive account where the Google Sheets with the crawl data will be created.<\/p>\n\n\n\n

    \u26a0Important:<\/strong> the Google account of this Google Drive must be the same one with which you connect to Looker Studio.<\/p>\n\n\n\n

    Check \u201cCreated timestamped folder in ouput\u201d.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Finally, check \u201cExport for Looker Studio\u201d and in Configure select all the metrics, clicking on the double arrow.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Click on OK to save everything<\/strong> and you will have the schedule activated.<\/p>\n\n\n\n

    I recommend that you check if everything has worked correctly after the first crawl. To do this, schedule a crawl shortly after making the configuration. <\/p>\n\n\n\n

    Whenever you want to check if a scheduled crawl is working well, go back to File > Schedule and then to History.<\/p>\n\n\n\n

    Here you have more information about scheduled crawls.<\/a><\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Step 4: Connect to Looker Studio<\/h2>\n\n\n\n

    When the first crawl is performed, a Google Sheets document is created in the Google Drive account you have chosen.<\/p>\n\n\n\n

    This file will have the name you gave the Task in the previous step. <\/strong><\/p>\n\n\n\n

    Check that it has been created.<\/strong><\/p>\n\n\n\n

    In Looker Studio you are going to click on Create data source.<\/strong><\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Choose the Google Sheets connector that you will find in Google Connectors.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Now select the Google Sheets document that corresponds to the Screaming Frog crawl and click on connect<\/strong>, above on the right.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    On the next screen you will see the fields that the report contains. <\/p>\n\n\n\n

    Click on Create report.<\/strong><\/p>\n\n\n\n

    The data source is already created. The report that opens can be closed and deleted. <\/p>\n\n\n\n

    <\/p>\n\n\n\n

    Before moving on to step 5, you are going to create another data source. The connector is \u201cChrome UX Report\u201d<\/strong>.<\/p>\n\n\n\n

    Go back to Create data source and search for the connector.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Configure the steps and connect this data source.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    <\/div>\n\n\n\n

    Step 5. Duplicate the Template and Connect your Data Source <\/h2>\n\n\n\n

    In the event that you have purchased the template I mentioned at the beginning<\/a><\/strong>, you must duplicate it from this section:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Now you must choose your data sources<\/strong> to replace the original ones in the template:<\/p>\n\n\n\n

    In the first field, you must select the Google Sheets<\/strong> data source that you have just created.<\/p>\n\n\n\n

    In the second field, select the Chrome UX Report<\/strong> data source that you have just created in step 4.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Click on Copy report<\/strong> and you will have your copy with your own data.<\/p>\n\n\n\n

    Keep in mind that the Google Sheets is completed with each crawl and in this way the Looker Studio report has more and more data to see the evolution.<\/p>\n\n\n\n

    In the first days the report<\/strong> will hardly have historical data, especially if you schedule crawls weekly or monthly.<\/p>\n\n\n\n

    This template can for example give you data like these:<\/p>\n\n\n\n

    Evolution of the Core Web Vitals over time:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Problems with titles and meta descriptions:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Data such as total URLs, images, CSS, JavaScript, status codes…<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Detail and evolution of indexability:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    <\/p>\n\n\n\n

    Configuring all this may take you a few minutes, but the time you can save is enormous and it will be very useful to have the onpage SEO of your projects controlled.<\/p>\n\n\n