A Screaming Frog SEO audit starts with a specific question: can a crawler discover your important website pages, retrieve them correctly, and read the signals you intend to send? Screaming Frog SEO Spider helps you answer that question by collecting URL-level evidence – response codes, internal links, directives, canonicals, headings, and other page attributes – so you can investigate problems instead of relying on assumptions.
In this tutorial, I will show you how to choose a crawl mode, define the scope, inspect the results, trace problems back to their source URLs, and verify fixes. We will follow one hypothetical service page, https://example.com/example-service/, from discovery through the final re-crawl. It is a teaching example, not a claim about a real client’s website.
A crawler’s findings are a snapshot of what it was configured and able to fetch. They do not by themselves establish which pages Google has indexed. Our working sequence is define the target URL → verify its discovery → inspect its response and indexability signals → trace source links → confirm the cause → implement and re-crawl. Each step produces evidence needed for the next, rather than a separate list of generic errors. If you need the broader workflow across Google Search Console, performance, rendering and implementation, start with my complete technical SEO audit; this article focuses on executing and interpreting a crawl in Screaming Frog.
What Can Screaming Frog SEO Spider Actually Audit?
Screaming Frog SEO Spider audits technical signals observed during a website crawl; it does not directly audit Google’s search index. Given a starting website and a set of crawl rules, the software discovers URLs and records responses, page elements and relationships among pages. Those observations help identify where to investigate and what evidence to send to a developer.
| Input | Crawler observation | Useful next decision |
|---|---|---|
| Starting URL and internal HTML links | Which URLs were discovered and their inlinks | Is an important page reachable through the site? |
| HTTP requests | 200, redirects, client errors and server errors | Is the response expected for this URL? |
| HTML and headers | noindex, canonical references, titles and headings | Do the declared signals match the intended page? |
| Crawl graph | Internal link sources and crawl depth | Is a priority page supported by accessible navigation? |
For example, if /example-service/ returns 200 but contains noindex, the crawler can reveal that conflict with the site’s intention. It cannot tell you, simply from that response, whether Google has already removed the URL from the index. If you are still learning why a page can be crawlable but not indexed, use the broader technical SEO guide for website owners before diagnosing the crawl in depth. Once the tool’s job is clear, choose the mode that supplies the right evidence.
When Should You Use Spider Mode Instead of List Mode?
Use Spider Mode when you want to follow a site’s discoverable links; use List Mode when you already have particular URLs to inspect. In Spider Mode, you enter a start URL and the crawler follows the eligible links it finds, subject to your configuration. It is the appropriate starting point for navigation, crawl depth, broken-link sources and URL discovery.
In List Mode, you upload a set of URLs or an XML sitemap. This is useful for checking a collection of important URLs after a migration, or testing URLs that are absent from normal navigation. But a list of URL responses does not reconstruct the site’s link graph. If a page is missing from Spider Mode and works in List Mode, the difference may be discovery, not server access. That distinction defines the inventory you need before starting.
What Should You Prepare Before Crawling a Website?
Prepare an authorized crawl scope, a list of priority pages, and a baseline of what the site is supposed to do. Before opening the tool, record the production hostname, expected HTTPS version, major page templates, known XML sitemap location, approximate URL volume and any planned exclusions. Check whether the website has login-only areas, staging restrictions or a firewall that could affect requests.
For our hypothetical site, the audit objective is: confirm that /example-service/ is internally discoverable, returns a successful response, communicates the intended canonical and remains eligible for indexing. Save this objective with the crawl date; otherwise, the same warning could mean a defect on one page and correct behavior on another.
Crawl responsibly. Have permission to test the website, avoid unnecessary load on a busy server and coordinate larger checks with its owner or developer. A short scope note makes the later interpretation defensible because it tells you what you intentionally included, rather than letting you mistake excluded URLs for missing pages.
How Do You Define Crawl Scope, Subdomains and URL Parameters?
Define which hostnames, paths and URL variants the audit includes before you interpret its totals. Decide whether www.example.com and example.com should be treated together, whether a blog subdomain is part of the exercise, and whether search, checkout or account paths should be outside the crawl.
Query parameters also need a purpose-based decision. A URL like /services/?sort=price might be an unimportant view, while filtered category parameters on an ecommerce website could create thousands of discoverable variants. Excluding every parameter makes the report tidier but can hide the exact faceted-navigation problem you were asked to investigate.
Record both the intended canonical inventory and the observed URL patterns. For example, if /example-service/?ref=email is discovered, don’t count it as a separate important landing page until you have compared its response and canonical with /example-service/. With that boundary in place, confirm whether your installed software version supports the required crawl configuration.
What Are the Free-Version Limits and Crawl-Load Considerations?
Screaming Frog’s free SEO Spider edition is limited to 500 URLs per crawl; advanced features require a paid licence. The vendor’s current feature comparison distinguishes the free baseline crawler from capabilities such as JavaScript rendering, advanced configuration, Search Console integration and saving or reopening crawls. Check that page again when publishing or updating this tutorial, since software features can change.
If the crawl reaches its URL limit, a missing page might simply never have been reached. That is different from a page excluded by robots.txt or a page that lacks internal links. On large sites, limit requests to what the server can handle, and record the crawl’s constraints. A reliable 300-URL sample is more useful than an aggressive crawl that disrupts the website or confuses failures with tool pressure.
The next step has two paths: a basic crawl you can start immediately, and a configurable workflow for advanced investigations.
How Do You Configure a Standard SEO Spider Crawl?
You can perform a basic Screaming Frog SEO audit by crawling your website, reviewing the results, identifying affected URLs, and checking the source pages responsible for each issue.
Step-by-Step: Run Your First Screaming Frog SEO Crawl
Step 1: Start a website crawl
Open Screaming Frog SEO Spider, select Mode → Spider, enter your website’s homepage URL in the address field, and click Start.
The software will begin discovering URLs through crawlable links according to its configuration.
Step 2: Review the crawled URLs
Open the Internal tab to examine the URLs discovered during the crawl. Check their addresses, content types, response codes, and indexability classifications.
Before interpreting any issues, confirm that the crawler reached the expected website and important pages.
Step 3: Find broken links and errors
Open Response Codes and select the Client Error (4xx) filter. Select an affected URL, then open the lower Inlinks panel to identify which page links to the broken destination.
If the link is incorrect, update it on the source page and verify that its new destination works.
Step 4: Check redirects and indexing directives
Use the Response Codes filters to investigate redirects and server errors. Review relevant Directives and Canonicals views to check whether important URLs communicate their intended indexing and canonical signals.
A non-indexable URL is not automatically an error. Compare its reported state with the purpose of the page.
Step 5: Audit page titles and headings
Open the Page Titles, H1, and relevant page-element views. Review missing or duplicated elements and investigate whether they indicate a genuine template or URL problem.
Repeated titles alone do not prove keyword cannibalization.
Step 6: Export the findings
Choose the relevant report or filtered URL table and click Export. For broken-link source information, use the applicable Inlinks export or available Bulk Export options.
Record the affected URL, referring source where relevant, observed issue, proposed correction, and verification requirement.
Step 7: Re-crawl after implementing fixes
Once a justified correction is published, crawl the affected URLs again using comparable settings. Confirm that the specific problem is resolved rather than assuming that a lower warning count means the website is fully optimized.
Free-version note: Screaming Frog allows up to 500 URLs per crawl without a paid licence. Advanced configuration, JavaScript rendering, integrations, and other licensed features are not included in this basic walkthrough.

Which Robots Rules and User-Agent Settings Should You Check?
Treat crawler identity, robots.txt permissions and HTTP authentication as separate mechanisms. A user-agent identifies the requesting client, robots.txt communicates crawl rules to applicable crawlers, and a 401 or 403 response is the server denying or restricting access. The same URL may be discoverable but inaccessible under one of these conditions.
Check the relevant robots file and the crawler’s observed status rather than assuming that a failed request is a Google indexing directive. If SEO Spider displays a blocked by robots.txt status rather than a normal HTTP response, do not report it as an HTTP 403: a crawl-rule block and a server rejection are different observations. A noindex directive is also different: Google must be allowed to crawl an accessible URL to see its noindex tag or header. Blocking that URL in robots.txt can prevent Google from reading the exclusion signal, so robots disallow and noindex should not be prescribed as interchangeable fixes. Some identity and robots configuration options may require the paid edition; verify what your installed version allows. Never change crawler identity or bypass security controls to access restricted areas without explicit authorization from the website owner.
If /example-service/ is linked from the homepage but the crawler receives an access error, save the request URL, the response and the authorization context. You have evidence of a fetch problem; you do not yet have proof that Googlebot experiences the same restriction. Rendering is a separate decision when the server returns HTML successfully but meaningful content depends on JavaScript.
When Should You Enable JavaScript Rendering?
Enable JavaScript rendering when important content or navigational links are created client-side, or when you suspect a rendering mismatch. SEO Spider’s JavaScript rendering is a paid feature; its standard crawl typically reads the initial HTML without running a full JavaScript-rendering workflow.
Imagine /example-service/ returns a page shell, but its link to /contact/ appears only after JavaScript executes. A raw-HTML crawl and a rendered crawl can produce different link inventories. If you are using the free edition, compare View Page Source with your browser’s Inspect / rendered DOM for a representative URL, and note the limitation of that manual check. Rendering can use more resources and time, so select it because the evidence requires it – not because every website uses JavaScript.
Once the mode, permissions and rendering choice are defined, begin the crawl and make sure it is collecting the expected site rather than a misleading subset.
How Do You Start a Screaming Frog SEO Audit and Verify URL Discovery?
Start the crawl, then validate the first results before trusting the complete report. Enter the start URL, click Start and watch the tool populate internal URLs, page types, response codes and progress. Confirm that the intended hostname appears, the crawler finds normal internal HTML links, and the rendering mode matches your scope note.
Compare early findings with a small known inventory: homepage, /services/, /example-service/ and perhaps a contact page. If the tool immediately reports only one URL, stop treating the result as a full site audit. Check the crawl’s starting point and discovery constraints first. If the crawl finishes at 500 URLs in the free edition, say the sample was limited, not that the site contains 500 pages.
For the running example, the first desired signal is a discoverable path to /example-service/. If the path is absent, determine whether the URL was never encountered or was found but could not be fetched.
Why Might Screaming Frog Find Fewer URLs Than Expected?
A low URL count can result from a wrong start URL, missing crawlable links, denied access, a crawl limit, excluded paths or JavaScript-dependent navigation. Investigate those causes in roughly that order instead of changing random settings until the count rises.
| Observation | Check next | Evidence to keep |
|---|---|---|
| Only one URL appears | Is the start URL correct and does its HTML expose links? | Start URL and its source links |
| Important page absent | Does any crawled page link to it? | Navigation/source URL and missing destination |
URL was found but returns 403 | Authentication, firewall and permission context | Request URL, response and test conditions |
| Crawl stops at its cap | Free URL limit or configured bounds | Total reached and software edition |
| JS-driven links missing | Raw anchors versus rendered DOM | Raw/rendered example and template |
An unlinked /example-service/ and a blocked /example-service/ require different fixes: the first is a discovery-path problem, the second a response-access problem. Neither should be called an indexing failure without checking Google-reported evidence. Once the crawl’s coverage is credible, read its overview as a set of observations – not a list of automatic tasks.

How Should You Read the Crawl Overview Before Individual Errors?
Read the crawl overview in a sequence that follows a page’s path: discovered URL → fetched response → indexability signals → internal relationships → page attributes. Begin with the Internal and Response Codes views, then inspect Directives, Canonicals, Inlinks and relevant page-element tabs for your target URLs.
| Observation | What it tells you | What it does not prove |
|---|---|---|
| 1,000 internal URLs discovered | The configured crawl encountered 1,000 URLs | Google has indexed 1,000 pages |
| 20 client-error responses | Those requests produced 4xx statuses | All 20 URLs should be restored |
| Page labeled non-indexable | The crawler found a disqualifying signal | The exclusion was accidental |
| High crawl depth | The page was reached after several link steps | The page cannot rank |
The numbers above are illustrative, not a measurement of Learn With Usaid or a client’s website. Use the overview to identify patterns worth validating, not to chase a perfect health score. Response status is the clearest first layer because an inaccessible page limits what you can learn about its contents.
What Is the Difference Between an HTTP 200 Page and an Indexable Page?
An HTTP 200 says the request succeeded; an indexable page must also satisfy relevant technical eligibility conditions, and Google still decides whether to index it. A 200 page with noindex may be successfully fetched yet intentionally excluded from indexing. A 200 page canonicalizing to another URL may be considered a duplicate candidate. A client-side-only link may not appear in your raw crawl at all.
The sequence matters: response code → robots directive → canonical signal → content and link evidence → Google-reported status. If /example-service/ returns 200, do not stop there. Check whether it is the version the owner wants indexed, whether its directives agree, and whether the crawler found it through appropriate links. These distinctions become practical when examining broken paths and redirects.
How Do You Find Broken Links and HTTP 4xx Errors?
Open the Response Codes tab and filter for client errors (4xx), then identify the pages linking to each failing destination. A 404 or 410 tells you a URL is unavailable; it does not tell you whether the URL should exist. Start by asking whether the failed destination is still useful to visitors and part of an intended journey.
| Linking page (source) | Requested destination | Response | Preferred action |
|---|---|---|---|
/example-service/ | /service-pricing/ | 404 | Correct the link if a valid pricing page exists |
/news/old-event/ | /expired-event/ | 410 | Keep removal if intentional; review links to it |
/services/ | /example-service/ | 200 | No broken-link action needed |
These are hypothetical URLs and responses. An actual repair might mean updating an internal anchor to a valid page, restoring accidentally deleted content or removing an obsolete reference. It does not always mean redirecting the broken destination to the homepage. For every 4xx row, first find the source that led people or crawlers there.

How Do You Identify the Pages Linking to a Broken URL?
Select a failing destination in the upper crawl table and examine its Inlinks panel to find source URLs and link context. You can also export the relevant inlinks when you need a developer-ready list. The source URL contains the link; the destination URL receives the request. Never reverse those roles in a fix ticket.
If /example-service/ links to deleted /service-pricing/, the recommended edit is normally made on /example-service/: update its anchor to the appropriate live URL. Record the original anchor, proposed destination and final checked response. If a broken URL has no recorded inlinks, search other URL sources and understand whether it was included in a list or sitemap rather than encountered in navigation.

With the source of a broken request identified, inspect redirects and server responses that can interrupt the same navigation path.
How Do You Investigate Redirects, Chains and Server Errors?
Use Response Codes to review redirects (3xx) and server failures (5xx), then follow each important URL to its final destination and test whether the behavior is repeatable. A single correct 301 move is different from a chain involving several hops, and a transient 503 during planned maintenance is different from persistent errors on revenue-critical pages.
Record a redirect in this order: old/source URL → intermediate URL(s), if any → final/destination URL → final response. Example: /old-service/ → /services-v2/ → /example-service/ → 200. Where the site controls the internal links, update them to point to the intended final page if this reduces unnecessary hops and preserves meaning. Check the destination’s relevance before considering any redirect change.
For 5xx responses, keep the timestamp, affected template and repeat test result; involve the developer before assuming the crawler caused the issue or that a plugin is responsible. The next question is whether successfully retrieved pages communicate the indexing and canonical rules you intended.
When Is a Redirect Chain or Server Error Worth Prioritizing?
Prioritize a response problem when it affects important pages, repeats across a template, interrupts users or blocks the intended crawl path. A long redirect path on hundreds of navigation links is usually more urgent than a historical redirect used by one retired campaign URL. A recurring 500 on service pages deserves investigation sooner than a single temporary error in an unimportant utility area.
Avoid ranking issues by warning count alone. Write the actual impact hypothesis: “Links from the main service navigation currently route through two intermediate URLs, making the preferred destination harder to maintain and monitor.” Then record a verification test, such as direct links returning 200 after the fix. We will reuse this evidence-first priority method at handoff; first finish the crawl’s directive and canonical checks.
How Do You Audit Robots and Canonicals in Screaming Frog?
Compare robots access, noindex instructions and canonical references against each URL’s intended purpose. These signals do different jobs. Robots restrictions control whether requests are allowed under relevant rules; noindex requests exclusion from results when seen; canonical references indicate a preferred duplicate or near-duplicate URL. They should not be treated as interchangeable buttons.
| Intended state | Crawler observation | Audit question |
|---|---|---|
| Public service page should be indexable | 200 with noindex | Was exclusion introduced accidentally? |
| Private utility area should remain inaccessible | Restricted or authenticated | Is access protection intentional and adequate? |
| Parameter variant should consolidate | Canonical points to main service URL | Do content and links support that preferred version? |
| Preferred landing page should stand alone | Canonical points elsewhere | Is this the right target? |
Use the Directives and Canonicals views where relevant, and check the source HTML or HTTP headers to confirm what the crawler extracted. A Screaming Frog canonical is a declared signal, not proof of Google’s selected canonical. For URL-selection decisions, consult Google’s canonicalization guidance. That requires separate Google Search Console evidence. The important next step is to distinguish expected canonicalization from a conflict that affects an intended primary URL.
Which Canonical and Duplicate Signals Need Manual Checking?
Manually investigate canonicals that disagree with the intended URL, point to an error page or conflict with important internal links. A self-referencing canonical on /example-service/ can be consistent with its intended role. A tracking-parameter version canonicalizing to /example-service/ may also be entirely appropriate. Neither is automatically a defect.
More concerning is a primary service URL pointing to a different service category, a canonical destination returning a non-successful response, or a sitemap listing one version while navigation consistently links to another. Compare the intended canonical target, HTTP status, content similarity, internal linking and sitemap entry before recommending a change.
Canonical relationships lead naturally to other page-level patterns: titles, headings and duplicate URL variants may reveal template or parameter problems that the response code alone cannot show.
How Do You Check Titles, Headings and Duplicate URL Patterns?
Use the Page Titles, H1 and content-related tabs to find repeated page elements and URL patterns, then validate whether the repetition reflects a technical template problem. Two category filters can legitimately share wording, while several unexpectedly indexable parameter variants with identical titles may indicate uncontrolled URL expansion.
For example, /example-service/?ref=summer and /example-service/?ref=winter may expose the same page title. The crawl shows the repeated metadata and URL variants; you must still check the content, intended canonical and business purpose before recommending consolidation. Duplicate titles do not prove keyword cannibalization and a crawler does not measure two pages competing in Google’s results simply by comparing H1 text.
This report’s purpose is to surface template and URL-pattern evidence. Keyword mapping and rewriting belong to a separate content decision. After checking which URLs say similar things, inspect how the site’s internal links distribute access to them.
How Do You Inspect Internal Links and Crawl Depth?
Select an important URL, review Inlinks and Outlinks, then inspect how many crawl steps were required to reach it. Crawl depth reflects the path followed under the tool’s configuration; it is not the same thing as the number of folders in a URL.
Suppose /example-service/ is linked only from a rarely visited article several clicks from the homepage. It may return 200 and be indexable but still lack a clear navigational route for users. The evidence to save is the linking source, anchor text, destination and discovered depth. A better fix might be adding a relevant link from /services/, not changing the service page’s URL merely to shorten its path.
You can only evaluate links that the crawl discovered. That creates an important blind spot: what about pages that exist but have no discovered inbound link?
Can Screaming Frog Find Orphan Pages from a Normal Crawl Alone?
No. A normal link-following crawl cannot reliably discover every orphan page because a true orphan has no internal link path from the crawler’s starting network. The crawler may still encounter a URL from other enabled sources, but its absence from a single Spider Mode crawl is not proof that the URL does not exist.
Compare the crawled URL set with an XML sitemap, an approved CMS inventory, or other authorized URL sources. Record where each additional URL came from. If /example-service/ is present in a sitemap but never found through site navigation, investigate whether it is intentionally isolated or missing a useful internal link. That leads directly to sitemap reconciliation – an exercise whose capabilities differ between free and paid editions.
How Do You Compare XML Sitemap URLs Against Crawled Pages?
Compare sitemap URLs with the link-discovered inventory to identify preferred pages that were missed, unexpected URLs and potential sitemap-only orphans. For the integrated comparison, the vendor’s XML sitemap audit tutorial describes a licensed site crawl configured to crawl XML sitemaps, followed by the relevant Crawl Analysis step to populate comparison filters.
In the free edition, you can upload or download an XML sitemap in List Mode to test the URLs inside it. That does not establish which URLs are orphaned from the site’s link graph, because List Mode alone has no comparable link-discovery inventory. Be careful not to present the free sitemap check as the same report.
| Observation | Additional evidence | Decision |
|---|---|---|
| URL in sitemap, absent from link crawl | Internal navigation and template links | Add a relevant route if it should be discoverable |
| Redirected URL in sitemap | Final destination and intended canonical | Maintain the preferred destination in the sitemap |
Utility URL marked noindex in sitemap | Business intent and directive | Usually remove from an indexable-page sitemap |
| Parameter variant discovered but not in sitemap | Canonical and content purpose | May be intentional; do not add every variant |
The principle is source of URL → intended state → observed state → action. An XML sitemap is a discovery aid, not an indexing guarantee. In the licensed integrated workflow, the Sitemaps filters that require it are populated after Crawl Analysis → Start; merely uploading a sitemap into List Mode cannot reproduce a link-versus-sitemap orphan comparison. The same URL can have different meanings depending on why it appeared in the report.
What Do Non-Indexable or Sitemap-Only URLs Mean?
A non-indexable or sitemap-only URL is a finding to classify, not an error to delete automatically. An old redirected URL could remain in an outdated sitemap. An intentionally private utility page might be correctly excluded. A primary service page might be an accidental orphan. A duplicate variant might correctly canonicalize to a different preferred address.
For /example-service/, inspect its status, canonical, robots instructions and actual internal-link sources before acting. If the sitemap lists it but normal links do not, decide whether the missing link undermines the reader’s journey. Call it sitemap-only within the scope of this configured crawl, not a proven sitewide orphan until other internal link sources and relevant templates have been checked. If it is a duplicate parameter variant, changing the sitemap may be enough; if the preferred service page has an unintended noindex, the issue is more significant.

A sitemap comparison tells us which URLs should be investigated. A separate rendering check becomes useful when the crawler and a browser disagree about the content or navigation visible on those URLs.
How Do You Compare Raw HTML with JavaScript-Rendered Content?
Compare the initial HTML with the DOM produced after JavaScript runs to see whether important links, headings, text or directives depend on rendering. On a licensed SEO Spider installation, choose JavaScript rendering from the Spider rendering configuration, run a representative set of templates and retain the original and rendered HTML where the tool and your settings allow it. Check Screaming Frog’s JavaScript crawling guidance for current controls.
Consider /example-service/: the initial HTML contains a navigation container, but the link to /contact/ appears only after a script runs. A standard raw crawl might miss that path while a rendered crawl reveals it. Record the source URL, the absent raw anchor, the rendered anchor and whether execution depends on user action. A free user can compare View Source and the browser’s rendered DOM manually, but cannot use SEO Spider’s paid JavaScript rendering feature in the free edition.
This difference identifies implementation risk; it does not prove Google could not render the link. To understand that risk, inspect exactly how and when the missing content becomes available.
Which JavaScript-Only Links or Content Are SEO Risks?
Prioritize JavaScript-only elements that are essential to discovery or understanding and may not be available through a stable crawlable route. A navigational anchor inserted automatically after load is different from a link that appears only after a visitor selects a tab, scrolls through a complex widget or supplies an input.
For our example, inspect whether the service page’s main description and contact link appear in the rendered DOM without extra interaction. If the link only appears after a click, record the relevant component, raw and rendered evidence, and a reproduction step. Your recommended change should make important content or links available reliably without forcing a crawler to simulate an arbitrary user journey.

After validating what your crawler can see, compare it with independent Google data only where the question genuinely needs another source.
When Should You Connect Search Console or Other URL Sources?
Connect Search Console data when you need to reconcile crawl discoveries with search-side evidence, not because every crawl requires an API connection. SEO Spider’s Search Console integration is a paid capability that requires appropriate property permissions. It can add certain performance-related signals or URL sources depending on the configuration, but it does not turn Spider Mode into Google’s indexing database.
Keep the measurements distinct. A crawl discovers URLs according to a current link graph; an XML sitemap lists preferred URLs; Search Console Search Analytics covers performance signals over selected dates; URL Inspection and Page Indexing describe other Google-reported evidence. These tools measure different URL populations and different moments in time. Do not call these the same denominator, and do not treat a crawler’s Indexable label as Google’s Indexed status.
If you use the free SEO Spider edition, inspect your own Search Console account separately. Record which property, source, date range and URL state each observation comes from. That context explains why apparently conflicting counts may both be reasonable.
Why Can Crawl Results and Google Search Console Data Disagree?
They measure different things at different times under different scopes. A Screaming Frog crawl captures the configured website response and link structure now. Google Search Console reports Google’s data under its own processing and reporting conditions, including delays and different URL grouping rules.
For a deeper explanation of how to interpret Google-side indexing evidence alongside technical crawl findings, follow my technical SEO audit guide.
For example, a page may be currently reachable and labeled indexable by your crawler while Search Console still reports a past noindex observation. Conversely, a URL shown in Search Analytics could have historical impressions even if you have since removed its internal link. Treat the mismatch as a question about source, date, scope and intended state, not proof that one tool is inaccurate.
Once those constraints are recorded, you can consider specialized crawl checks without turning the article into separate structured-data, hreflang and media tutorials.
Which Structured Data, Hreflang or Media Checks Belong in This Crawl?
Run specialized checks only for the templates that actually use those features. For a service page with structured data, inspect whether the markup appears and whether it matches visible content, then validate eligibility with an appropriate dedicated tool. For international pages, inspect whether hreflang references reciprocal, valid alternate URLs. For image-heavy templates, review whether important media URLs are accessible and represented appropriately.
Not every website needs every check. A local service website with no regional alternates does not need an elaborate hreflang section merely to complete a checklist. Some advanced validation capabilities are licence-gated, so refer to your installed version and the vendor’s feature matrix before giving a click-by-click instruction.
Remember the boundary: technically valid structured data does not guarantee rich-result display, and a crawler-observed hreflang entry does not prove Google selected the intended regional URL. After optional checks, the real deliverable is a usable set of evidence, not a screenshot of a green dashboard.
How Do You Export Crawl Findings and Source URLs?
Export the affected URLs together with their sources, observed technical signals and intended fixes. Use a relevant tab and filter, then its Export action to capture the selected findings. For source-link reports, inspect the tool’s available Inlinks or Bulk Export options; menu wording and licence access may vary by version. Do not export only a total error count and expect the developer to guess where the problem lives.
A compact issue register might look like this:
| Affected URL | Source / template | Observed state | Intended state | Evidence | Proposed action | Owner | Retest |
|---|---|---|---|---|---|---|---|
/service-pricing/ | /example-service/ | 404 | Live pricing route | 4xx + inlink | Update broken anchor | Content / dev | Link returns 200 |
/example-service/ | Service template | noindex | Public indexable page | Directives + source HTML | Confirm and remove accidental rule | Developer | Re-crawl directive |
/old-service/ | Main navigation | Multi-hop 3xx | Direct final navigation | Chain export | Point anchor to final URL | Developer | Single direct request |
All three rows are hypothetical examples; do not use them as a real client case study. Keep the crawl date, configuration, supporting screenshots and confidence level with the register. This converts the crawler’s observations into a format where every fix can be reviewed and reproduced.
How Do You Confirm That a Reported Issue Is a Real Problem?
Replicate the observation on a representative URL, confirm what the business intended, and compare the two before you label it a defect. If a URL returns 404, inspect whether it was deliberately retired. If a page is noindex, ask whether it is a checkout step, internal search page or a primary landing page. If a canonical points to a different URL, verify that the target is actually the preferred equivalent.
For /example-service/, compare the crawler’s noindex finding with the page owner’s goal of appearing in search. Use the raw HTML or headers to verify the directive and check whether it affects the entire service template. That gives you a supported diagnosis and a confidence level, instead of inventing a severity number from a tool’s visual warning.
A useful test report therefore separates observation, intended state, confirmed cause and recommended fix. In the handoff, include the source page that needs its anchor updated, the destination page that received the failed request, the owner, and the retest condition – these are different fields, not two names for the same URL. The classification also explains why warnings should not be treated as interchangeable errors.
Why Are Some Crawling Warnings Not Errors?
A warning can describe deliberate behavior that is correct for the page’s purpose. A discontinued URL may appropriately return 410; a private account screen may correctly be excluded from indexing; a retired address may legitimately redirect to its replacement. Those should be documented, not automatically “repaired.”
The same signal changes meaning when it affects an important public URL. A noindex on /example-service/ is potentially serious if that is the page the business intends to rank, whereas a noindex on a private utility page may be expected. Context turns raw rows into decisions. Once an issue is confirmed, prioritize by affected business pages, scale and confidence – not by how brightly the software highlights it.
How Should You Prioritize and Hand Off Crawl Fixes?
Prioritize verified issues by affected URL value, repeatability across templates, likely search-access consequences, implementation effort and dependency order. Repairing a persistent server failure on key service pages generally deserves attention before adjusting harmless duplicate titles on archived event pages. But even a small issue can be urgent when it affects the only conversion page for a valuable business service.
Turn each selected finding into a ticket using this chain: expected state → observed state → confirmed evidence → likely cause → requested change → responsible owner → acceptance test. For our example, the ticket might say: “/example-service/ should be eligible for indexing; the live HTML contains an unintended noindex; confirm the template source, remove only the unintended directive, and verify the response on this page and comparable service pages.”
Avoid broad instructions such as “redirect all 404s,” “remove all canonicals” or “make every page indexable.” Those actions can create worse problems. A technical SEO audit is useful when the next person knows which URL to change, why the change matters and what result will count as success.
How Do You Re-Crawl and Verify Technical SEO Fixes?
Re-crawl the changed URLs or representative templates under comparable conditions and compare the new evidence with the original finding. Keep the same relevant host scope, rendering mode and crawl constraints so you do not attribute configuration differences to a successful fix.
For /example-service/, confirm that the intended internal link path still works, the URL returns the expected response, and the accidental directive is absent from the appropriate HTML or headers. For redirect fixes, retest the actual source URL and final destination. For an internal-link change, confirm the referring page now points directly to the intended target. Also spot-check neighboring pages using the same template to catch regressions.
Record the date, test URL, before/after observation and remaining limitation. For the running /example-service/ example, a satisfactory technical retest means the expected internal link is present, the response is correct, and the accidental noindex is no longer observed; any later Google indexing change is a separate observation. A fixed crawler signal is not an immediate Google index update; review Google-reported status separately over time when needed. This completes the main workflow – configure, discover, inspect, confirm, fix and verify – and leaves a few closely related questions that help you interpret the result correctly.

Can Screaming Frog Tell You Whether Google Indexed a Page?
No. Screaming Frog can inspect a page’s technical indexability signals, but a normal crawl does not independently verify that Google has indexed the page. A URL can return 200, be allowed for crawling, carry a sensible canonical and still not be indexed or selected as canonical by Google.
Use Google Search Console’s indexed URL information and the URL Inspection tool to investigate Google’s reported state. Keep the difference between indexable according to the crawl and indexed according to Google explicit in your report. This prevents site owners from treating a green crawler label as a ranking promise.
Which Screaming Frog Audit Tasks Require a Paid Licence?
The free edition is suitable for a limited baseline crawl, while advanced execution and integrations generally require a paid licence. Use the current Screaming Frog pricing and features page as the source of truth; exact commercial terms or feature availability can change.
| Task | Free edition | Paid licence |
|---|---|---|
| Spider/List crawl of up to 500 URLs | Yes | Yes, with higher crawl capacity |
| Basic response codes, headings, link inspection | Available within free crawl limits | Available |
| Advanced Spider configuration | Restricted | Available |
| JavaScript rendering inside SEO Spider | No | Available |
| Search Console/Analytics API integration | No | Available |
| Save/open crawl projects and advanced comparisons | Restricted | Available |
| Integrated sitemap-vs-link crawl analysis | Not equivalent to a simple List Mode sitemap check | Supported with required workflow |
If you only need to find a broken internal link on a small website, a free crawl may be enough. If you need repeatable scheduled audits, JavaScript rendering, advanced data joins or a larger URL inventory, choose the licensed workflow and document the configuration. Tool choice should follow the question, not substitute for a diagnosis.
What Should You Do After Finishing the Screaming Frog Crawl?
Finish your Screaming Frog SEO audit with a scoped, reproducible action plan – not merely a list of flagged URLs. The final deliverables should include the crawl settings and date, an affected-URL inventory, source inlinks, verified response/directive/canonical findings, intended states, assigned owners and a defined re-crawl acceptance test.
In our hypothetical example, /example-service/ began as an important page we wanted crawlers to reach and interpret correctly. The audit followed its internal link path, checked its response and indexability signals, compared supporting URL sources, documented confirmed problems and retested the fixes. That is the practical value of Screaming Frog: it turns a vague concern about visibility into evidence a site owner and developer can act on.
For the broader decision of what to investigate beyond a crawler – including Search Console, performance, rendering and prioritization across the website – use the complete technical SEO audit guide. If your website needs specialist diagnosis or implementation support, you can also discuss your technical SEO requirements.
A single Screaming Frog crawl provides a snapshot of your website’s technical condition. To detect new issues after website updates, establish an ongoing technical SEO monitoring process that compares future crawl results, identifies regressions, and verifies whether fixes remain effective.

