Sitemaps: How to Improve Your Website’s SEO With Easy Crawlability

Think about any road trip you’ve ever been on.  No matter how short or long the journey was. 

Did it involve planning a route with a map to some description? Even if it’s just putting a postcode into a satnav.

Without a map, your road trip could potentially have been a disaster. And it’s unlikely you would have ridden along without any hiccups along the way. 

Now, imagine Google bots arriving on your website – much like you tooting along in your car – and trying to navigate through your pages with only internal links to help get them there.

A sitemap, much like a real map, is crucial for the journeys completed on your website. Both by your users and search engine bots. 

As the name implies, a sitemap is a visual and systematic representation of your website. It helps users and search engines to find information and to navigate through your website. Without it, some of your most critical web pages could go by missed. Which could be detrimental to the success of your website and your business as a whole.

In this respect, sitemaps are considered to be a fundamental feature of your website. Both in terms of its SEO and usability. Sitemaps need to be systematic, clear, and easy to navigate. Their main role is to enhance optimisation and enable effective communication between you, your visitors and search engines. 

So, whether you’re just starting to put your website together, or planning a complete design overhaul, take some time to familiarise yourself with sitemaps. As well as their importance, and how to implement them on your own website.

A sitemap is essentially what it says on the tin. It’s a file that lists all of the pages on your website. It offers a roadmap as to how they are related to one another. 

Search engines like Google read this map when they crawl your site to learn valuable insights about your webpages. As well as to understand which pages and files you think are most important for your website. 

For example, a sitemap can tell search engines:

• When a particular web page was last updated
• How often you make changes to it
• And any different language versions you may have for a particular page

Sitemaps can come in all shapes and sizes. They can be inputted as XML or HTML code (more on that later). And you could have just one sitemap for your website, or one per set of pages. For instance, such as your blog posts, your images or product categories. The list is endless.

XML, or, Extensible Mark-up Language, as it is otherwise known, is a format for encoding information about your website in an easy-to-read way for search engines.

An XML sitemap doesn’t look too dissimilar from a plain list of URLs. But it does have some extra information. This information comes in the form of tags. And there are two common tags you should be aware of:

If you own a webpage which is frequently updated, then you will want to include a change frequency tag in your XML sitemap.

In code, this looks like this:

<changefreq>

And it signifies to a search engine how often you update the content on a particular webpage.

The more frequently a page is changes, the more frequently it must be crawled. Therefore, include the tag on your high-value and regularly updated pages to help inform Google’s bots that a page should be crawled as often as possible. So that any new information can be found and submitted to Google’s index of pages.

As the name suggests, a priority tag highlights to search engines which webpages are the most important to your website. 

When Google’s bots crawl a website, they will only crawl a certain number of pages each time they visit. If a website has thousands of pages to crawl, it’s important to highlight which pages Google’s bots should prioritise.  

Using the priority tag within your website’s XML sitemap will help Google to identify those pages and index them first. To do so, include the following snippet of code within the URL in your sitemap: 

<priority>

The priority rankings go from 0-1 in .5 increments. The closer to 1, the higher the priority level for a page. 

For example, let’s say that you had a high-value product page. But your sitemap lists it as low priority with a level of around 0.1. Then Google may not re-visit that page or detect any improvements on a page for hours, days, or sometimes several weeks. Crucial information could be missing from the search engine’s index. As a result, relevant searchers may not be able to find your important changes. 

If you were to combine these two tags together, then those pages will be crawled first. Similarly, lower value pages, such as your website’s privacy policy page may not be updated that often. As such, may not be a priority in a website’s crawl.

In this instance, you may give it a tag such as a <lastmod> with a date which states when the page was last modified. Website crawlers like Google can then check the sitemap of a website against its index and determine whether the last modified tag’s date of any pages has changed and whether it should recrawl them.

By ranking your pages in such a way, you can ensure Google crawls the right content when it’s needed the most. And you can ensure your most valuable pages are updated as regularly as possible. So that your searchers can discover it in a page of search engine results.

An HTML sitemap is a visual representation of a website’s structure. Like an XML sitemap, it lists all of the important things in order. But in a more user-friendly way, using words instead of code. 

HTML sites are more convenient for the user than the search engine; they make it easier to find a page when it is difficult to in a website’s main navigation menu or within internal links. 

You’ll typically find them at the bottom of a webpage. Here, you’ll see links to important pages such as the privacy policy or a brand’s terms and conditions. 

As they require a lot more maintenance to remain visually coherent, they are not as common as XML sitemaps. For websites with thousands of pages, they can be near impossible to maintain. As a result, they are rarely used.

The short answer is, yes. Every website should have a sitemap.

In terms of user experience on your website, sitemaps can be particularly beneficial. More specifically, an HTML sitemap can act as a visual aid for your site visitors. Especially on how to find valuable pages of information that may not necessarily be in your website’s menu navigation. For instance, such as the FAQs or Privacy Policy.  

But beyond user experience, sitemaps play a fundamental role in your website’s SEO. In fact, they assist search engines and help them to navigate around your site. And to understand the relationship between all the pages.

This is particularly helpful for websites with hundreds or even thousands of URLs. For example, like eCommerce stores or blog sites with hundreds of articles. Because you can create multiple sitemaps for a website. So you can segment and manage sections of your websites more easily. 

For example, if you manage an eCommerce website, then different sitemaps per product category can make product page management easier. You could have a category for women’s tops, women’s trousers, men’s tops, men’s trousers and so on. 

The most significant benefit of a sitemap is that you can use it to notify search engines when you have made changes to your website. Rather than waiting for Google to crawl them at its own pace.

Using Google’s Search Console tool, you can submit your sitemap directly to Google. This alerts them to a list of pages it should crawl next. Rather than waiting for Google to find your sitemap and notice  any <lastmod> tags, you’ll be ensuring that every page you want crawled will be shortly after you submit it. 

So, whether you’re building a website from scratch or planning to optimise your website’s SEO strategy, a sitemap is essential to the process. Building one isn’t hard to do, but you do have to be methodical in your steps to get it done correctly.

If you already own a sizable website, then the thought of having to plan and create a sitemap for every single webpage that exists on your site probably fills you with dread. 

Luckily, almost every website that you create on a Content Management System (CMS) will have sitemaps automatically built-in.

For example, the popular CMS platform WordPress has a sitemap file built-in with automatic updates. Thus, eliminating the need for you to update it every time you create a new webpage. You can also install the Yoast SEO plugin for WordPress too. Which, if you wish to, allows you to segment pages and posts.

Even if you don’t have a website that’s built on WordPress, you’ll notice that almost every CMS has its own sitemap management system. Although some of these may require a plugin.

It’s very rare that you come across a CMS which does not have a sitemap system built-in to their platform. In this instance, you will need to create one manually. Then upload it to Google so it can crawl any new pages or bulk changes you have made to your website.

Although this may seem quite daunting, the great news is that once you’ve mapped out your entire site, the future process will only involve adding any new webpages you add to your website.

Fortunately, there are plenty of online sitemap generator tools out there. Which can make creating your own that bit easier. Some of our favourites include:

• Slickplan – An easy drag-and-drop interface that allows you to create elegant and professional-looking sitemaps. Can be exported as an XML file to be uploaded to Google.
• SEO Spider (part of Screaming Frog’s website software) – A multifunctional tool that will crawl your website and pull all your URLs for you into their sitemap planner.
• Mindnode – A mind-mapping tool that allows you to visually map out your website sitemaps before you create the final version. Note, this will not create the final sitemap for you, but it’s great for visual planners.

Once you have a complete sitemap, you want to submit it to Google’s Search Console to have your pages crawled again. 

Again, this can be done with ease, as long as you have a Google Search Console account for your website. Just follow these three easy steps:

1. Log into the Google Search Console account for your website.

2. In the main menu navigate to the “Index” dropdown and select “Sitemap.”

3. In the box titled “Add a new sitemap,” enter the URL of the sitemap you have uploaded to your website but without your  domain name and press submit. These URLs will probably look something like this: “/sitemap_index.xml” or “/page-sitemap.xml”

And there you have it – uploading your sitemap in three simple steps. It really is as easy as that!

Once you have hit the submit button, Google will then schedule its bots to crawl those uploaded URLs within a matter of days or weeks. You can track the status of your sitemap’s crawl by keeping an eye on the “Status” column under your “Submitted sitemaps” box.

Everyone wants Google to crawl every important page of their website. But sometimes, pages that have no internal links that point to them are often hard to find. As a result, they can be missed from your website’s indexing. 

And this is where a sitemap is particularly important for your website.

Essentially, a sitemap is a list of every single page of your website in order of importance. They are read by search engines like Google who prioritise which pages they then crawl when they next visit your site. This means your most important pages will always be indexed first.

If you have a new, important webpage. Or you regularly update a valuable page with new information. Then you may want to manually submit a sitemap via Google’s Search Console. As this will ensure Google crawls your content as soon as possible. 

Sitemaps can come in either XML format or as HTML. With XML being the most popular for those with lots of webpages. You can also segment sitemaps into different categories. Especially if you have thousands of pages to keep tabs on. 

Fortunately, most of us never have to worry about how to create our sitemaps. Because popular CMS platforms like WordPress have sitemap system tools already built-in to do the hard work for you. However, it’s important to regular review these. In order to ensure your pages are being prioritised as you would like them to. 

Now you have an understanding of how important a sitemap is for your website. As well as how to create one, it’s time to review (or create yours). So start submitting it to Google’s Search Console to speed up the process of crawling your most important web pages and having that new information indexed.

Sitemaps play an important role in your website’s SEO. In fact, they help search engine robots to crawl your pages and index them in order of importance.

But sitemaps and other off-page SEO elements play a small role in your website and its visibility in search engines.

It’s common knowledge that to make a long-term success of your website, you need consistently high-quality and relevant content. Which also needs to be properly optimised for SEO. On top of this, you also need to ensure you adequately promote your content. Particularly when you begin your digital marketing efforts. Because during this time, your website has very little discoverable content on search engines. 

This of course is a large-scale operation. And one which requires a complete overhaul of your SEO strategy and a robust content marketing strategy in place. This is to ensure you take all the right steps for long-term success. 

For many small businesses or new website owners, this can be a difficult task to plan on your own. Especially if you have little experience in this area. 

As such, many turn to the help of a marketing consultancy such as ours. A marketing consultancy who has the expertise and resource available to create a comprehensive SEO strategy. And one which can align all your marketing channels to ensure you take all the necessary steps to increase your website traffic over the long-term. 

If this sounds of interest to your business, then why not get in contact? My team and I have over a decade’s worth of experience in the industry. As such, we will be more than happy to walk you through some previous examples of our work.