Optimus Sitemap Generator
Optimus Sitemap Generator (OSG) is a universal XML sitemap generator that works by crawling your website, avoiding excessive bandwidth overhead by only scanning the contents of pages that have changed since the last time the sitemap was generated.
|Linux (32-bit)||1.0 – 2012-05-08||osg-1.0-i386.tar.gz||1.3 MB|
|Linux (64-bit)||1.0 – 2012-05-08||osg-1.0-amd64.tar.gz||1.2 MB|
||Generate sitemap.xml by crawling the pages on example.com|
||Generate sitemap.xml.gz by crawling the pages linked to on example.com/science/|
||Generate sitemap.xml by crawling the pages linked to on a.com and b.com, crawling up to 10 pages at once|
||Generate sitemap.xml by crawling the pages linked to on example.com, showing additional information about the crawling process|
./osg without any parameters to get an overview and explanation of all available options.
- Crawls pages only once, even if they’re linked to from many pages
- Reads existing sitemap and avoids scanning pages that haven’t been updated
- Crawling an arbitrary number of URLs simultaneously
- HTTP KeepAlive and session re-use (when applicable)
- Warns if any pages on your site don’t load, return a 404 Not Found status, etc.
Customizable User-Agent (
Q: How do I install and use OSG?
A: On Windows, download and extract the zip file above, then run osg.exe either through a command prompt, or by right-clicking osg.exe, making a shortcut, changing the parameters for that shortcut (in Properties) to e.g.:
“D:.exe” -c 10 sitemap.xml example.com, and then running it. On Linux, copy the link for your architecture above, then run
curl -s <link> | tar xvz, and you’re good to go.
cd osg, and run e.g.
./osg -c 10 sitemap.xml example.com.
Q: Do I need to have WordPress, W3 Total Cache, WP Super Cache, memcached, … to use OSG?
A: No. OSG is completely indifferent to what powers your website. As long as there are links on the site, OSG can generate a sitemap for it.
Q: How do I generate an XML sitemap regularly?
A: The easiest way is to set up a cron job. On most Linux distributions you can do this by adding a cron entry using
crontab -e. The entry can be e.g.
/5 * * * /home/patrick/osg /home/patrick/sitemap.xml example.com, which will run OSG every five minutes. For more information, see Ubuntu’s Cron Howto.
(Note that Cron’s environment/path is very minimal, and you might need to use full paths to your commands.)
If you have any problems with OSG, or have a question that isn’t answered in the FAQ, please search through or send a message to the Optimus Sitemap Generator group (firstname.lastname@example.org).
Here are the highlights from the past releases to the current:
|1.0 – 2012-05-08||
Optimus Sitemap Generator (OSG) is released under the MIT license.