How to conserve your sites resources
1. Make sure you have a robots.txt file in place, make sure the paths are correct, add in a crawl delay, so bots that use this file don't hammer your site. There are a few known "bad bots" out there that can be worse then an attack on your site and consume many resources.
Add the following to the beginning of your robots.txt file
User-agent: * Crawl-delay: 30
2. Block Bad Bot's with htaccess, you can use the following info and add to your existing .htaccess file or create a new one if you don't have one.
126.96.36.199/16 rogue chinese spider
order allow,deny deny from 188.8.131.52/18 deny from 184.108.40.206/18 deny from 220.127.116.11/24 deny from 18.104.22.168/19 deny from 22.214.171.124/24 deny from 126.96.36.199/18 deny from 188.8.131.52/17 deny from 184.108.40.206/21 deny from 220.127.116.11/18 deny from 18.104.22.168/17 deny from 22.214.171.124/16 deny from 126.96.36.199/16 deny from 188.8.131.52/24 deny from 184.108.40.206/19 deny from 220.127.116.11/18 allow from all
3. Register on google's webmaster tools and set the crawl rate to "slow"
You can change the crawl rate (the time used by Googlebot to crawl the site) for sites that are at the root level—for example, www.example.com and http://subdomain.example.com. You can't change the crawl rate for sites that are not at the root level—for example, www.example.com/folder.
Note: Crawl rate refers to the speed of Googlebot's requests during the crawl process. It doesn't have any effect on how often we crawl or how deeply we crawl your URL structure.
To change the crawl rate:
The new crawl rate will be valid for 90 days.
- On the Webmaster Tools Home page, click the site you want.
- Under Site configuration, click Settings.
- In the Crawl rate section, select the option you want.
4. Turn off or increase time limit on searching for the guest Group, you can adjust this per usergroup.
- Admincp > Members Tab > Select the Guest Group > Select the Global Tab > "Can use search system" > Set to "NO"
- Admincp > Members Tab ? Select the Guest Group > Select the Global Tab > "Number of seconds for search flood control" Set this to a very high limit if you leave search on, ie: 30-60 seconds.
- Admincp > Members Tab > Select the Guest Group > Select the Global Tab > Can view member profiles and member list? > Set to "NO"
- System Settings > Advanced > CPU Saving & Optimization
You can also monitor your site's usage in cpanel, log into your account, then select HTTP CPU Usage Statistics as seen below.
Once you open this up, you will see a graph of your usage, there are two items, Realtime = the time your site's processes take to execute, and Executions = total executions on your site. You can select the Daily / Weekly view as well for more data.