By default, KillBot blocks all simple HTTP bots (except search engines and other trusted bots from the whitelist). The following types of bots will be blocked:
- checkers and validators,
- SEO analysis tools,
- bots that scan robots.txt and XML files,
- technical export bots (for example, 1C integration bots),
- any other third-party bots that are not search engines and are not included in the KillBot whitelist.
Without additional settings, KillBot allows access to:
- search engine bots (Baidu, Google, Yandex etc.),
- messenger bots,
- other known and useful bots from the whitelist (Telegram, Facebook, etc.).
All other bots are blocked by default.
You need to allow access if you need:
- export products from your website,
- SEO analysis tools to check your website (title, description, page structure),
- generate XML feeds for ads or marketplaces,
- allow website images, CSS, JavaScript and other resources to be accessed from outside the website,
- and similar tasks.
- these bots need to be allowed manually.
How to allow bots in KillBot
0. Temporarily disable KillBot Screen during scanning
This is the easiest option if you do not want to find and add bot IP addresses or User-Agent values manually.
Go to the "Integration Method" settings and select "DNS Protection: OFF".
Remember to enable it again after the scanning process is finished.
Then click "Save". Do NOT click "Reload DNS".
1. Allow access by IP address
If a bot or analyzer has a fixed IP address, the safest method is to add this IP address to the allowed list.
This method is suitable for:
- SEO services,
- business integrations,
- internal data export systems.
IMPORTANT! Add your website server IP address to the allowed list. This allows KillBot to pass all internal requests from your own server.
2. Allow access by URL path
You can allow access to a specific URL path for any IP address.
Examples:
- product XML feed path,
- 1C export path,
- any other URL paths that can be accessed from different IP addresses.
3. Allow access by User-Agent
You can allow all visitors with a specific User-Agent.
Please note: your website will also be accessible to any other bot that uses the same User-Agent.
Add User-Agent values like this: KillBot allows requests when the User-Agent contains all specified words (separated by dots).
Example: Yandex.bot allows any User-Agent that contains both words: "Yandex" and "bot".
Important: do not remove the default values (yandex, google), otherwise search engines will not be able to index your website.
Allow access to website images and other resources
For example, if you allow a product XML feed for Yandex:
- the XML file usually contains links to product images,
- to process the feed correctly, image URLs must also be allowed.
By default, KillBot blocks opening images directly from outside the website.
To allow image export, add the image path of your website to the allowed list.
Image URLs are often targeted by DDoS attacks
Many websites crash when the same image is downloaded by 100–200 connections at the same time. This is a common DDoS scenario, and the hosting provider may temporarily disable your website. For users this looks like a serious problem, but usually it is caused by too many parallel downloads of the same file.
Therefore, when allowing access to images, you should consider this risk.
What to do if I do not know the bot IP address or User-Agent?
In this case, check the verification page header returned by KillBot.
KillBot adds the bot IP address and User-Agent to the page header. This makes it easy to find the required information and add the bot to the allowed list.
Example: I want to allow access for the checker https://page-speed.ru/tools/metatags/
If I enter my website address into this tool, the checker will receive the KillBot verification page. The page header will contain the IP address that made the request to KillBot. This IP address can be added to the allowed list.
If the analysis tool does not show the page header, many tools provide the received HTML code instead. In this case, you can find the User-Agent or IP address inside the HTML code.
To allow the analyzer shown in the screenshot above:
- - either add IP address "52.22.66.203" to the allowed list,
- - or add "Validator.nu" to the allowed User-Agent field.
Summary
- KillBot blocks all bots by default except those that are allowed to access your website. This is normal and provides better protection.
- The safest options are allowing access by IP address or by specific URL paths.
- Be especially careful with images and other resources, because they are often used for attacks.
How to allow access for Screaming Frog SEO Spider Website Crawler
This program runs on your own computer, so its IP address can change.
The easiest way is to add a direct connection to your domain using the hosts file.
How to edit the hosts file:
-
Open Notepad or another text editor as Administrator.
Otherwise, Windows will not allow you to save the changes. -
In the editor, click File → Open and enter this path:
C:\Windows\System32\drivers\etc\hostsYou may need to change the file type to "All Files" at the bottom of the window.
-
Add a line in this format:
123.123.123.123 domain.com
Save the file and close the editor.
You can check if the settings are correct:
Open Command Prompt (CMD) or PowerShell in Windows 11.
Press Win + X and select Terminal.
You can also right-click the Start button and choose Terminal.
In the terminal, enter:
ping domain.com
The result should show the IP address of your own server, not the KillBot Screen IP address.
If you see the KillBot Screen IP address, it means:
- the hosts file was not edited with Administrator rights, or
- the file was not saved correctly.
Try editing the file again.
Please be careful when using Screaming Frog SEO Spider Website Crawler.
This program can create a high load on your server. If you use cheap shared hosting, your hosting provider may block your website because of high resource usage.
This has already happened with one of our customers.