What are 'Crawlers & Bots'?

[Muthu]

Crawlers and bots are more or less the same they are used to automate the work which is repetitive.

Crawlers are used by search engines to make a copy of the webpage in their database, consider a google bot / crawler visits your page then google will be having a copy of your webpage and analyze the page and rank them accordingly, Also they make a list of links available in your website and form a hierarchical order of webpages available, making one page as home and others as child pages. This type of crawling makes the search engines check for new pages available, so it can update its (search engines) database.

Some times bots are deployed for many reasons like IRC bots (scripted bots) available in chat rooms for blocking spammers and others who misuse their service. Wikipedia also uses bots for analyzing the content of a webpage to check for copyright violation, archiving old pages etc.. . There are also many kinds of bots available like spamming bots ( usually in good old Y! Messenger chat rooms), DDOS bots (malware infected systems which perform DOS attack on certain IP when commanded) ...

List of wiki bots used: en.wikipedia.org/wiki/Wikipedia:Bots

Since you are owning a website take a look at the link to a video

https://www.video2brain.com/en/lessons/s...d-crawlers

~Hope this clarifies you ..