The Robots Are Coming

If you have a website and you check its data with Google Analytics, you will probably notice that a (significant) part of the surfers come from different places. China, Japan, Australia, Poland, C'Yela, Argentina and the list goes on. Where do all these types come from and did the border police fall asleep on guard? Well, the answer is that many surfers are not surfers at all. That is, we are talking about entries by automatic programs known as robots. Yes, just like Agents B"Matrix" Only this time you are in the role of Keanu Reeves. 

 

The robots we are talking about are programs that make up the lion's share of web browsing traffic. In fact, according to Du"H. published in 2015, they make up about 40% (!) of all internet surfing, and a study conducted a year earlier found that robot surfing traffic is on the rise and its growth rates are about 10% from year to year. 

 

Now for the million dollar question, should you worry about it? Well, the answer is "not necessarily". Experts estimate that most of the growth in browsing traffic is from robots "friendly fire". The robots are programs that scan the websites, look for relevant information, and help collect data for the benefit of ratings and positions in the network (did someone say Google?). Their advantage lies in the fact that they are able to scan a lot of data in a short time, and therefore improve processes and upgrade the visibility of the various websites. 

 

The software company Incapsula, which deals in information security and cloud services, deeply researched the issue. boyfriend'The serious and diligent ones at Incapsulanitro information of no less than about 1.5 billion robotic visits among about 20,000 of their customers (and about 250 countries). In an ABC News interview with Mark Gapen, co-founder of Incapsula, the latter noted that the study sought to examine "What is really going on out there (on the internet)?". According to him, the research included both small websites and large and complex websites.  

 

The results indicated two main reasons for the boom in bot surfing traffic "the good ones". First, constant improvement in their technological capabilities. Second, an increase in the proportion of robots that are programmed by companies specializing in search engine optimization (in Hebrew – SEO companies). 

 

 

 

However, not everything is rosy in the robotic world. Along with the good robots that are able to index websites and gather information about them, there are also robots "malicious" which are operated by spammers, hackers and those with various interests. boyfriend'Incapsula estimates that about 30% of robots are not "are user friendly". While the relative percentage of malicious robots remains unchanged, Incapsula indicates a decrease in the number of robots that carry out spam actions on the network. The reason for this, in their opinion, is the latest updates from Google (Panda, Penguin and Co') which fight spam links and black hat SEO. 

 

Another type of robots is "Impersonating robots". These, according to their name, try to impersonate human surfers for different purposes. In the case of"reasonable" These are elements of website promotion, and in the worst case hacking and sabotage. "What all impersonating bots have in common is that they try to use a false identity", Gaffen explains in an interview with ABC, "For example, they present themselves as robots on behalf of Google, but do not behave as expected and do not come from appropriate IP addresses". Gaffen points out that the operating methods of the impersonating robots are different but the goal is always the same – Infiltrate the site and break through its security walls. 

 

Most of the security software you install will not be able to completely prevent the robots from entering your website. Moreover, you don't always want to block this traffic since a significant part of the robots (about 70%) are robots "are good". Just like good bacteria in the body. However, since the entries of the robots significantly disrupt the data you see in Google Analytics, you have the option of removing them from the statistics. In order to do this, you must enter the Admin tab in Analytics, select View Setting and mark the field at the bottom "Bot Filtering".