What exactly is robots.txt file?

The robots exclusion protocol (REP), or robots.txt is a text file which stands at the root of a web application, indicating those parts of a website that you don’t want to be accessed by the Search Engine Crawlers.
The robots.txt file uses the Robots Exclusion Standard, a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers).

Now if you’re on a way to building a web application, you should consider the fact that you’ll need a robots.txt only if your site includes content that you don’t want Google or other search engines to index, thereby blocking URL’s from search engine crawlers.

Understand the limitations of robots.txt
Before you build your robots.txt, you should know the risks of this URL blocking method. At times, you might want to consider other mechanisms to ensure your URLs are not findable on the web.

Robots.txt instructions are directives only
The instructions in robots.txt files cannot enforce crawler behavior to your site; instead, these instructions act as directives to the crawlers accessing your site. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not. Therefore, if you want to keep information secure from web crawlers, it’s better to use other blocking methods, such as password-protecting private files on your server.

Different crawlers interpret syntax differently
Although respectable web crawlers follow the directives in a robots.txt file, each crawler might interpret the directives differently. You should know the proper syntax for addressing different web crawlers as some might not understand certain instructions.

Your robots.txt directives can’t prevent references to your URLs from other sites
While Google won’t crawl or index the content blocked by robots.txt, we might still find and index a disallowed URL from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the site can still appear in Google search results. You can stop your URL from appearing in Google Search results completely by using your robots.txt in combination with other URL blocking methods, such as password-protecting the files on your server, or inserting indexing directive meta tags into your HTML.

You can test your robots.txt with the robots.txt Tester