This package is part of the SeoToolkit package, but can be installed/used on its own! The SeoToolkit package is a v9/v10 package that combines multiple functionalities in one like redirects, meta fields, robots.txt and more. It is aimed at the idea of making SEO easier to use within your website and take less time to implement it.
Robots.txt gives you an easy interface to edit your robots.txt. After installing the package, you'll get the /robots.txt path that'll display your configured robots.txt. Everything about the robots.txt is editable in the CMS itself. There is also validation on your robots.txt, so that your robots.txt is actually valid.
You can get started with this package by downloading the following nuget package:
dotnet add package SeoToolkit.Umbraco.RobotsTxt
If you are running on v10, use the 2.x versions. If you are using v9, use the 1.x versions.
Documentation can be found here: https://seotoolkit.gitbook.io/useotoolkit/
How to use
After installing, you should now see a new section called SeoToolkit. If you don't see it, you may have to give yourself access to it through the users section.
Then go to the Robots.Txt node and click on it. It'll show you the robots.txt for your website. Here you can customize it to your liking.