urllib.robotparser
Stdlib — NetworkingPython 2.0+Advanced
Parse robots.txt files for crawling permissions
Quick Info
- Documentation
- Official Docs
- Python Version
- 2.0+
- Dependencies
- None — Python Standard Library
- Install
Included with Python
Learn by Difficulty
Quick Example
python
import urllib.robotparser print(f"Module: urllib.robotparser") print(f"Contents: {dir(urllib.robotparser)[:10]}")
The urllib.robotparser module is part of Python's standard library. Parse robots.txt files for crawling permissions.
Try in PlaygroundTags
stdlibnetworkinginternet