FMCSA DOT Crawler

Overview

The FMCSA DOT Crawler is a robust tool designed to extract detailed information from the Federal Motor Carrier Safety Administration's (FMCSA) SAFER database. This scraper collects data about publicly registered vehicles, including legal names, contact information, addresses, and operational details. It is perfect for logistics companies, data analysts, and transportation researchers looking to gather structured FMCSA data.

Features

• Comprehensive Data Extraction: Scrapes legal names, DBA names, phone numbers, addresses, operational stats, and cargo types.
• Concurrent Crawling: Handles multiple requests simultaneously with adjustable concurrency limits.
• Error Resilience: Retries failed requests with backoff logic to handle temporary issues.
• Proxy Support: Utilizes proxy rotation to minimize blocking and ensure reliable scraping.
• Customizable Input: Configure starting URLs, request limits, and proxy settings.

Use Cases

• Building datasets of motor carriers for analysis.
• Monitoring FMCSA updates for compliance or reporting.
• Integrating carrier data with logistics systems.

How It Works

This scraper uses the Apify SDK along with Crawlee and Cheerio to process FMCSA SAFER database pages. By sending POST requests with specific parameters, it retrieves and parses the necessary data. The process includes:

1. Input Parsing: Accepts user-provided starting URLs and crawl parameters.
2. Web Crawling: Sends POST requests to the FMCSA database and fetches HTML responses.
3. Data Parsing: Processes the HTML using Cheerio to extract structured information.
4. Data Output: Stores the extracted data in the default Apify dataset.

Input Schema

The scraper accepts the following input fields:

Field	Type	Description
maxRequestsPerCrawl	Integer	(Optional) Maximum number of requests to process.
startIndex	Integer	(Optional) Starting index for DOT IDs to crawl.
proxyConfiguration	Object	(Optional) Proxy configuration settings for scraping.

Example Input

1{
2  "maxRequestsPerCrawl": 1000,
3  "startIndex": 1,
4  "proxyConfiguration": {
5    "useApifyProxy": true
6  }
7}

Output

The scraper outputs the data in the following format:

Field	Type	Description
_dot_num	String	Unique DOT number for the record.
legal_name	String	Legal name of the motor carrier.
dba_name	String	Doing Business As (DBA) name, if available.
phone	String	Contact phone number.
physical_address	String	Physical address of the carrier.
mailing_address	String	Mailing address of the carrier.
power_units	String	Number of power units operated by the carrier.
drivers	String	Number of drivers employed by the carrier.
x_cargo_carried	Array	List of cargo types the carrier is licensed to carry.

Example Output

1{
2  "_dot_num": "2802023",
3  "legal_name": "Example Logistics LLC",
4  "dba_name": "Example Trucks",
5  "phone": "555-123-4567",
6  "physical_address": "123 Main St, Springfield, IL",
7  "mailing_address": "PO Box 456, Springfield, IL",
8  "power_units": "50",
9  "drivers": "75",
10  "x_cargo_carried": ["General Freight", "Household Goods"]
11}

Need More Features?

If you'd like to add new data fields to this scraper or need a custom scraper for another purpose, feel free to file an issue or get in touch! We are open to customizing the scraper to suit your needs.

Why Choose This Scraper?

• Efficient: Processes a high volume of requests with intelligent retries and session management.
• Customizable: Tailor input options to suit your scraping needs.
• Reliable: Leverages robust error-handling and proxy configurations for uninterrupted scraping.

Proxies and Anti-blocking

This scraper uses the Apify Proxy or your custom proxy settings to reduce the risk of being blocked. It supports automatic session management for smooth operation.

Resources

• Apify SDK Documentation
• CheerioCrawler Documentation

Get Started

• Clone this scraper or use it directly on the Apify platform.
• Customize the input settings.
• Run the scraper and export the data in JSON, CSV, or Excel format.

---

Enhance your data collection capabilities with the FMCSA DOT Crawler. Get started today!