Website Content to Markdown for LLM Training
🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠
📘 Website Content to Markdown Scraper for LLM Training
This powerful Apify Actor transforms web content into clean, readable Markdown format, perfect for training Large Language Models (LLMs). It's an essential tool for AI researchers, data scientists, and developers working on natural language processing tasks.
✨ Features
- 🌐 Scrape content from multiple web pages
- 📝 Convert HTML to clean Markdown format
- 🧠 Generate high-quality training data for LLMs
- 🔍 Intelligent main content extraction
- 🚀 Fast and efficient with concurrent scraping
- 🕵️♂️ Stealth mode to avoid detection
📥 Input
Configure your scraping job with these options:
urls: List of URLs to start scraping from
📤 Output
For each scraped page, you'll get:
- 🔗 URL of the page
- 📄 Main content in Markdown format, ideal for LLM training
💡 Use Cases
- 🤖 LLM Training: Prepare web content as high-quality training data for language models
- 📚 Content Aggregation: Collect articles and blog posts for research or curation
- 📊 Web Analysis: Extract text content for sentiment analysis or topic modeling
- 📑 Documentation: Convert web-based documentation into Markdown for easy integration
- 🔍 SEO Analysis: Extract and analyze content from competitor websites
🚀 Getting Started
- Set your input parameters in the Apify console or via API
- Run the Actor and watch as it transforms web content into Markdown
- Access your results in JSON format, with Markdown content ready for LLM training or further processing
🆘 Support
If you encounter any issues or have questions, please reach out through Apify's support channels.
Transform web content into clean, LLM-ready Markdown with just a few clicks! 🚀📝🧠
Input Example
A full explanation of an input example in JSON.
1{ 2 "urls": [ 3 "https://apify.com", 4 "https://www.google.com" 5 ] 6}
Output sample
The results will be wrapped into a dataset which you can always find in the Storage tab. Here's an excerpt from the data you'd get if you apply the input parameters above:
And here is the same data but in JSON. You can choose in which format to download your data: JSON, JSONL, Excel spreadsheet, HTML table, CSV, or XML.
1[ 2 { 3 "url": "https://apify.com", 4 "markdown": "# Apify: Full-stack web scraping and data extraction platform
Apify is the largest ecosystem where developers build, deploy, and publish data extraction and web automation tools. We call them Actors.
[
### TikTok Data Extractor
clockworks/free-tiktok-scraper
Extract data about videos, users, and channels based on hashtags or scrape full user profiles including posts, total likes, name, nickname, numbers of comments, shares, followers, following, and more.
](https://apify.com/clockworks/free-tiktok-scraper)[
### Google Maps Extractor
compass/google-maps-extractor
Extract data from hundreds of places fast. Scrape Google Maps by keyword, category, location, URLs & other filters. Get addresses, contact info, opening hours, popular times, prices, menus & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.
Compass
](https://apify.com/compass/google-maps-extractor)[
### Instagram Scraper
apify/instagram-scraper
Scrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.
Apify
](https://apify.com/apify/instagram-scraper)[
### Website Content Crawler
apify/website-content-crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Apify
](https://apify.com/apify/website-content-crawler)[
### Amazon Scraper
junglee/free-amazon-product-scraper
Gets you product data from Amazon. Unofficial API. Scrapes and downloads product information without using the Amazon API, including reviews, prices, descriptions, and ASIN.
Junglee
](https://apify.com/junglee/free-amazon-product-scraper)[
### Build your own Actor
you/new-idea
Apify gives you all the tools and documentation you need to build reliable scrapers. Fast.
You? 🫵
](https://apify.com/templates)
[
### TikTok Data Extractor
clockworks/free-tiktok-scraper
Extract data about videos, users, and channels based on hashtags or scrape full user profiles including posts, total likes, name, nickname, numbers of comments, shares, followers, following, and more.
Clockworks
](https://apify.com/clockworks/free-tiktok-scraper)
[
### Google Maps Extractor
compass/google-maps-extractor
Extract data from hundreds of places fast. Scrape Google Maps by keyword, category, location, URLs & other filters. Get addresses, contact info, opening hours, popular times, prices, menus & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.
Compass
](https://apify.com/compass/google-maps-extractor)
[Browse 3,000+ Actors](https://apify.com/store)
Trusted by global technology leaders
Not just a web scraping API
---------------------------
Easily integrate
Zapierany appGitHubGoogle SheetsPineconeAirbyteKeboolaGoogle DriveSlackZapier
with Actors
--------------------------------------------------------------------------------------------------------------
Build reliable web scrapers. Fast.
----------------------------------
### We love open source
Apify works great with both Python and JavaScript, as well as Playwright, Puppeteer, Selenium, Scrapy, and Crawlee - our own web crawling and browser automation library.
[](https://crawlee.dev/)
```
1import { PuppeteerCrawler, Dataset } from "crawlee";
2
3const crawler = new PuppeteerCrawler({
4 async requestHandler({ request, page, enqueueLinks }) {
5 await Dataset.pushData({
6 url: request.url,
7 title: await page.title(),
8 });
9 await enqueueLinks();
10 },
11});
12
13await crawler.run(["https://crawlee.dev"]);
```
Publish Actors. Get paid.
-------------------------
### Reach thousands of new customers
Building and running a SaaS is hard. Building an Actor and selling it on Apify Store is 10x easier. Get visitors from day one.
#### No upfront costs
Publishing your Actor is free of charge—the customers pay for the computing resources. New creators get $500 free platform credits.
#### Rely on Apify infra
Actors scale automatically as you gain new users. You don’t need to worry about compute, storage, proxies, or authentication.
#### Billing is on us
Handling payments, taxes, and invoicing is a painful part of running a SaaS. Apify does all that and sends you a net payout every month.", 5 "timestamp": "2025-01-10T07:10:53.476Z" 6 }, 7 { 8 "url": "https://apify.com/actors", 9 "title": "Actors - fast and easy scraping in the cloud · Apify", 10 "markdown": "Actors are serverless cloud programs that run on the Apify platform and do computing jobs. They are called Actors because, like human actors, they perform actions based on a script.
### Long-running serverless jobs[](#long-running-serverless-jobs)
Apify Actors can perform time-consuming jobs that are longer than the lifespan of a single HTTP transaction.
### Publish your Actor[](#publish-your-actor)
Join hundreds of developers who share their Actors on Apify Store and earn money from coding.
[Go to Apify Store](/store)
### Auto-generated user interface[](#auto-generated-user-interface)
Actors can easily define a user interface for their input configuration. Take advantage of lower-level features and settings, or run Actors using our API.
[Learn about Input Schema](https://docs.apify.com/academy/deploying-your-code/input-schema)
Host code anywhere
Edit your code on our platform, fetch from a Git repository, or push from your machine.
Docker support
Actors run inside Docker containers on Apify servers. Use a custom Dockerfile.
Ready for scale
Run as many Actors as you need. The Apify platform provisions the necessary resources.
Custom memory and CPU
Assign each Actor any RAM volume needed. CPU share is allocated automatically.
Command-line tool
Develop and test your Actors locally, push them to the Apify platform when you're ready.
Logging
View and download logs to debug your code and monitor performance on production.
Actorize your Scrapy spiders[](#actorize-your-scrapy-spiders)
-------------------------------------------------------------
Deploy your Scrapy code to the cloud with just a few commands. Turn your Scrapy projects into Actors, run, schedule, monitor and monetize them.
[Learn more](/run-scrapy-in-cloud)" 11 }, 12 ... 13]
Related Actors
- 📄 Article Content Extractor - Extract clean article content and metadata from any web page with structured output.
- 🔍 Keyword Density Checker - Analyze webpage content for keyword density and frequency with precise calculations.
- 🤖 AI-powered Search - Transform search queries into structured AI-powered summaries with references.
- 📚 arXiv Search Scraper - Extract comprehensive research paper data with detailed metadata.
- 🔬 Nature Search Results Scraper - Extract research article data with comprehensive metadata.
- 📚 Medium Posts Search Scraper - Extract detailed article data from Medium's search results.
- 📚 Substack Posts Scraper - Scrape Substack posts and articles with comprehensive content data.
- 🌐 URL Metadata Crawler - Extract comprehensive metadata from web pages including meta tags and Open Graph data.
- 📝 YouTube Description Extractor - Extract complete descriptions from YouTube videos automatically.
- 📚 WikiHow Article Scraper - Scrape WikiHow articles with detailed step-by-step content.
- 🔍 Google News Scraper - Collect up to 5000 news articles with flexible search options.
- 📚 PubMed Search Scraper - Scrape research papers and academic articles with comprehensive metadata.
- 📚 Goodreads Book Scraper - Extract comprehensive book data and content from Goodreads.
- 📚 Medium User Posts Scraper - Extract detailed post data from Medium user profiles.
- 📚 Substack Publications Scraper - Scrape detailed publication information from Substack.
Frequently Asked Questions
Is it legal to scrape job listings or public data?
Yes, if you're scraping publicly available data for personal or internal use. Always review Websute's Terms of Service before large-scale use or redistribution.
Do I need to code to use this scraper?
No. This is a no-code tool — just enter a job title, location, and run the scraper directly from your dashboard or Apify actor page.
What data does it extract?
It extracts job titles, companies, salaries (if available), descriptions, locations, and post dates. You can export all of it to Excel or JSON.
Can I scrape multiple pages or filter by location?
Yes, you can scrape multiple pages and refine by job title, location, keyword, or more depending on the input settings you use.
How do I get started?
You can use the Try Now button on this page to go to the scraper. You’ll be guided to input a search term and get structured results. No setup needed!