Lidl Category Scraper

🔎 What is the Lidl Category Scraper?

The Lidl Category Scraper is a web scraping tool designed to extract product data from Lidl online store category pages. This actor retrieves product information from specified category URLs, providing structured data for various analytical and monitoring purposes. It leverages the Lidl API for reliable data extraction.

🧾 What data can the Lidl Category Scraper extract?

The Lidl Category Scraper extracts the following data points for each product listed on a Lidl category page:

• code: The unique product code assigned by Lidl.
• label: The product name or description.
• type: The product type.
• url: The direct URL to the product page on Lidl.de.
• gridbox: A dictionary containing detailed gridbox data from Lidl's API, providing additional product attributes.
• price: A dictionary containing the product's price information:
  • amount: The product's price amount.
  • currency: The currency of the price (e.g., EUR).

💼 What use cases does the Lidl Category Scraper support?

The Lidl Category Scraper is valuable for a range of applications where monitoring Lidl's product catalog and pricing is important:

• Price Monitoring: Track price changes for specific products or categories over time.
• Competitor Analysis: Compare Lidl's product offerings and prices with those of other retailers.
• Product Research: Gather data on product specifications, types, and availability for market research purposes.
• Inventory Tracking: Monitor Lidl's product catalog to identify new products or changes in existing product listings.
• Automated Catalog Updates: Integrate scraped data into your own product catalog or database for automated updates.

📖 How to use the Lidl Category Scraper?

1. Create a free Apify account: Sign up for a free Apify account at https://apify.com.
2. Open the Lidl Category Scraper: Navigate to the Lidl Category Scraper on the Apify Store (replace this with the actual Apify store link if available).
3. Enter the Category URL(s): Provide one or more Lidl category URLs from which you want to scrape product data.
4. Configure Maximum Offsets (Optional): Adjust the max_offsets parameter to limit the number of API requests per category (each offset retrieves 100 products). This helps prevent the scraper from running too long on large categories and manage API usage.
5. Start the Scraper: Click the "Start" button to initiate the scraping process.
6. Download the Results: Once the actor has finished running, you can download the scraped data in JSON, CSV, or other supported formats from the actor's dataset. You can also access the data programmatically via the Apify API.

📥 Input

To run the Lidl Category Scraper, provide the following input parameters:

• start_urls (Required): An array of Lidl category URLs to scrape. Each object in the array should have a url property. Example:

  1[
  2    {"url": "https://www.lidl.de/h/hoehenverstellbare-schreibtische/h10036915"},
  3    {"url": "https://www.lidl.de/h/gartenmoebel/c13272"}
  4]
  

• max_offsets (Optional): The maximum number of offsets to scrape per category. The actor fetches 100 products per offset. A higher number will scrape more products, but will also consume more resources and take longer. Defaults to 999. Use if you don't need to get all products.

  1{
  2  "max_offsets": 50
  3}
  

  This would scrape 50 * 100 = 5000 products per given category. Important: The actual limit of products per category may also have a limit on Lidl servers.

  Full example input:

  1{
  2  "start_urls": [
  3    {"url": "https://www.lidl.de/h/hoehenverstellbare-schreibtische/h10036915"}
  4  ],
  5  "max_offsets": 50
  6}
  

  This is a full example that combines both available parameters.

📖 How to use the Scraper?

1. Create a free Apify account
2. Open Lidl Lidl.de Onl Scraper
3. Add either a search URL or a search query
4. Click Start and wait for the results
5. Download the results in JSON, XML, or CSV format or connect the actor to your backend via API

📥 Input

To run the actor, simply enter either a:

• Search URL: The URL of the search you want to scrape (e.g., https://www.lidl.de/h/hoehenverstellbare-schreibtische/h10036915)
• Search Query: The name of the object you want to search for (e.g., Basecap)
• Maximum number of items: The maximum number of items to scrape. The actor will stop when it reaches this limit (e.g., 10)
• Proxy configuration: Choose between default proxy settings, Apify's RESIDENTIAL proxy, or a custom proxy URL

You can customize the search further by using a startURL from a search you did directly on Lidl.de. This will also allow you to filter based on location. If you enter both a start URL and a search query, the actor will perform both actions.

📤 Output

The results are stored in the default dataset associated with the actor. Each item is an ad, having the following format:

1{
2    "code": "100233919",
3    "label": "Zaun-Anbauset Chaussee, zum Aufschrauben",
4    "type": "product",
5    "url": "https://www.lidl.de/p/zaun-anbauset-chaussee,-zum-aufschrauben/p100233919",
6    "gridbox": {
7        "country": "DE",
8        "sequence": 1738815147602001,
9        "data": {
10            "ageRestriction": false,
11            "awards": [],
12            "brand": {
13                "logo": "https://www.lidl.de/assets/gcp0431aea4457a49dbaecb31eb60a91cca.jpeg",
14                "name": "ALBERTS",
15                "url": "/q/search?q=alberts",
16                "id": "GAHA",
17                "showBrand": true
18            },
19            "canonicalPath": "/p/alberts-zaun-anbauset-chaussee-zum-aufschrauben/p100233919",
20            "canonicalUrl": "/p/alberts-zaun-anbauset-chaussee-zum-aufschrauben/p100233919",
21            "category": "Kategorien/Baumarkt/Bauen & Renovieren/Z\u00e4une, Tore & Zubeh\u00f6r",
22            "categorySecondaryPath": "0/41/4112/411223",
23            "dealOfDay": {
24                "active": false
25            },
26            "designTheme": "default",
27            "disclaimers": [
28                {
29                    "id": "code_100233919_a428dd36",
30                    "type": "CODE",
31                    "value": "Basic"
32                },
33                {
34                    "id": "code_100233919_b3c9c3d6",
35                    "type": "CODE",
36                    "value": "Price"
37                }
38            ],
39            "displayFreeDelivery": false,
40            "energyLabels": [],
41            "erpNumber": "100233919",
42            "flashSales": false,
43            "fullTitle": "ALBERTS Zaun-Anbauset Chaussee, zum Aufschrauben",
44            "gs1Attributes": [],
45            "havingPrice": true,
46            "havingThreesixty": false,
47            "havingVariablePrices": true,
48            "havingVideos": false,
49            "highlightedSeals": 0,
50            "ians": [
51                "100233919"
52            ],
53            "image": "https://www.lidl.de/assets/b671ffe5448a18e934a2f8922478458a.jpeg",
54            "isUsedProduct": false,
55            "imageList": [
56                "https://www.lidl.de/assets/b671ffe5448a18e934a2f8922478458a.jpeg",
57                "https://www.lidl.de/assets/5423a04e2b8f386740358879e9e973cf.jpeg",
58                "https://www.lidl.de/assets/da6b3effc107c07d5efc445507ca0fcf.jpeg"
59            ],
60            "keyfacts": {
61                "analyticsCategory": "Kategorien/Baumarkt/Bauen & Renovieren/Z\u00e4une, Tore & Zubeh\u00f6r",
62                "description": "Inklusive Zubeh\u00f6r<br />",
63                "fullTitle": "ALBERTS Zaun-Anbauset Chaussee, zum Aufschrauben",
64                "moreDetails": "Weitere Produktdetails",
65                "supplementalDescription": "Anbau-Set f\u00fcr Chaussee-Grundset <br />Zaunh\u00f6hen: ca. 1.000 mm oder 1.200 mm <br />Farben: anthrazit oder schwarz-matt",
66                "title": "Zaun-Anbauset Chaussee, zum Aufschrauben"
67            },
68            "lidlPlus": [],
69            "multipack": false,
70            "online": true,
71            "preventSelling": false,
72            "price": {
73                "currencyCode": "EUR",
74                "currencyCodeSecond": "",
75                "currencySymbol": "\u20ac",
76                "currencySymbolSecond": "",
77                "discount": {
78                    "deletedPrice": 336.02,
79                    "discountHasStar": false,
80                    "fromRecommendedPrice": true,
81                    "percentageDiscount": 33,
82                    "showDiscount": true,
83                    "showFrom": false,
84                    "showPercentage": true,
85                    "showUpTo": false
86                },
87                "hasStar": true,
88                "hasVat": true,
89                "price": 225.04,
90                "recommendedPrice": 336.02,
91                "specialTaxes": [],
92                "variantsHaveDifferentPrices": true
93            },
94            "productId": 100233919,
95            "productType": "HEAD_VARIANT",
96            "productOrigin": "progress_event",
97            "quickAddToCart": false,
98            "regions": [],
99            "ribbons": [],
100            "renderedTs": 1738815147,
101            "seals": [],
102            "stockAvailability": {
103                "availabilityIndicator": 3,
104                "badgeInfo": {
105                    "badges": [
106                        {
107                            "text": "lieferbar",
108                            "type": "AVAILABLE_ONLINE"
109                        }
110                    ]
111                },
112                "onlineAvailable": true,
113                "minOrderableQuantity": 1
114            },
115            "store": false,
116            "title": "Zaun-Anbauset Chaussee, zum Aufschrauben"
117        },
118        "productId": 100233919,
119        "meta": {
120            "sapProductCategory": {
121                "code": "411223",
122                "name": "Z\u00e4une, Tore & Zubeh\u00f6r",
123                "pathCode": "0/41/4112/411223",
124                "pathName": "Kategorien/Baumarkt/Bauen & Renovieren/Z\u00e4une, Tore & Zubeh\u00f6r"
125            },
126            "campaignPaths": [],
127            "fullTitle": "ALBERTS Zaun-Anbauset Chaussee, zum Aufschrauben",
128            "preview": false
129        },
130        "action": "UPDATE",
131        "language": "de",
132        "eventType": "GRIDBOX",
133        "id": "100233919"
134    },
135    "price": {
136        "amount": 225.04,
137        "currency": "EUR"
138    }
139}