Elements to consider when using historical data from web scraping

Subscribe now

The Context

DataBoutique.com recently enabled historical data access, a long awaited feature that unlocks value both for data buyers and sellers.

Simply put, it allows one-click access to past data, offering different options in terms of cost and granularity. And for the benefit of many, keeping the same data-schemas adopted on current data collections.

When to use historical data

See the past (to predict the future)

The most common quesiton an historical dataset can answer, is “how were things in a specific point in time?” or “When did this thing start?”, and ideally capture trends that can help us have a clearer forecast for the future. “Is the price of a product growing?”, “Is the number of hotels in that reservation website rising?”

A variation on this is to simply check what happened on a particular date or who first did something. “Was this product cheaper on that day?” “Which website started the discounts first?” “Was this retailer selling this product also before that date?” And so on.

Websites like wayback-machine are inadequate to provide proper time-tracking of fast-moving websites, like e-commerce or reservation platforms, getting web scraped historical data is the only way to answer these questions.

Back-test hypotheses

We can take it one step further and look at the past to test hypotheses. “Is a heavy-discount practice on an e-commerce anticipatory of poor website performance?” “Was the competing retailer running discounts when my website suffered a decline in sales?” “Are those two brands synching their price-change strategies?”.

This approach is more complex than just “looking at the past,” as it often involves looking for a statistical correlation between two phenomena, higher granularity of the dataset, and longer timeframes for testing.

Longer timeframes imply that either you or someone in your organization has had enough foresight to start scraping that website two years before you needed it, or you find someone who is already (consistently and continuously) doing it (that’s what Data Boutique is for).

Train A.I. (and other fun stuff)

An even more advanced use case of historical data is to train automations: Once you’ve seen the past, once you have tested your hypotheses, you can train algorithms, AI, or other decision-making processes to take action when specific conditions happen. Stop discounting as soon as the competitors stop, dynamically adjust prices, automatically buy an item when price conditions are met, or raise a red flag when a distributor does something they are not supposed to.

Whatever you want to build, test it on the past, or (recommended) train AI to do that for you.

Things to consider

Here’s a list of essential elements to remember when approaching historical data. The topic is more complex than this, but we’ll start with the founding blocks:

History length

Arguably, the most basic element to consider is when the collection started. Now, with traditional datasets (financial transactions, stock prices, air temperature, etc.), we are used to seeing very long time series dating back decades.

In web scraping, the situation is different, to say the least. With few notable exceptions (data providers targeting a single website for years), finding datasets with less than a year or just a few months of history is not infrequent. This is part of web scraping: It costs money to keep collecting data from a website, and there are simply too many websites to choose from. While scraping today’s website content can be considered a commodity (it doesn’t really depend on who is doing it, as long as the quality is met), historical data is a differentiating factor.

Frequency

Historical data can be offered at different levels of granularity. One scan per month can be enough in some cases, while others require higher frequencies (weekly or daily).

The finer the granularity, the more information (and noise) can be found, but it is also heavier to manage. On Data Boutique, we offer three levels:

• Monthly: One snapshot per month (12 a year) is ideal for simple research, long-term trend analysis, and initial exploration for finer work to be done on more granular frequencies (as it’s cheaper)

• Weekly: One snapshot per week (52 a year) is approximately four times larger than the monthly one. It’s quite bulky but perfectly suitable for most back-testing and training purposes. Given its higher cost, we recommend trying a cheaper run with the monthly one before evaluating it as an effective way to assess its potential.

• Daily: The most granular historical dataset we allow on the platform. This is for heavy-duty usage (like some revenue-estimating projects).

Quality Factors

What quality elements impact time series? We name a few:

• Completeness: Are data points missing? Are there significant gaps in the collection? A continuous collection is preferable, but as experience tells us, web scraping has quite a bumpy pipeline (we all confide in AI-aided scrapers to fix that). Gaps in the collection, unfortunately, do happen. Data Boutique provides a completeness indicator designed precisely for this.

• Point-in-time and gap-filling: Gap-filling is the technique of " filling the gap” in history by interpolating two data points. The opposite (leaving this as it is and not filling the gap or changing the content afterward) is point-in-time data. Data Boutique is committed to delivering data as close to the original format as possible, delivering point-in-time data. This ensures that buying historical data has the same result as buying it as it gets published.

• Date-picking method: When creating a monthly or weekly time series, data providers often choose one specific date for their collection (e.g., every first/last day of the month/week). On Data Boutique, these dates are homogeneous and defined as the last day of the month/week (or the closest available).

• Quality Assurance: Quality procedures (domain, completeness, consistency, and ground truth checks) on Data Boutique are the same as those applied to current data. Again, this is to provide a uniform service for buyers who get historical data vs. those who get the data over time as it gets published.

Final remarks

Using historical data can be enormously powerful but carries a heavier workload. The ones listed here are just some of the elements to consider.

As always, we encourage the community of developers and data providers to join the conversation on our channels (we have a friendly Discord server). We are happy that some data sellers activated this option (historical data is an opt-in feature), and that some users have already experimented with historical data purchases.

Historical data has also been included in data bundles (our cost estimation tool for large data packages), and we’ll soon add more features to play with.

That was all for this edition.

Thanks for reading,

Andrea

About Data Boutique

Data Boutique is the data marketplace for web scraping. We make buying and selling data faster and safer for everyone.

Subscribe now

Web scraped historical data access

Subscribe now

In Brief

You can now purchase historical data for web-scraped datasets on DataBoutique.com. This allows you to access data collected in the past and integrate it into your applications, whether for analytics, SaaS, or AI.

Highlights:

• Consistent structure: Historical data maintains the same structure as current data, with identical fields, definitions, and quality controls. You can seamlessly add future data to the same structure and use the same ETLs.

• Flexible pricing: Choose from daily, weekly, or monthly details to adapt your budget to your specific use case.

• Easy testing: Free samples are available, allowing you to review the data before making a full purchase.

What is Historical Data

Definition

Historical data refers to web data collected by our vendors in the past. This allows data buyers to access the entire collection history of each vendor and enables vendors to further monetize their past efforts.

Historical data includes:

• Removed Data: Information that is no longer available on a website, such as discontinued e-commerce products.

• Altered Data: Data that has been updated or changed, such as prices, locations, and the number of reviews.

• Seasonal Phenomena: Data on seasonal events like discounts or promotions valid only during specific times of the year.

While the Internet doesn't retain past data, capturing it allows us to recreate and utilize it effectively.

Benefits and Use Cases

Access to historical data is particularly valuable for applications such as:

• Market trend analysis

• Competitive benchmarking

• Legal compliance

• AI training

In market analysis, historical data helps identify trends over time, such as seasonal changes and shifts in consumer behavior. This enables businesses to make better strategic decisions and predict future market conditions.

For market analytics, historical data allows companies to analyze past market dynamics and competitor actions, helping them optimize their strategies and stay competitive.

Overall, historical data is a powerful tool for improving AI, understanding market trends, and enhancing business decisions.

Features of Historical Data

Structure

Historical data has the exact same structure as current data, which means:

• Schema Consistency: You can find the structure definition under the “Schema” section of the platform.

• Seamless Integration: Use the same ETLs to integrate it as you do with current data, with minimal changes required.

• Expandable Database: Append future data to the historical collection effortlessly, allowing your database to grow over time.

Pricing

Price transparency is our cornerstone. Historical data pricing is structured to fit smoothly into your budget:

• Clear Pricing: Prices are clearly stated for all options.

• Cost Breakdown: Prices are linked to the cost of extraction, the number of snapshots included, and the depth of history. Shorter histories are priced lower, while more granular details reflect in the overall cost.

• Flexible Packages: Choose from different packages based on the level of detail needed (daily, weekly, or monthly).

Testing

Trying before buying is crucial for data purchases. Aside from free samples, which also apply to historical data, users interested in high-granularity details (like daily data) can test the collection's effectiveness with low-granularity data (like monthly data) at a fraction of the cost before committing to a larger purchase.

How to Access Historical Data

For data buyers

When a data vendor opts to activate history, you will find the available historical data details listed on the dataset. You can purchase the historical data separately from the current data extraction by clicking the dedicated button below each historical data option.

For example, The Cettire US pricing dataset, currently trading at 9.00 EUR per collection (the full catalog of a single day of the e-commerce website)

has a history dating back 11 months (if you read this post in the future, you’ll find a longer history) for 99.00 EUR

A 50 weeks of historical data (weekly detail) at 360.00 EUR

Or a daily history, dating back to June 15th, 2023, for 1.521.00 EUR

Hitting the “buy” button will give you access to the history file you’re looking for.

For data sellers

Activating history is an opt-in feature, and soon it will become a self-service option. If you want early access to sell the historical data you have already uploaded or wish to add historical data collected before joining DataBoutique, please contact us on Discord or simply reply to this email.

QA and Pricing of Historical Data

QA (Quality Assurance)

The same QA processes used for current data are applied to historical data. Historical data is submitted by data vendors over time as current data. Once validated, it is stored and made available for bulk purchase if the vendor agrees. This includes:

• Domain Validation: Datasets are linked to a schema, with each field validated against a domain (e.g., currency, country codes, prices). This ensures standardization and excludes unwanted content, such as personally identifiable information (PII).

• Website Validation: Before acceptance on DataBoutique, all websites are verified for public accessibility and compliance with click-wrap terms of service.

• Completeness Validation: Peer review verifications (independent checks on expected content) are continuously performed on submitted content.

• Time-Series Consistency Validation: The content (e.g., price trends, and item counts) is checked over its own history to ensure the stability of published data.

Pricing

The cost of extracting data varies significantly from website to website, depending on the effort and expenses required. Pricing is set by data vendors, with historical data following the same principles.

Historical data is priced based on the number of snapshots contained in the selected historical detail. The more data points a history file has, the higher the price, reflecting the single extraction cost, including a discount factor depending on the time length.

Next on Our Roadmap

Enabling historical data was a significant milestone on our roadmap. Next, we plan to:

• Integrate Historical Data into Data Bundles: Make historical data purchases available as part of our "data bundles."

• Activate "Request to Activate History" Feature: Allow buyers to request the activation of historical data from sellers.

• Link Historical Data Frequency to Future Updates: Connect the frequency of historical data updates to future data updates for seamless integration.

Stay tuned for coming releases, and spread the word, in case this email was forwarded to you.

About Data Boutique

Data Boutique is the data marketplace for web scraping. We make buying and selling data faster and safer for everyone.

Subscribe now

Mapping actors and solutions

Subscribe now

Why a Map?

The “Map” is a global directory of all actors working with web data: from extracting to cleaning, trading to analyzing, and turning it into insights.

We needed a map because we trade with so many types of users that multiplying the connections between them is highly valuable for everyone, including us.

Drawing inspiration from similar initiatives like Crunchbase, we have opened our archives to the public for collective enrichment. Originating from our personal notepads, Notion, and Airtable documents, it is now a community-driven resource.

Web Data is a very fragmented ecosystem, which is thrilling, but dispersion can sometimes be a drawback for growth. Hence the initiative.

Want to be included? Read on.

Who can be on the Map?

Actors

The actors involved are companies, organizations, and freelancers (within the boundaries of privacy laws) who explicitly offer solutions related to web data. Regardless of size, geography, or sector, anyone who helps access, use, and make sense of web data.

Everyone can suggest a company to be included in the list.

Solutions

A solution is a service commonly offered to the market used for, with, or in relation to web data. The guideline is that they clearly relate to web data.

Mobile proxy, dynamic pricing, competitive intelligence, data marketplaces are valid solutions; enterprise ERP, CRM are not.

Solutions are specific (i.e. Hotel Dynamic Pricing, Residential Proxy) and don’t refer to specific providers or product names.

It is the solution that allows an actor to be listed on a map, even if a company covers various services, but one of them is in relation to web data (i.e. they do consult on how to integrate web data) they are allowed - as long as there is an explicit reference in their service portfolio.

Actors can be associated with more than one solution, and this can evolve over time. Any use can suggest a new solution and can suggest new or ceasing relationships between actors and solutions.

Who can contribute

Building an ecosystem map is a collective work. Just like the Crunchbase model, anyone can suggest entries. Contributions will be moderated, with the purpose of keeping high-quality in the map.

Actors and Solutions can be suggested by:

1. Registered Users: Any individual can sign up for a free account on Data Boutique and start contributing. Registered users can add profiles of companies, solutions, and individuals involved in the web data ecosystems.

2. Company Representatives: Individuals affiliated with a company, such as founders, employees, or authorized representatives, can claim their company's profile on Data Boutique. By claiming a profile, they can update and manage the information presented about their company, ensuring it is accurate and up-to-date, as well as posting news and links to external material.

How to Consult the Map

The Ecosystem map is public; anyone, registered or not, can access it.

The map is organized in Actors, the directory of companies, and Solutions, the directory of solutions. Both Directories are interrelated by actor-solution links.

Go to Ecosystem Map

How the Ecosystem Map works

Our model operates under straightforward rules: To qualify for inclusion, actors must provide a verifiable service related to web data.

We categorize these services broadly, accommodating a diverse spectrum from scalable one-to-many services—like proxy network providers and scraping tools—to bespoke one-to-one solutions tailored for specific needs, such as custom data extraction or consultancy for business intelligence and analytics. Additionally, our map includes everything from basic, low-level raw data services (data feeds) to more complex, high-level offerings that provide insights (market intelligence) and everything in between.

The Benefits of Being on the Map

Actors: Exposure and Lead generation

Organizations benefit from the map by gaining exposure to a wider, targeted audience at the exact moment when they are looking for a solution. They state clearly what their services are and where they position themselves in the market.

Posting relevant updates enables brand recall and awareness to a specific audience, clustered by search intent (technology, sector, solution). The ability to post CTA links in the posts also enables lead generation.

Users: Fast access to service providers

Data buyers can find who is offering the solutions they are looking for, have faster access to solutions, and understand the different offerings. The faster users extract value from data, the more they use it, which - in the end - is our goal.

Want to be featured?

You’ve reached the right place! Simply go to databoutique.com, register for free, and submit your company from the actor’s page of the Ecosystem Map.

We’ll post more on this map's evolution in this newsletter's coming editions!

About Data Boutique

Data Boutique is the data marketplace for web scraping. We make buying and selling data faster and safer for everyone.

Subscribe now

Working with data? List your solution on Data Boutique

Subscribe now

An Open Platform Optimized for Web Data Utilization

Data Boutique’s mission is crystal clear: Maximize web data adoption.

To achieve that, we built a platform with the highest exposure to web data extractions. We stay agnostic about how data is used. It could be embedded in SaaS solutions, market research, or AI training.

But not all users require raw data. Often, the (in)expressed need is for more complex solutions that require enrichment, transformation, aggregation, and extraction of insights, often in a graphical form.

End-users might land on Data Boutique when attracted by the “website name” they want to monitor (IKEA, H&M, Nike, Gucci, Tesco, Douglas, Walmart etc), but the solution they need might go beyond raw data.

That is why Data Boutique hosts third-party solutions: Market intelligence, dynamic pricing, research, consultancy, AI, and commerce analytics that are ready to be used. But also custom extraction services, scraping technology as well as data handling, vertical industry expertise, and additional research. Raw data can’t talk, without the help of these expertise.

By being hosted on Data Boutique, they are shown to the user right when searching for that data.

Who Lists on Data Boutique?

• Scraping Technology Providers: Our community is primarily made of data scrapers, both freelancers and data farms, who represent the ideal audience for scraping technology providers, from anti-blocker solutions to AI-based scraping and proxy network providers.

• Data providers: Even if not actively selling on Data Boutique, web data providers are present on the platform to intercept non-standard, custom-made requests from data buyers; these can range from scraping websites not listed on Data Boutique to performing custom frequency extractions.

• SaaS and AI platforms: Market intelligence, commerce analytics, and industry-specific tracking tools have the perfect spot to showcase their platform, capturing users in their exact point of attention

• System integration and consulting: Companies building custom solutions and bringing their vertical industry expertise to end-users

• ETLs, Quality Assurance, BI, ML tools: A lot is happening in the big data scene, including technology, databases, and professions. Emerging solutions can meet new users when they are searching for data.

Scraping technology providers currently listed

Scraping solution profile page of Kamaleo

Industry expert in grocery

Why list on Data Boutique?

Data Boutique uses a bottom-up approach to make data easily discoverable by data scientists, market analysts, and other stakeholders. This means the audience is diverse in geography, industry, and organization size.

This makes listing on Data Boutique interesting for solution providers for the following reasons:

Extended reach

Data Boutique’s reach is niche but capillary, extending the audience of the solution provider and raising its awareness in an otherwise very fragmented market.

Brand Recall

Solution providers reinforce their message with additional touch points when their existing clients enter Data Boutique to avoid dispersion.

Lead Generation

Custom Call-to-Action can convert interest to leads alongside use cases and marketing material.

Value for Data Boutique

Listing solutions on Data Boutique is free. The goal is dual: not to disperse users who are not looking for raw data and to incentivize them to use data-based solutions, ultimately leading to more demand for raw data.

How to List a Solution?

If you work with data and would like to feature your solution, the first step is to create a company profile. From your profile, click the “Add company” button and follow the instructions to create a solution.

If you want more info on how to list your solution, reach out, and let’s talk!

List your Solution

About Data Boutique

Data Boutique is the data marketplace for web scraping. We make buying and selling data faster and safer for everyone.

Subscribe now

Case Study: MatchesFashion Inventory Outflows

In a previous discussion, we explored the utilization of web-based data to monitor Key Performance Indicators (KPIs) during significant market events, including the instance of the closure of MatchesFashion.com.

This detailed examination focuses on the strategic application of inventory level data, acquired through web scraping techniques, to enhance competitive and investment research within the retail sector.

Analytical Perspective: Understanding Market Dynamics

Our investigation centers on the impact of a major retailer, with annual net revenues surpassing 400 million USD, ceasing operations during a peak period of full-price sales. This situation often leads to market disruptions, as competitors may face challenges when inventory is released into the market at substantially reduced prices.

Research Hypothesis: Evaluating Inventory Redistribution

We aim to assess the reallocation of inventory in the wake of a retailer's closure. By comparing inventory data at a detailed SKU level from March 11th, 2024, with data from March 20th, we can observe changes in inventory levels across different brands.

Key Considerations

• The analysis period spans nine days, a concise timeframe that, despite its brevity, offers valuable insights due to the recency of the event. Analysts are encouraged to extend this research to gain a broader perspective, as data accessibility is assured.

• We assume no new inventory additions during this period, considering the retailer's declared cessation of business. Therefore, observed net changes are attributed solely to inventory outflows.

• It's important to note that inventory reductions are not exclusively due to direct sales. Factors include stock clearance to discount outlets, returns to brands under concession agreements, and other non-sale channels.

• Inventory outflows captured in this analysis may not align with consumer purchase data from credit card transactions, especially if inventory is reclaimed by brands, sold through B2B channels, or in regions outside the credit card data provider's scope.

• The analysis values inventory outflows at retail prices post-discounts. However, actual sales may incorporate additional reductions, B2B transactions may occur at lower wholesale prices, and returns to brands might not be monetarily compensated.

• This study does not provide geographical specifics. The data was sourced from the UK version of the retailer's website, which caters to multiple regions, leaving the final destination of sold items indeterminate.

Where to find the Data

The analysis was based entirely on Inventory data available on Data Boutique and updated daily. Any user, analyst, or researcher can access and download the latest data to continue this study.

Go to Data

Research Findings

Our focus was on the top 50 brands, ranked by the number of items reduced over the 9-day period. The analysis presents both the percentage change in units and their corresponding value at retail prices.

A greater decline in value compared to units suggests that higher-priced items were predominantly affected.

Overall, the findings reveal a 3% reduction in units and a 5% decrease in value among the top 50 brands within the nine-day timeframe, detailed further in the accompanying chart.

This case study exemplifies the potent application of web-scraped data for in-depth market analysis, offering valuable insights for retailers, brands, market analysts, and professionals in the luxury, fashion, and web-scraping domains.

About Data Boutique

Data Boutique is the data marketplace for web scraping. We make buying and selling data faster and safer for everyone.

Iscriviti adesso

Data Boutique is the data marketplace for web scraping. We make buying and selling data faster and safer for everyone.

Subscribe now

How Data Boutique Works #2: Volume Discounts

Large volumes of data

Offering discounts for large-size orders is an ordinary component of commerce. Sellers give up a portion of the unit margins in exchange for a bigger unit count.

This is also true for web-scraped data. Since Data Boutique's mission is to make data trades smoother, an embedded volume-related discount system is in place.

Understanding buying needs

There are multiple reasons why data buyers want different volumes of data, depending on their data usage, development stage, and horizon. The same buyer can behave differently over time or in different projects.

Retail vs. Professional data buying

Retail data buying refers to frugal, one-off data purchases. The buyer picks one dataset from the shelf and buys it, maybe for a research project or to try to better study its content. Retail buyers typically buy low volumes of data and are difficult to forecast as they are not planned in advance. They don’t represent a profit for data sellers individually but do when aggregated, even if keeping a high-risk profile.

Professional data buying refers to systematically using datasets within company processes, with recurring data updates and a known set of websites to monitor. It is a planned, regular, repeated data order, often very large, consisting of hundreds or thousands of datasets. It represents a highly stable revenue stream for data sellers, but the scale of their operations requires keeping costs as low as possible and substantial volume discounting.

Professionals start as Retail: The journey to large, wholesale purchases starts with one buy. It’s faster, cleaner, and fairer than the free trials in the rest of the data industry. Free trials are good for proprietary, unique datasets but not so much for web-scraped content.

Build new vs. Consolidate existing

When purchasing data, the more certain we are of the scope, the more discounts we can leverage. We have less leverage for discounts in a newly designed BI system where the scope is not consolidated (we might want to test which websites are more interesting to monitor). But once this scope is certain, we can increase the discount by committing to longer contracts.

This dynamic approach increases flexibility: Data buyers can try new datasets to see which has the most value and then later consolidate the scope to reduce costs.

Example:

If I want to build a discount alert system, but I’m not yet sure of which websites are more interesting to watch, or in how many countries, or if doing it weekly or daily, I could do short-term tests on each website, before confirming I want them regularly.

The volume-related pricing system variables

At Data Boutique, we apply the following elements of volume discounting, and they all cumulate to the final paid price:

Order size

Order size discount is proportional to the pure economic size (EUR value) of the order: Generically speaking, the bigger an order is (more websites, countries, higher frequency), the more discounted it is.

Frequency

The higher the refresh frequency, the higher the discount. Discounting increases progressively if the refresh order is only once, once every month, every week, twice a week, or daily.

Commonality

Buying different extractions from the same website is discounted more than buying extractions from different websites. This is related to the difficulty of web scraping. Example: Ikea Italy is similar to Ikea France, so I get a discount when buying Italy AND France, rather than buying them separately.

Duration

The longer the time horizon, the higher the discount. Data Boutique currently supports a 12-month minimum duration or a 36-month minimum duration for buyers with a long perspective, which gives a substantial opportunity to reduce data costs.

Loyalty

The more active a buyer is, measured in Recency, Monetary, and Frequency terms (RMF), the more they add value to the network; part of this value is given back to the buyer through discounts. The more active a buyer is, the more advantages there are.

Transparency is speed

Buying and selling are tough elements of human interactions. Our job is to remove all obstacles that prevent this exchange from happening smoothly.

The faster the trade happens, the less money both parties waste.

Thanks for reading so far. Subscribe to this newsletter or join our Discord channels to speak with us.

Iscriviti adesso

Data Boutique is the data marketplace for web scraping. We make buying and selling data faster and safer for everyone.

Subscribe now

How Data Boutique Works #1: Data Auctions

Data Auctions

Data Boutique is a web-scraped data marketplace where prices are set with an auction system between buyers and sellers.

Prices are public and transparent to everyone.

What is negotiated: The price for a full website scan

Sellers and buyers negotiate the price of a full website scan of a single date.

Example:

In this example, the Seller asks 6.00 EUR for a full Mr. Porter's scan.

If I wanted to purchase this dataset once, I’d pay 6.00 EUR. If I wanted to have the same dataset every week for a year, the cost would be 24 EUR in a month (4 weeks) or 312 EUR in a year (52 weeks).

When buyers and sellers negotiate, they negotiate on the 6 EUR price.

Each website is negotiated and priced independently: The Norauto dataset is priced differently than the Mr Porter dataset.

Buyers: the maximum price willing to pay (bid)

Buyers can express the highest price they are willing to pay (bid). This will be visible anonymously to all sellers.

Bids can be placed on existing websites (those already trading on Data Boutique) and websites not yet listed (where a seller is not offering it for a price yet).

Example: A dataset is has an asked price of 50 EUR, and a buyer bids for 10 EUR. All sellers get notified of the bid. If someone finds it profitable, they may accept, or continue negotiating.

If a bid (buyer’s price) is too low, it might not attract sellers, as they will not profit from it.

Factor buyers need to consider before making a bid:

• Urgency: The more urgency, the higher the price, as you need an attractive offer.

• Commitment: In the eyes of a seller, buying data only once is less interesting than buying data every week or every day for a year. This information is part of your bid, so one-off purchases might need a higher bid price than continuous (daily, weekly) ones to be considered interesting.

• Availability: If a website is not yet on Data Boutique, the bid would be the sole revenue for the seller. Being the first to ask a website might require a more generous bid to be attractive.

Sellers: The minimum price willing to take (ask)

When a dataset is published, the price shown is the asked price.

The asked price is public and applied to all buyers.

When an ask is too high, few (or no) buyers will purchase it, as they will find more convenient ways to collect it, or their case study will not be profitable anymore.

Factors sellers need to consider before setting the asked price

• Existing vs. new extractions: You are a seller. If you are already scraping a website, the additional costs to list on Data Boutique are minimal, and you can afford to ask for a lower price. Building a new extraction from scratch will cost you more. List first the ones you do, you have more margins.

• First is better than lowest: It is less efficient to engage in price downward spirals with other sellers than it is to be the only one serving that website. If you can choose, go for the one with no other sellers.

• Buyer alternatives: This is the real price cap. Buyers always have the alternatives of commissioning web scraping to someone else or web scraping internally, so the price must be considered competitive against that. If the price asked is too high, some buyers might even decide to desist and not pursue the dataset anymore.

Pre-negotiation

Buyers often are not ready to buy. They need a price quote before they can make a decision.

Sellers, on their side, don’t want to pay the extraction costs if they are unsure they will make the sale.

This is where pre-negotiations step in: They state intentions to buy and sell under specific conditions before money is spent or data is extracted.

Example: A buyer might be interested in a website, but has no idea about the cost. They bid for 10 EUR (with no additional information, buyers will bid low). A seller replies the minimum ask will be 100 EUR (they have no additional reassurance of future purchases so they ask high). The buyer may then consider raising the bid to 30 EUR, including a commitment to weekly data refreshes. This will be more attractive for sellers.

Additional options like automatic purchase (AP) - the automatic execution of the order when a seller matches the bid - or Delivery Assurance Deposit (DAD), a small confirmation fee deposited to the seller upon delivery, deducted from the final price if the buyer confirms the purchase, are also methods of lowering the risks from buyers and sellers involved in the trade.

Subscribe to get updates

Thanks for reading so far. If you like collecting, using, analyzing, squeezing, or torturing data, visit our website and subscribe to this newsletter.

Want to speak to us? Join our Discord channels.

Iscriviti adesso

Data Boutique is a data marketplace focused on web scraping. We make it simpler to match those who collect data with those who know how to use it.

Subscribe now

Web data in emergency situations: The MatchesFashion case

The Emergency:

On March 7th, Business of Fashion reported that Matches (Formerly MatchesFashion) will be shut down by its new owner, Frasers Group. Frasers bought the retailer for 52M GBP from Apax partners just two months earlier. In 2017, Apax Partners purchased MatchesFashion at a reported $1 billion USD valuation.

A relevant online retailer shuts down, and this means both trouble and opportunity for many businesses.

What may happen now, and why it’s key to use data to monitor the situation

The shutdown of MatchesFashion may have several key consequences for other online luxury retailers in the short and mid-term:

1. Market Redistribution: Competitors may experience an influx of customers seeking alternatives. This could boost sales and market share for those able to capture the displaced customer base, shifting competitive dynamics.

2. Supply Chain Impact: Current Matches inventory needs to find a way in the off-price market; there might be bargains to be made. Brands distributed by Matches might seek new partnerships and renegotiate terms.

3. Price and Discounting Strategies: Initially, there might be an increase in promotional activities as retailers attempt to attract MatchesFashion's former customers. However, in the mid-term, there could be a reassessment of discounting strategies as the market stabilizes and retailers focus on brand value and customer loyalty.

4. Investor Sentiment: The closure of MatchesFashion, coupled with Farfetch's delisting, could dampen investor enthusiasm for the online luxury retail space, especially for companies reliant on a similar multibrand strategy. This might lead to more cautious investment, both from public market investors and private equity firms, impacting the availability of capital for growth and expansion.

5. Market Perception: The broader perception of the online luxury retail market may shift, prompting companies to more clearly articulate their value propositions and differentiation strategies. There could be an increased focus on sustainability, exclusivity, and personalized customer experiences as key drivers of value in the eyes of both consumers and investors.

What data to use if you are a retailer, a brand, a supply chain operator, or an investor

In the wake of MatchesFashion's closure, three critical data points demand heightened attention from competitors, brands, supply chain operators, and investors:

• Prices: Pricing (in)discipline is the first factor that pulls consumer attention, it will be a key element for all players to monitor.

• Discounts play an even more crucial role in driving consumer traffic and sales volumes, especially in a shifting landscape where capturing MatchesFashion's former customer base becomes a strategic priority.

• Inventory levels offer insights into MatchesFashion (an their competitors) stock.

Each data point provides strategic insights that can help stakeholders adapt to market changes, identify opportunities, and make informed decisions amidst the evolving competitive dynamics following MatchesFashion's exit from the market.

Geographical coverage: fashion luxury has global coverage, and prices and discounts change significantly by region, so we suggest a global coverage (USA, Canada, UK, France, Germany, Turkey, Middle East, Singapore, China, Japan, and Australia) at least at a weekly rate.

Websites: we recommend monitoring at least the top 10 MatchesFashion competitors listed here:

1. Farfetch: A leading global platform for luxury fashion, also in troubled waters.

2. Net-a-Porter: Part of the YOOX NET-A-PORTER GROUP, owned by Richemond.

3. Mytheresa: A Munich-based luxury e-commerce platform, publicly listed.

4. SSENSE: A Canadian-based online retailer.

5. Moda Operandi: Runway pre-order runway collections, as well as in-season items.

6. Luisa Via Roma: An Italian retailer.

7. Yoox: Part of the YOOX NET-A-PORTER GROUP, Yoox is known for its extensive range of end-of-season stock.

8. Giglio.com: An Italian online fashion retailer.

9. Cettire: A global online marketplace, listed in Australia.

10. Italist: A global online marketplace. Together with Cettire, they have been capturing part of the Farfetch outflows of boutiques.

Where to get this data ASAP

Databoutique.com: It is the fastest way to access web-scraped data. It’s already there, clearly priced and used by hundreds of companies.

You could also do the collection yourself, but by the time you’re ready, the crisis will be over.

Go to Data Boutique

Subscribe and spread the word

If you like to collect, use, analyze, squeeze, or torture data, join this newsletter and spread the word. Want to speak to us? Join our Discord channels.

Iscriviti adesso

Data Boutique is a data marketplace focused on web scraping. We make it simpler to match those who collect data with those who know how to use it.

Subscribe now

How to Estimate Sales Using Inventory Web-Scraped Data

What is web-scraped inventory data

Some e-commerce websites disclose stock levels of their available products.

Some do it to offer better customer service, like IKEA, or to communicate scarcity and give a sense of urgency, like sneaker-heads Mekka StockX. Other websites sell one-item-only products, like second-hand clothing website Vestiaire Collective or Tesla’s internal used vehicle store. Others, even if not displaying the information graphically, have it in plain HTML code:

Inventory data can be accessed with web-scraping techniques and be a powerful source for competitive and investment intelligence

How to estimate sales from inventory data

Inventory is point-in-time information on the quantity of an item. Monitoring how this quantity changes gives insights into how many items have been sold.

IKEA example: the quantity of the metal bowl BLANDA BLANK (see picture) in a store of Milan is 2.439 on the 26th of February 2024.

The next day, this value is 2.418, which is 21 units lower

We can assume that in the time between our observations, in the Milan store of IKEA, 21 metal bowls BLANDA BLANK were sold at 6.95 EUR each, for a gross value of 146 EUR.

Suppose we repeat this for every item in the store, for every IKEA store worldwide, for every day of the year: We have a real-time hyper-detailed revenue estimate for IKEA.

Assumptions

As inventory changes for many reasons, not only sales, some assumptions must be made to interpret data correctly. The key points we need to consider are:

1. Frequency: The more frequent observations, the better. Daily (or intraday) capture is desirable, but that can drive up costs. I worked on a project on a specific luxury website and found that weekly snapshots were 98% as reliable as 20-minute snapshots (at a 99.7% lower cost!). Industry-specific considerations apply: Fast-moving consumer goods are different than luxury bags.

2. Number of stock points: We need to understand (or at least estimate) how many stock points (warehouses) serve a specific website. If a website is served by a central warehouse for all of Europe, we need only one extraction (i.e., France) that represents all others. But if, like the IKEA case, we have one stock point for every store.

3. Type of stock points: Some websites disclose online and offline warehouses, so we know we have visibility for online and offline sales; sometimes, instead, we have only visibility on the online warehouses and zero visibility of what happens offline. This is a material difference when we want to understand what kind of revenue we are trying to estimate.

4. Restocking policies: Having a good awareness of the restocking policies of the industry is key for data interpretation. The more we know, the more our estimate will be correct. Are goods restocked once or twice every season (like in luxury goods) or at intraday frequency? This has a direct impact on our choice of frequency and data interpretation.

5. Inventory transfers: This is similar to restocking, except the goods move between one warehouse and another. We might notice -1.000 units less in IKEA store A and two days later +1.000 units more of the same object in IKEA store B.

6. “Dead stock” management (unsold inventory): What happens when a product reaches the end of its shelf life? Monitoring this is key for some important ESG-related metrics, like unsold inventory.

7. Canceled/Returned orders inventory management: What happens when an order is canceled? How does it appear on inventory? We found this behavior can be a source of a lot of noise in the data if interpreted incorrectly.

8. Noise: Data is only as good as the system that generates it. Delays, errors, lags, misalignments, coding errors, manual data entry errors. Be prepared to have a model that keeps that into account.

Differences with other data sources

Before building a dataset from web scraping, it is worth understanding how this compares to alternative datasets in the market.

One popular dataset among professional investors is credit (and debit) card transaction data. This data contains financial transactions of specific user groups.

While credit card data is more reliable in terms of absolute numbers, has a consistent history, and will be more stable over time, web-scraped data can offer deeper insights, and given the cost difference of collecting the latter, in our previous experiences, it is a valid integration, if not an alternative to the former.

Data Quality

• Scope: Credit card data works the same on every website, regardless of whether they disclose inventory. Web-scraped data requires the website to disclose it.

• Reliability: Credit card data are real transactions, while inventory-based calculations are estimates and heavily linked to the assumptions made.

• Completeness: Credit card data are usually limited by card or bank’s brand, often related to the demographics of the user base. On the other hand, web-scraped data can show all transactions but is limited to the available warehouses.

• Cross-retailer consistency: Credit card data have the same feed regardless of the website. Web scraping depends on the website and what kind of information is disclosed.

• Time consistency: Credit card data is a consistent feed (doesn’t change structure) over time. On the contrary, web scraping is exposed to website changes and anti-block technology. If a day’s collection is missing, it is lost.

• Historical data: Credit card data go back in time for years. Web scraping is only as old as the day you start doing it. This is a differentiation element for those who start collecting it.

Use Cases

• Transactions by demographics: Credit card data can be enriched with (anonymized) consumer data, giving insights into demographics. Web scraping has no visibility on the demand side and only makes assumptions based on the change in supply.

• Transactions by geography: Credit card data can be enriched with location data (for physical transactions) or user (anonymized) data for online transactions. Web scraping can go only as far as location information of the underlying stock point.

• Transactions by product: Credit card data has almost no data on the products within an order; they see the transaction as a whole. Web scraping on the contrary has visibility on products, category, composition, description, price and discounts.

• Average Order Value (AOV): Credit card data show the order as a whole, giving insights on average order value (AOV), whereas web-scraped cannot see if a product was purchased alone or in a group with others.

• Margins and discounts: Credit card data only shows the price paid. Webs scraped data have visibility on the discount offered and can be very precious for estimates or models on margins.

• Inventory Turnover Ration (ITR): Web scraped data gives the metrics for all inventory KPIs, including ITR and sales velocity.

• Working Capital Invested in Inventory: Once embedded in the model, the cost (or value) for items stored in a warehouse, we can have a real-time estimate for working capital invested in it.

• Unsold inventory (ESG related): Unsold inventory is a huge topic in many industries, with big implications in ESG. Web scraped data can measure this, whereas credit card data lacks insight.

• Stock Breaks: When an item is Out of Stock, it ceases making sales. Web scraped data can identify this.

ROI - Return on Investment of Data

• Time-to-Market (TTM): The time it takes from when you get the data until you can start actually using it is in favor of credit card data because it is ready to use. Web scraped data has an initial time investment required to build history.

• Cost of data: The big downside of credit card data is its cost. Web scraping, especially when accessed via marketplaces like Data Boutique, has data costs of orders of magnitude lower.

• Differentiation: The big point. Credit card data are extremely popular and spread in the investment community, offering little differentiation. Web scraping, given the investment in time it takes to build history and the fact that every website is a separate investment, is still underused.

Conclusions

The reason for this deep-dive into inventory data is that we are releasing the Ecommerce Inventory Schema on Data Boutique, where this kind of data can be accessed faster and easier than web scraping first-hand.

Accessing web-scraped data on inventory can be a differentiating asset for financial analysts, market researchers, and consumer goods competitive intel.

We hope this piece helps contextualize the power and opportunity of web-scraped inventory data for a future set of applications that use it.

If you’re interested in collecting inventory data (and potentially selling it on Data Boutique), we recommend these readings:

• The Web Scraping Club THE LAB #27

• The Web Scraping Club THE LAB #28

About the Project

Data Boutique aims to increase web data adoption by creating a win-win environment for data sellers and buyers. If you operate in data, join our community.

Visit Data Boutique

Thanks for reading and helping our community grow.

Iscriviti adesso

Data Boutique is a data marketplace focused on web scraping. We make it simpler to match those who collect data with those who know how to use it.

Subscribe now

Tracking Farfetch [Part 2]: Kering's Exit

Last week, we wrote about how to use data to monitor Farfetch. This is a follow-up post to see data in action and look closely at the rumors of Kering’s brands exiting Farfetch (new rumors on Farfetch offloading NGG brands also just came out).

Last week’s post:

This is what the situation looked like on the 26th of February, compared to 100 days before:

How to read data:

• SKU count: The number of shoppable items on Farfetch aggregated by brand. A negative change means there are fewer products today of that brand than there were 100 days ago

• Inventory: Farfetch does not hold inventory, but it displays the inventory that the merchants (boutiques, brands, and retailers) decide to share on Farfetch. When inventory goes down, merchants share less on Farfetch today than 100 days ago. When inventory increases, brands (or merchants) fill their warehouses and make deeper inventory available on the channel.

• Countries: Farfetch allows consumers to “fetch” items from boutiques far away (hence “far-fetch”), so countries do not mean that the inventory is in that country. It means the merchant makes the inventory shoppable in that country. A brand might want to restrict the availability of its products in certain regions while still leaving them available in others.

Business interpretation of the chart:

• Kering’s brand Gucci is reducing its presence on Farfetch, in line with rumors. It will need to replace sales generated by Farfetch with the direct-to-consumer website.

• Other Kering brands are also decreasing, but not (yet) as much. This might be impacted by the share of retail vs wholesale of other brands.

• Michael Kors is also lowering exposure, but other brands are more or less stable, if not increasing. Burberry has grown its product availability in China.

• With the decrease of Gucci's presence, Farfetch is losing a major anchor brand, which, on the one hand, leaves room for others and, on the other hand, decreases the attractivity of the platform.

How to build this report at home

This is a simple yet effective use of data. If you want to build your own, monitor other brands, or track closely the situation as it evolves, here’s what you can do: Ask the right questions.

• Should you do it? If you or one of your clients work for - or have money invested in - Farfetch/competitors of Farfetch/brands selling on Farfetch, you probably should. If not, then reading the news will just suffice.

• What to look for? My suggestion is to K.I.S.S. (Keep It Simple, Stupid). We want to see who is jumping ship and who’s staying in. A clear purpose is the best way to extract real value from it.

• How much should you invest in this (ROI)? Depends on how much skin in the game you have, but spending as little as possible is always good advice: If Farfetch goes belly up, you won’t need this tool long after. I would recommend spending a maximum of 200/300 EUR per month in data and max 2 hours per week for a member of your team to run this, and you can have your reports in your inbox every Monday by noon (you could automate it way more, according to your needs, but we stick to super basics here).

• What to monitor? Stick to the bare minimum: USA, UK, China on a weekly or monthly basis, SKU count, and inventory. That should do the job.

• What tools to use? I wish to tell you Excel would be enough, but it’s quite a bit of data and might become bulky to handle. If you have someone on your team who can handle Excel like a pro, give it a try. MS Access, PowerBI, Tableau, or any of those will do the trick even better.

• How to do it? It’s easier than it looks:

  • Pick the data SKU and inventory

  • Download the files every week/month (manually or via API)

  • Open the files with Excel / MS Access / PowerBI / Tableau or whatever you have (automate with API if you know how)

  • Create a pivot table with summary statistics: Summarize the inventory by brand and country (sum of units) and count the rows (SKU count).

  • Save it on a summary Excel, which has only summary data, and make it pretty so you can send it via email.

  • Delete the files you downloaded: It saves space. Don’t worry, you can always download them again, once purchased they won’t disappear.

Go to data

I hope this hands-on post was helpful.

About the Project

Data Boutique aims to increase web data adoption by creating a win-win environment for data sellers and buyers. Join our community. It’s free. More can be found on our Discord channels.

Visit Data Boutique

Thanks for reading and helping our community grow.

Iscriviti adesso

Data Boutique is a data marketplace focused on web scraping. We make it simpler to match those who collect data with those who know how to use it.

Subscribe now

Tracking How Bad The Situation At Farfetch Really Is

We’ll walk through a highly relevant case study for web data in to consumer goods

The 4Bn USD problem

4.4bn USD worth of fashion items were sold through Farfetch in 2023, but many in the industry fear 2024 for the fashion marketplace might mean the end.

Over the past six months, Farfetch faced significant challenges, including financial difficulties and strategic missteps that led to its delisting from the NYSE. Following these events, Neiman Marcus has decided against pursuing technological integration with Farfetch, and Kering announced its intention to remove its brands, including Saint Laurent, Gucci, Bottega Veneta, and Balenciaga, from the platform, which historically has been significant for Farfetch.

If Farfetch’s situation were to worsen, several groups of stakeholders would face significant risks: Brands with direct concessions on the platform could lose a vital sales channel, while boutiques might suffer due to their increasing dependency on the online retailer. Investors, including the most recent Coupang, employees, and consumers, also stand to be affected.

The speed of information

Given the significant interests at stake (4.4Bn USD in sales), it is crucial to understand not only where this is going to end but also how fast this is happening. So that those impacted the most can have the best chance to make business-saving decisions.

Why use web data

Short answer: Because it’s the best, sometimes the only, source of information many have.

Long answer: Financial results are slow to be reported (Farfetch delayed announcing quarterly results before the crisis), insider information is not granted and may be partial, biased, or misleading.

Credit card transactional data and email receipts are also a great way to track this.. if you work in a hedge fund with deep pockets for data, data scientists, and data infrastructure - but this is not the case for the rest of the class.

What stands out in the open, for everyone to see, is how the website behaves towards its customers. This is the ultimate touchpoint between the platform and the source of their revenue: customers.

What data

Traffic data

This is not strictly web data (at Data Boutique, we are often asked if they can be scraped… no, they can’t), so I’ll put it in. It is extremely relevant and also accessible. Similarweb is a great place to start.

Website traffic gives a timely measurement of the footfall of the website, which is critical to have a pulse on how the demand side of the business is responding.

Web scraped data

Web scraped data tells us how the supply side of the business is acting. Let’s see what web data can tell us and where to find it:

Brand presence

How fast is Kering pulling out from Farfetch? In which geographies do they start first? Where will they leave next? What other fashion groups or brands are and will be next?

The basic e-commerce data on Farfetch (schema E0001) for the UK, USA, Europe, China, and Japan are the perfect (and affordable) place to get this info.

Discounts

When things go south, in the wild e-commerce world, discounting and promotion are often the easiest levers to pull, so being on the watch for when and where this lever will be pulled is key. The problem with discounts also affects Farfetch's competitors since once someone starts, the others are forced to follow, in a domino effect.

Get the same data mentioned above; it will also cover this.

Price discipline

Discounting is not the only price hack common in fashion. Going rogue with arbitrages, especially in countries far away and difficult to track, can be challenging. To track this, we will need, in addition to the E0001 datasets mentioned above, the E-ADD-CODE-0001 dataset, which provides the original SKU numbers (on Farfetch called Brand ID) that allow cross-platform product matching.

Concessions

Who are Farfetch merchants? Will they stay, or will they go? When will they jump ship? Although Farfetch does not disclose merchants’ names as clearly as other players like Yoox or Zalando do, it is still visible and accessible from the same E0001 datasets mentioned for discounts and brand presence (if you have any trouble making it work, feel free to reach out).

The same datasets will serve this as well.

Inventory

Inventory depth is also very relevant in retail analysis. The newly released dataset E-INVENTORY for Farfetch covers this topic. Although Farfetch does not own inventory, it shows the depth of stock brands in concession, and merchants are letting Farfetch see. Super hot dataset, used for GMV estimates as well.

Conclusions

Whether you work in a large corp with great data analysis capabilities or a small venture, access to web data can offer valuable, timely insights on how this farfetched crisis is evolving.

About the Project

Data Boutique aims to increase web data adoption by creating a win-win environment for data sellers and buyers. Join our community. It’s free. More can be found on our Discord channels.

Visit Data Boutique

Thanks for reading and helping our community grow.

Iscriviti adesso

Data Boutique is a data marketplace focused on web scraping. We make it simpler to match those who collect data with those who know how to use it.

Subscribe now

The hidden costs of asking for quotations in the data market

Asking a vendor for a quotation is a back-and-forth hidden-costs-generating process with many inefficiencies.

Building complex solutions involves asking vendors for packages that are not priced yet

Asking for a quotation in B2B projects is often unavoidable.

Whenever building new or tailor-made solutions, we need to buy stuff in packages that have not yet been priced, produced, or assembled the way we need them, and data is a frequent case.

Often, quotations refer to components of larger projects, which also need to provide an estimate.

Examples: A firm bidding for a tender and asking a vendor for data collection costs; the IT department researching the market after being asked by Marketing for a BI solution; a startup evaluating the economics of different technology stacks to adopt.

It all revolves around trying to understand the cost of something that has not (yet) a price tag on it.

Quotations are a cost for sellers…

The process represents a cost for the sellers: The time spent on poorly written briefs and designing packages to address a request they have little information on ultimately builds up their operating costs.

Once the quotation is submitted, the waiting for the feedback of the entire decision chain begins, made of reminders, check-in emails, and rescheduled calls.

Then come scope revisions, alternative scenarios requests, and bulk discounts. Despite this, as often happens, the project might not even be won.

This is part of the CAC (Customer Acquisition Costs), eventually paid by buyers, as it will be factored into the price.

… and for buyers too

Inefficiencies also hit buyers: Time and resources spent writing down the requests, reading and scoring responses, and asking for revisions and alternative scenarios.

A long quotation-asking process means it takes longer to buy stuff, which means either we buy less frequently (and build fewer solutions) or hire more staff to do it. Both cases kill ROI.

From whatever perspective you look at this, it’s a dollar-killing inefficiency.

Unbearable for small-size trades

While this is acceptable in illiquid markets (as illiquid as a once-every-five-year-million-dollar deal), it becomes an unbearable friction for small to mid-size trades (hundreds or a few thousand dollars multiple times a year).

Web scraping data - the business we’re in - falls in the latter case.

How we solved it: Schemas and Bundles

Since our mission is to make access to data faster, cheaper, and safer, solving this money-burning problem was a priority.

What we did was the following:

1. Standardizing the building blocks

2. Empowering buyers to play before they buy

1. The building blocks (Schemas)

Simplifying starts from the foundations: To make the process smoother, we must identify simple, modular building blocks for our data use cases. The tricky part is to simultaneously be simple and valuable.

Standard schemas provided the answer: Buyer-oriented data structures that serve “atomic” data needs. More complex solutions can be delivered just by assembling more schemas together.

Working in web scraped data made this part easier: Websites usually come in standardizable categories, such as e-commerce, classifieds, store locators, travel and booking, and so on.

When you buy an “E0001 Schema” you know what you get, regardless of the website. This took so much complexity from the equation and made life easier for everyone: Buyers and sellers.

2. Play before you buy (Private Bundles)

Once we had modularity, creating complex configurations was its natural evolution:

Data requests like “a dataset to monitor daily prices and promotions of H&M products in France” or “a dataset to measure inflation in Germany, the UK, the USA, Canada, China, and Japan for all Kering brands” is an exercise of assembly.

Inspired by tools like the AWS Pricing Calculator, we created Private Bundles, a feature where users can simulate their data packages.

With Private Bundles, users can change the scope and frequency, assemble atomic blocks as they need, save their selections, and share them with their team, with no commitment to buy.

Private Bundles are a safe environment where users can experiment different ideas before committing to buy

A Private Bundle enables to:

• Start planning on initial ideas and saving them for later, defining data specs as you go and keeping draft versions as more information comes in;

• Have a real-time cost projection, and see how this changes when you choose different data types, vendors, data refresh frequency, and content;

• Factor in volume discounts, as they are automatically inserted in the calculation;

• Contribute with other team members to its definition, sharing the final draft with the decision-makers.

Improve the speed of thinking data projects

The speed of project design improves significantly as a buyer can formulate, correct, and rephrase their project brief directly on the platform, obtaining a real-time cost quotation.

To save even more time, you can start from a Public Bundle, clone it, and work from there: Example of a public bundle.

Often, buyers need to evaluate multiple scenarios, inclusive of data costs, before they can make a decision.

We make this process faster.

Go to Data Boutique

About the Project

Data Boutique aims to increase web data adoption by creating a win-win environment for data sellers and buyers. Join our community, it’s free. More can be found on our Discord channels.

Visit Data Boutique

Thanks for reading and helping our community grow.

Iscriviti adesso

Data Boutique is a data marketplace focused on web scraping. We bring together those who collect data with those who know how to use it.

Subscribe now

What’s New on Data Boutique

PREFACE TO THIS EDITION

This edition opens a new series of posts on relevant content shared on the web-scraped data exchange during the past month.

Table of Contents

1.For data buyers

• Case Study: How to match products on Farfetch

• New Datasets: Cosmetics on Sephora

• The Cost of Data: Asked price stats by schema

2.For data sellers

• Feature: Your Profile Pages on Data Boutique

• Tips: Automating Upload and Validations on the Data Boutique S3 Bucket

1. For data buyers

Case Study: How to match products on Farfetch

Farfetch is arguably the most relevant marketplace in the online fashion and luxury space, and price and promotion monitoring on this website is highly relevant.

In price monitoring, matching products across retailers is required to measure potential price misalignments between platforms. Multibrand retailers typically use internal product codes, as the apparel industry rarely uses standard codes, such as EAN or GTIN, making the job harder.

Farfetch exposes an additional code, the “ID Brand” code, the closest thing we have to a unique product identifier (SKU). We have a dedicated schema to capture any additional code e-commerce websites may have: The E-ADD-CODE-0001 schema.

Why use the ID Brand code

It makes price comparison with other websites easy, especially official brand websites. You can use it to match Farfetch and other websites (that you can find on Data Boutique), or with additional product list files you may have.

How to use the E-ADD-CODE-0001 schema

The E-ADD-CODE-0001 schema is essentially a lookup/cross-reference table: It contains the list of product codes used in other schemas, such as E0001, and associates it with the “ID Brand” code.

In this example, it associates the product_code 20267286 with the ID brand MA10554G0HMJ5.

The importance of having a separate schema

Why a separate schema? Two reasons:

1. Cost: ID Brand codes are visible in Product Detail Pages (PDP), which are more costly to scrape. By keeping it separate, we keep the price information in a cost-efficient schema. The E-ADD-CODE-0001 schema can be accessed in combination with an E0001 schema (plain data on prices), the first just once every month or less, the latter more frequently, dropping the cost factor roughly by an order of magnitude.

2. Easier to find, easier to request: With a separate schema it’s easier to spot when a website has this information available, with no need to investigate the sample file. A buyer will have immediate visibility, and when missing, they can request an E-ADD-CODE-0001 schema for that website; no further specification is needed. Data discovery gets a lot smoother this way.

   Go to This Dataset

New Datasets: Cosmetics on Sephora

The cosmetics industry is a hot sector for web scraping. The first dataset appearing on Data Boutique refers to the retailer Sephora, the leading beauty and cosmetics retailer owned by LVMH. Sephora offers beauty, skincare, and fragrance products from various luxury brands.

The dataset currently lists product prices for France and Poland and is offered by WebDataWatch, one of the most reliable data sellers on Data Boutique.

Go to This Dataset

The Cost of Data: Asked price stats by schema

The data price on Data Boutique is set by buyers and sellers simultaneously. The factors that influence price are:

• Cost of extraction

• Demand volume and stability

• Reputation of seller

Since the cost of extraction varies significantly based on the website and the depth of scraping required, it is helpful to look at price statistics by schema.

Here is a snapshot of what prices look like today on Data Boutique.

2. For data sellers

Feature: Your Profile Pages on Data Boutique

Starting this month, sellers can have a profile page hosted on Data Boutique, where all datasets offered are grouped together.

We want data providers to be able to showcase their work and attract leads. This can translate into job opportunities inside and outside of Data Boutique.

Profile Pages have the purpose of:

• Establish the seller’s brand

• Showcase their work

• Enable lead generation

• Have a space to share relevant news, marketing material, and updates.

How to activate the profile page

Since there is a lot of activity on the platform, these indications may change in the future (so just be aware of this if you are reading this post in the future):

Once logged-in (sign-up is free), from the top menu bar pick “Selling” and then “My Company Profile”.

From here you can handle your public messages, you can edit the information displayed on the Profile Page, and most importantly you can change the settings to:

• Make your Profile Page public

• Accept direct messages from any Data Boutique user

You are free to turn on/off any of these options.

Tips: Automating Upload and Validations on the Data Boutique S3 Bucket

When selling data on Data Boutique, data providers need to upload their files on an S3 bucket. An automatic validator tests the content and returns an approved/refused response.

A huge shoutout goes to Riccardo Lunardi, a seller from Italy, who shared some Python code to automate this process. Riccardo created a piece of code (class) that can update files to the contacts and check for their approval (check the article to know how). Thanks a lot, Riccardo🤗.

Go to Guide

About the Project

Data Boutique aims to increase web data adoption, by creating a win-win environment for data sellers and buyers. Join our community (its free!) to be part of it. More can be found on our Discord channels.

Visit Data Boutique

That was it for this month. Thanks for reading and helping our community grow.

Iscriviti adesso

Data Boutique is a web-scraped data marketplace. Unlike generic-purpose ones, this addresses specific, well-defined needs in a transparent, interest-aligned space.

Subscribe now

Generic data marketplaces are broken

Four years have passed since Amazon launched AWS Data Exchange, and not AWS nor any other player proved to be that accelerating agent in the growth of the segment everyone hoped for.

As of today, the coverage of datasets is minimal:

• AWS Data Exchange lists 4.109 data products

• Snowflake Marketplace lists 2.427 data products

• Databricks Marketplace has 143 providers (which include solutions providers as well, not only data providers).

Far off a market that’s supposed to be “the new oil”.

Marketplaces fail to capture a significant part of the exchanges: According to Grand View Research, the Data Marketplace global market size in 2023 was 1.2 billion USD: The Alternative Data Market - a sub-niche of data - was 6 times bigger (7.2 billion USD), and Web Scraping 5 times that much (6 billion USD).

Cross-referring to this research, marketplaces capture less than 2% of the exchanged volumes.

.

What’s the problem here?

Large and well-funded corporations have tried to address this, so why hasn’t it worked out yet?

As my all-time most-quoted essay reads, the role of marketplaces is to lower transaction costs.

Marketplaces are just as powerful as the value they bring to the table. And that value has not been enough yet.

The marketplace is pretextual to platforms: interests are misaligned

Data is platform-agnostic. But those who built major data marketplaces today are not: Their intent is to push their technologies, not data commerce.

In fact:

• AWS Data Exchange wants you to use the AWS stack

• Snowflake Data Marketplace works as long as you are a Snowflake user

• Databricks, Tableau, Qlik, SAP, Informatica, and others all are in support of a specific technology

They get dollars when a user uses the platform, but it makes no difference if a buyer purchases external data products or uses their own. They have zero incentive to optimize for a sale. They won’t invest in it more than the bare minimum, and that explains the following point.

Search is not the problem.

Generic data marketplaces are just a collection of providers with little more than search features.

But if we were to break down the transaction costs for data (search, evaluation, negotiation, price, and ingestion), the search component turns out to be just a minor element.

On top of a 5k USD- often 10X this much - data contract, as a buyer you’d still need to enter the website of the vendor, call their sales team, get a quote, negotiate a sort of trial, and onboard the data - each time with a different vendor.

Finding the vendor is hardly the hard part.

There is simply too much work to be done off-platform, talking and negotiating with the vendor. Actually, there is so much work that the role played by the marketplace (search) is not that different than those of search engines with SEO.

Understanding the price, understanding what data I am buying, and comparing that data with alternatives is the problem.

What is needed to change

Marketplaces are trading all data like a gigantic blob inside a chaotic bazaar where sellers scream their best offers. This doesn’t instill trust.

1. Align interests

Marketplaces that have skin in the game will work for the benefit of buyers and sellers. When interests are aligned, all parties involved will profit only when the exchanges on the platform take off.

A “take rate” business model is the one that provides the best alignment of interests with everyone, buyers included. When the platform has stakes in a deal, you can be sure they’ll optimize for it.

2. Make pricing clear

The bull must be taken by the horns: Price is the most critical element in B2B negotiations. Today’s data market is a closed-door negotiation place, making the market illiquid.

This is the contrary of a marketplace, where a more extensive, prosperous economy flourishes because it is liquid, and goods are bought and sold rapidly.

The price for an exact service needs to be visible and comparable. “Free plan available” or “Subscription options available “ won’t work.

Price needs to be a transparent, outspoken element, allowing for comparison among like-for-like products.

3. Address the transaction pain points

A platform needs to drive faster and more frequent (liquid) transactions. To achieve this, we believe a vertical specialization is needed. We picked web scraping because it’s a market we understand very well, and we optimized Data Boutique for it.

The advantage of being vertically specialized is that it allows one to enter the specifics of the purchase journey, from discovery to evaluation, to purchase, to fruition, to after-purchase support.

Let’s take web scraping: Data Quality is a big headache for the buyer. Since we only do this kind of data, we can have a given set of quality checks before the data product is listed.

This way prices become comparable, we can have a like-for-like benchmark, and the purchase decision can be made faster.

This is what buyers and sellers look for and the ultimate end goal of a marketplace: More sales.

Subscribe now

Annex - screenshots I took so you know I didn’t make this up

Subscribe now

About the Project

Data Boutique is a community for sustainable, ethical, high-quality web data exchanges. You can browse the current catalog and add your request if a website is not listed. Saving datasets to your interest list will allow sellers to correctly size the demand for datasets and onboard the platform.

More on this project can be found on our Discord channels.

Thanks for reading and sharing this.

Iscriviti adesso

Data Boutique is a web-scraped data marketplace: Because the smartest way to get this data is to ask those who already collect it.

Subscribe now

The Impact of ChatGPT on the Web Scraping Industry

November 2023 marks one year since the public release of ChatGPT. The model is trained on large-scale web-scraped data and can now access the internet in real-time.

Given the latest trends, what are the impacts on this data economy?

tl;dr: ChatGPT doesn’t get rid of the need for web data, but it speeds up how you get it, and how you get insights out of it.

Does ChatGPT solve the web data need?

Do we still need to extract data from the web when we can ask ChatGPT?

Unfortunately, while ChatGPT has access to a lot of data, it doesn’t have access to all the data you need, not in the format you need, and wouldn’t deliver it in the way you need for a large-scale web-scraping use case.

Think of applications like revenue optimization for hotels: Features like browse with Bing or the Kayak plugin do offer some relief to this, but all efforts hit a wall when trying to build something intrinsically more robust than search engine results.

When researching the fashion industry, the Shein discovery ChatGPT plugin offers some basic level results, but its primary purpose is to make you shop on Shein, not help craft your brand strategy.

When building your own LLM, or a market intel SaaS, you need data extraction from the web - and ChatGPT’s help can go a long way:

1. It can help with the extraction problem (data supply), by making it faster

2. It can help us once we have it (data demand) to make sense of it

Supplying the data

Without entering details, systematically extracting data from websites burns resources in two ways:

1. Time - writing the code, executing it, and checking the results for completeness absorbs FTEs pretty heavily, and it’s getting more and more complicated than it was just 5 years ago. The most innovative applications use generative AI to save time by making the writing of the code faster, assessing quality issues on the retrieved data for easier code maintenance, and reducing the time to market of the software to keep it in high reliability. A private version of the model (like the ChatGPT enterprise plan) can help build proprietary assets.

2. Money - websites often have costly anti-scrape measures that require costly anti-anti-scrape countermeasures. Little can be achieved on this side of the problem with generative AI. It can teach you how to fish, but it still remains a damn expensive sport.

Using the data (demand)

The use of AI in business intelligence has long been a feature that was explored by many platforms, yet with not exceptionally satisfying results.

Today ChatGPT Advanced Data Analysis feature (ex code interpreter) and plugins like Noteable allow you to get a lot done on this side:

• You can upload your data (the web data you just collected or has someone collect for you) or connect with your database (maybe a better choice since limitations on upload barely hit the definition of “big data”)

• Interact directly with data through textual prompts, which is very useful, especially on first-time onboarding of datasets and quality assurance and ingestion process design.

The current state of these technologies works better in interaction with established technology stacks of Data Lakes/Data Warehousing or Business Intelligence tools like PowerBI or Tableau and will likely serve as co-pilots rather than replace them.

The main advantages we have seen are:

1. Boost in analytics capacity for one-off analysis, like an M&A due diligence - traditionally incompatible with the timeline of building a custom BI project - allowing to process larger amounts of data way faster

2. Faster data onboarding on long-term projects, which, in addition to private versions of the model like ChatGPT Enterprise, allows the use of internal data and building proprietary models.

Using the data to train LLMs (more demand)

Finally, a word on the most data-hungry applications in town: LLMs. Maybe the strongest effect we have seen in web scraping was the rise of new LLMs inspired by ChatGPT.

Training models is a data-devouring activity, which for many models means scraping, scraping, and again scraping. Once trained, the models still need data to process, be refined, and (again) trained.

Since the launch of our marketplace, the share of AI applications in our user base has kept growing: From early-stage startups to established SaaS, new requests for high-frequency, high-volume, billion rows pricing/product/URL data feeds have only increased.

The need for data is so intense that they would be spending more time fixing scrapers than working on the AI, had they decided not to source the data elsewhere - as one user told me, “We are an AI company, not a web scraping company!”.

Conclusions

We have seen a growth in demand and supply, triggered as well by the advent of large language models and by the growth in stakes in the digital economy.

But web scraping is often kept under cover (“The first rule of web scraping is: you do not talk about web scraping” is the opening line of the webscraping subreddit), and it is still living its wild-west moment, where everyone is out on their own, making it difficult to measure and compare.

A lot has been done on the software side, but too little on its fuel (data). Join our project if you want to make web scraping more efficient.

Subscribe now

About the Project

Data Boutique is a community for sustainable, ethical, high-quality web data exchanges. You can browse the current catalog and add your request if a website is not listed. Saving datasets to your interest list will allow sellers to correctly size the demand for datasets and onboard the platform.

More on this project can be found on our Discord channels.

Thanks for reading and sharing this.

Iscriviti adesso

Data Boutique is a web-scraped data marketplace. If you’re looking for web data, there is a high chance someone is already collecting it. Data Boutique makes it easy and safe to buy data from them.

Subscribe now

The Hard Economics of Selling Data

As already seen, it would make enormous sense to buy pre-scraped data instead of building a new code from scratch. Yet, many efforts that have been made in the past to sell datasets didn’t catch up.

Why is that? Why do companies hire or commission external consultants to scrape rather than search for pre-scraped data? Why is it build preferred to buy?

Selling data independently can be hard, as unit economics pull against it. But things look differently when we understand the market.

Web scraping is a commodity

We talked about this here: Since it would be feasible for me to hire someone to code a scrape for a fair share of websites, web scraping can be considered a commodity.

In other words, buyers have alternatives, and pricing datasets right might not be that easy.

The price trap

Price options are constrained, in fact. We can’t ask for too much, as few would buy (they have too many alternatives), but we can’t drop the price too much to sell it to more people, because it would be anti-economical (it costs too much for us to reach that customer, and cheap products might cannibalize existing ones).

Not Worth Buying: The Cost of Alternatives

Some data are unique, and others are a commodity. The commodity, by definition, has trouble in being priced too high. Every buyer will in fact consider the cost of alternatives.

This cost varies from buyer to buyer, but we can consider it the lowest of the following:

• Cost and time of learning web scraping internally

• Cost and time of hiring someone who can web scrape

• Cost and time of commissioning the web scraping to a third party

• Loss/ damage for not pursuing web scraping at all

This caps the maximum price a data seller can ask, as there is so little liquidity (so few buyers) at high prices.

Not worth Selling: Customer Acquisition Cost (CAC)

Since data can’t be sold for a lot, trying the VOLUME strategy, (low prices for many) seems the natural alternative.

However, sellers are not incentivized to pursue it, as they’d be operating at a loss due to high Customer Acquisition Costs (CAC): The distance a buyer will have to travel to find a pre-scraped dataset is longer than the distance to find a web-scraping expert who can do the work.

A stalemate. Where everybody loses, as the information market, which could trigger so many use cases, can’t find its way out.

How Marketplaces change this

The good news is, if there wasn’t a way out of this, humanity would still be stuck with in-house agriculture, as this applies to any generally available technology.

The model that humanity found to solve this stalemate is the marketplace. The - once physical now digital - place where buyers go and meet multiple sellers to do more, diverse purchases.

Trading goods and services in a structured marketplace has two major effects that allow the market to get out of the price trap and access the benefits of pre-scraped data:

1. Lowering the transaction cost

As stated by in his brilliant essay on marketplaces, marketplaces' purpose is to lower the buyer's effort (transaction cost).

When considering web scraped data, the buyer’s effort can be broken down into:

1. Search: The effort it takes to find a reliable data provider. Among all the features a data marketplace covers, this is the bare minimum: Providing a decent UI to find data providers.

2. Auditing: The effort (time and cost) it takes to test the result and ensure we can use it. Since web data has legal and quality implications of its own, a dedicated marketplace for web data makes sense. We have been working a lot on this point at Data Boutique, as auditing (quality and legitimacy) is often a deal-breaker.

3. Negotiation: The effort it takes to understand and negotiate the conditions for the sale. Another big one. When a buyer can seamlessly test, buy, and refresh data from multiple vendors under the same roof of Terms and Conditions, it’s like removing roadblocks with a bulldozer. You realize an industry has a price issue when prices are never displayed upfront. We have always pursued transparency, as it only speeds up transactions at the end of the day.

4. Price paid: The actual price paid. We don’t believe the price is too high for web data. We believe the pricing model, inherited from consulting, is broken. We firmly believe consumption-based pricing is the best way to align buyer’s and seller’s interests.

5. Execution: The effort it takes to actually ingest the data and make something useful out of it. Almost every data marketplace handles this, mainly because the interest of captive marketplaces is to use the other services, not the marketplace itself (Snowflake, Databricks, Tableau, Qlik, etc.)

When properly orchestrated, the buyer can have many advantages for transacting over a marketplace, and the transaction cost decreases (not necessarily the price).

2. Lowering the CAC for sellers

Sellers have upsides here, too. Since a buyer is already there, the cost to reach out to them is way lower than fetching them from far.

This is more true in some markets than others. The goods or services need to be a commodity (many sellers), with a fragmented buy-side.

The effect on the CAC is very relevant. Sellers can compete on value and price, attracting more buyers and lowering CAC.

The battle for leaner prices also has huge advantages for sellers: The reduction in price is more than counterbalanced by the growth in the Total Addressable Market (TAM) created.

We will discuss more on the benefits of marketplaces in future posts, as there are many facets to understand that make this model specifically adapt to solve the data distribution problem.

About the Project

That was it for this week!

Data Boutique is a community for sustainable, ethical, high-quality web data exchanges. You can browse the current catalog and add your request if a website is not listed. Saving datasets to your interest list will allow sellers to correctly size the demand for datasets and onboard the platform.

More on this project can be found on our Discord channels.

Thanks for reading and sharing this.

Iscriviti adesso

Data Boutique is a web-scraped data marketplace. If you’re looking for web data, there is a high chance someone is already collecting it. Data Boutique makes it easy and safe to buy data from them.

What is the legal context for Web Scraping?

Me and my long-time business partner Pierluigi Vinciguerra have been in the web scraped-data business for quite a while now. We have been collecting and selling data to some of the largest companies in Asia, Europe, the UK, and the USA, from real estate, hedge funds, consumer electronics, fashion, and luxury.

We have always paid great attention to its legality, and if you are a web data user or are involved in web scraping, you should care too.

While web scraping is as old as the Internet itself, it has never found a proper place in the legislation of many countries (here is a recent post by Quinn Emanuel legal firm for the US market).

Although not given a dedicated space by lawmakers, it still is so wi(l)dely used, that some high-level conclusions can be drawn.

Self-guidance organisms

Professional investors, such as hedge funds, were among the first to adopt some form of self-guidance for web scraping, as they operate in an SEC-regulated market, with a specific accent on Material-Nonpublic Information (MNPI) and the risks of insider trading.

These self-guidance organizations are the Investment Data Standard Organization IDSO and the Alternative Data Council of FISD. They have done and still do an excellent job in shedding light on a very complex and sometimes very technical subject. Please comment if you know of other organizations, especially in other industries.

The rise of genAI and the need for web data

The recent surge in genAI web data demand mixed with the “move-fast, break things” philosophy of Silicon Valley, drove a rise of lawsuits hitting companies of any size.

Just as a reminder, here is a non-exhaustive list of the three most recent lawsuits related to web scraping that come to mind:

• September 2023, a class action was filed against Microsoft for how ChatGPT used privacy data

• July 2023, Google hit with class-action lawsuit over AI data scraping

• June 2023, Lawsuit says OpenAI violated US authors' copyrights to train AI chatbot

Now, as the mission of Data Boutique is to promote the safe and fair use of web-scraped data, we will use this post to try to define the high-level context. We hope this will help someone out there.

Web Scraping Legal Context

General advice: We are not lawyers; don’t take our word as pure gold, and speak to your legal counselor if you have questions on web scraping. Data regulation changes materially by geography and over time. We do our best to provide helpful information, but this is our own interpretation and might be inaccurate.

When it comes to web scraping, there are two elements we need to consider:

1. The nature of data, which impacts our right to buy, use, transform, and sell it

2. The visibility of data, which strictly regards the act of collecting, if it is in breach of property laws or existing contracts or agreements

1. The nature of the data collected

The main attention of regulators and lawmakers is toward protecting citizen’s privacy and copyright holders’ interests. There are three different types of protected data:

1. Personal Data

2. Intellectual Property

3. “Sui generis” Database

Personal Data

Personally Identifiable Information (PII) is information that, when used alone or with other data, can help identify an individual. Privacy laws protect the rights of citizens, like the European General Data Protection Regulation (GDPR) or the California Consumer Privacy Act (CCPA).

If you collect an individual's name from LinkedIn, for example, without her/his consent, you are very likely violating something. So, as a rule of thumb, don’t.

Intellectual Property

Intellectual Property (IP) is content that is the result of human creativity, from images to audio, to written content, computer code, and so on. Humanity has used variations of this concept since 1474, and it is very well protected from a legal standpoint.

To be considered Intellectual Property, the content needs to be original and have a form of uniqueness. A long text form, like a NYT article, fits the definition. A generic text like “Blue T-shirt” does not.

In general, in order to use, transform, or sell intellectual property, you need to have IP owner’s consent.

There are exceptions, like the fair use of copyrighted material and the debate to extend it also for training AI (allowing copyrighted material to be used by AI even without the consent of the owner). More exceptions to the use of copyrighted material exist around the world, and the situation is always evolving.

“Sui Generis” Database

Sui Generis Property Right (or Database Right) is a form of protection of databases, regardless of their originality, that required great effort to be built. If there was a substantial investment, either financial or of work, in creating, validating, and presenting the database itself, the law protects it. It exists in the EU, the UK and other countries, but - as of today, not in the USA.

There are exceptions to the collection and use of copyrighted information related to the purpose of the data collection (academia and scientific research can) or the extent of the scraping (scraping a little is tolerated, scraping the whole website, not).

What is not covered?

If web scraping for personally identifiable information (PII), IP-protected data o EU databases exposes to risks, non PII and non IP protected, does less so. This is where it gets interesting (and yes, debated).

When is content not covered by IP? When is a database not sui generis covered?

Are prices and timetables of an airline covered by some form of IP? This 2015 European Court of Justice ruling stated that Ryanair flight prices and timetables did not constitute IP content (prices and timetables are factual measures, which do not have the intrinsic element of originality) and the database did not have the right to be covered under EU Database legislation. Web scrapers can still be sued for breaking website terms and conditions.

Are ChatGPT texts, or Midjourney images, protected by copyright? A federal judge ruled that artificial intelligence or AI-generated content can’t be copyrighted. If this was also confirmed by other rulings and geographies, entire AI-generated websites could not have their content protected.

These two points mark an interesting line for web scraping. As long as it doesn’t violate the law, these data seem legit to collect.

2. The visibility of data

Data on the internet can be accessed in many ways, some more legal than others.

It is easy to understand that violating computer systems to access information is different than accessing the information available to anyone with no particular technical skill.

Non-public information

Any internal website, company portal, or anything that prevents an outsider from seeing the information, is non-public. Only employees of a company see it, if it’s information that is reserved to a single individual (like a bank account) and needs user’s credentials to access, is non-public.

If the web scraped data has the consent of the user, and the conditions for this consent are met, then scraping is possible. But if some of these elements fail, web scraping is very risky. A basic rule of thumb: If you need to hack it to see it, don’t scrape it.

Public information

When anyone with an internet connection can see the information, that is public.

Some websites, even if public, are required to agree to terms and conditions to see content. If you don’t agree, you get no access.

It is highly possible that these terms prohibit web scraping (see LinkedIn example here).

According to self-guidance organizations mentioned above, only Terms and Conditions that require explicit acceptance by users (click-wrap ToS) are valid, while generic footnotes (browse-wrap ToS) do not provide sufficient ground for enforcement.

Robots.txt

This is probably the most relevant element for web scraping activities: A little file called robots.txt under www.websitename.com/robots.txt is the explicit indication from websites to robots.

This little file, optionally present in many websites, tells robots where they are allowed to crawl. This is not the law. It’s a “de facto standard”, born in 1994. It’s a way the website owner is telling bots where to go.

For example, the Farfetch.com file https://www.farfetch.com/robots.txt contains this line:

Allow: /it/*?*lang=it-IT

Which tells bots they are allowed to crawl the www.farfetch.com/it/ part of the website.

How can a data buyer know?

Assessing if web scraped data violates IP, database laws, robots.txt files can be quite exhausting. This is why we promote safe access to data.

Web scraping is only a tool, just like a hammer. You can use the hammer to build useful things like a bridge or a house, or use it fraudulently to smash a car’s window and steal from it. Understanding the difference defines how we interact with society.

We do our best to make take web scraping - at the foundation of so many technologies or websites we know or use - come out of its “Wild West moment”.

Web data is everywhere. We need to start treating it and trading it properly.

About the Project

That was it for this week!

Data Boutique is a community for sustainable, ethical, high-quality web data exchanges. You can browse the current catalog and add your request if a website is not listed. Saving datasets to your interest list will allow sellers to correctly size the demand for datasets and onboard the platform.

More on this project can be found on our Discord channels.

Thanks for reading and sharing this.

Iscriviti adesso

Data Boutique is a web-scraped data marketplace. If you’re looking for web data, there is a high chance someone is already collecting it. Data Boutique makes it easier to buy web data from them.

Web data is an enabler, not the core.

Recent decades have seen the creation of countless applications that use web data, from generative AI to market intelligence, from search to dynamic pricing. Data is the new oil (they said 10+ years ago).

But let’s not be confused: Web data is the enabler for these apps, it is not the apps.

Data is an enabler, just like servers are. Yet nobody owns servers anymore: We access them “as a service”.

Data is an enabler, just like business software is, from CRM to analytics tools. 20 years ago, companies would write their own. Today, accessing them “as a service” is the best practice.

Web data is an enabler. Once it made sense to collect it internally. Now, web scraping has become a resource-eating, risk-exposing activity. The number of challenges a scraping developer has to go through, from chasing websites’ code dynamics, to circumventing anti-bot software, significantly raised the bar.

5 reasons to ditch web scraping and start buying it as a service (DaaS)

Should a company switch from in-house scraping for public web data to Data as a Service (DaaS)? Let’s see when it makes sense.

I use the term “public web data” as it implies that it is accessible and collectible by anyone with the proper tools, and it is not something exclusive to the company that will be using it.

1. Cost

Reason number one for switching to DaaS: Cold, hard cash.

Think of all the cost components of in-house web scraping - people involved, proxy providers, hardware, tools, or the annual fee from data farms when outsourced.

These costs are:

1. Very inelastic - once engaged, they’re not very sensitive to scaling down the frequency of the scraping

2. Hard to tell if you’re paying too much (there is no price benchmark for this)

By switching to DaaS, these costs get SUPER elastic, and you know you are paying fair market prices - the town things combined often end in a 100X cost reduction for low-frequency data refreshes.

2. Scalability

How fast can an in-house team add 10 websites? And how fast can they stop those websites and start doing 10 others instead?

This is maybe the largest advantage of DaaS.

• You can scale in frequency (going from monthly data collection to weekly or daily) can be handled on the fly and changed (scaled-up or scaled-down) anytime, as many times as necessary. There are no commitments.

• You can scale in scope: If we want to add (or remove) 10 websites similar to the one we are collecting already, this also can be handled on the fly, with no technical knowledge or execution delay.

As a consequence, you can build very fast, very cheap PoC (Proof of Concept) at very little cost, and then scale them up to production right when needed.

3. Talent allocation

Spending time figuring out what line of code broke in your scraper might not be the best use of your talent’s time. Why? Because there are hundreds of other talents out there doing the same thing, for that exact same website.

It would be much more efficient to spend it on how to use this data, how to transform it, structure it, change domains, convert, and lookup values. All activities that were already there, even before, but they were understaffed.

If you free your talent’s time from activities that can be found on the market, and have it on stuff critical and differentiating, you get all hands on what’s really differentiating.

4. De-risking

Keeping web-scraping internal? Be prepared for due diligence, maintain policies, adhere to regulations, and disclose procedures and logs. For those selling data to hedge funds and other regulated entities, this may not come as a surprise, but copyright and privacy lawsuits are getting frequent also for AI and SaaS, regarding their web scraping activities.

Again: Procedures, log, disclosure. Is this the best way to allocate your talent’s time?

Or, when you think this is not core to you, buy from a platform instead, and have this sorted.

5. Strategic Positioning

All this finally leads to the core question: Is your company a SaaS/AI company, or is it a web scraping company?

The alignment of all activities on your strategic positioning helps focus on what’s differentiating about your company. Just like it is not core to hold and maintain servers, and develop and maintain CRM software, maybe it’s not web scraping when this can be found on the market.

Is web scraping core instead? Then monetize it.

This industry has a lot of talented players. Many of which, having worked for years in web scraping, are just not yet ready to jump to DaaS. I understand that.

Are you more confident in your own data? Is your cost base competitive with what you find on Data Boutique? Is your data acquisition pipeline so strong you’d trust it more than what you’d find on a marketplace?

Fantastic. You should then monetize this capability, and sell on Data Boutique. It’s a win-win.

About the Project

That was it for this week!

Data Boutique is a community for sustainable, ethical, high-quality web data exchanges. You can browse the current catalog and add your request if a website is not listed. Saving datasets to your interest list will allow sellers to correctly size the demand for datasets and onboard the platform.

More on this project can be found on our Discord channels.

Thanks for reading and sharing this.

Iscriviti adesso

Data Boutique is a web-scraped data marketplace. If you’re looking for web data, there is a high chance someone is already collecting it. Data Boutique makes it easier to buy web data from them.

Network Effects: When Value Comes from Others Joining in

TL;DR: Network effects in Data Boutique favor buyers who refer buyers, and sellers who refer sellers, because of mutual self-interest. Plus: The launch of our Partnership Program: Long-term value sharing with our network.

Join The Program

Buying and selling web scraped data is a fragmented business, and this fragmentation causes hidden costs.

Whether you buy it or hire someone to collect it, it’s a one-on-one relationship between buyer and seller: Finding each other, selecting, testing, validating, and negotiating the transaction takes time.

You must go through the same process when you need it again. Every. Single. Time.

This is not the case for data marketplaces (heavily managed marketplaces1, to be precise): Once on the platform, every time you buy data, most of these costs disappear. From the second purchase onwards, transaction costs are lower: Data streams with the same format, structure, delivery method, quality controls, and payment methods.

But there is more: Every single person who joins the platform adds value to all other actors.

More buyers is good for buyers (cost-wise)

What is obvious is that more buyers is a good thing for sellers.

What is less obvious is that more buyers is also a good thing for buyers. Let’s see why.

1. Data selling (with pre-scraped datasets) is a fixed-cost activity. Once the collection costs are covered, every new sale simply adds margins.

2. The more sales units, the more margin a seller has to lower prices and attract more buyers.

In practice: If a data product is purchased by just one client, the seller is forced to keep higher prices to cover costs. But this is less than ideal, as the convenience vs. in-house web scraping would be smaller.

When more clients buy the same product, scale economies will give the seller room to lower unit prices and attract more sales. Competition, in fact, is not seller vs. seller, but seller vs. in-house scraping.

Although buyers may be skeptical about suggesting to others where to source data, in the end it’s a self-interest mechanism: Data Buyers have a direct incentive to refer users to the platform as it lowers their future costs.

Let’s say I know Peter. Peter is using web data for image recognition in fashion e-commerce. I am using web data for market intel in fashion. I want Peter to join the marketplace, as in the long run costs will go down for me.

More sellers is good for sellers (density)

Let’s look at the other side: Sellers.

Web scraping is highly competitive. Try posting a web-scraping job on Upwork and see how many apply, you’d be surprised. So, it’s natural for sellers to be reluctant when other sellers join a marketplace, as they are afraid of competition.

In a talent marketplace (Upwork, or Fiverr), this is true: The more people doing web scraping join in, the higher the competition. It’s a losing price-squeezing game.

But the economics of data marketplaces are different: If sellers offer different data products - and we as market makers need to make sure this is incentivized - more sellers means more products. It’s a win-win, here’s why:

Data products, unlike others, are not mutually exclusive: They are mutually reinforcing. There is a higher chance buyers searching for dataset A, will purchase dataset A and B, if B is available and adjacent to A, as it completes the information set.

An example: Let’s say I’m a data seller offering H&M data, and I know Tina. Tina is scraping ZARA website. It is my interest to have Tina with me on the platform offering ZARA, because this raises my sales of H&M if we’re together. The sum is greater than the parts.

Now, if Matthew joins, offering MANGO or BERSHKA, we’re getting closer to building a cluster of data for fast-fashion brands, that we alone would have trouble building, given the cost of collecting all of the brands individually.

Some may argue, I could have scraped ZARA directly, instead of calling in Tina, but the reality is I would harvest a bigger gain if I scraped UNIQLO, or FOREVER21, and still called Tina in for ZARA.

Sellers have the incentive to put their effort in adding websites when others don’t cover them, and calling in others, when they do. This allows them to grow the graph faster, and reach together a larger audience, than they’d be able to do alone.

Plus: Launching The Partnership Program

We have seen network effects. Let’s speak about the Partnership Program:

Soon after its launch, we saw Data Boutique’s impact reach beyond the core web-scraping community: Decoupling data collection from data usage created new applications. Without the burden of data sourcing, we enabled teams to access data and create PoCs and MVPs quickly, safely, and with low investments.

From Business Intelligence to AI, from price monitoring to market research, applications are countless.

This is why we decided to open our Partnership Program: To strenghten our outreach to those applications, BI teams within corporates, AI startups and geeks. We offer a multi-year value-sharing program, rewarding the growth of the network.

Join The Program

Enter the program and refer data-hungry companies. It’s a variable-sum game.

Joint, all win.

About the Project

That was it for this week!

Data Boutique is a community for sustainable, ethical, high-quality web data exchanges. You can browse the current catalog and add your request if a website is not listed. Saving datasets to your interest list will allow sellers to correctly size the demand for datasets and onboard the platform.

More on this project can be found on our Discord channels.

Thanks for reading and sharing this.

Iscriviti adesso

Data Boutique is a web-scraped data marketplace.

If you’re looking for web data, there is a high chance someone is already collecting it. Data Boutique makes it easier to buy web data from them.

Join our Platform to learn and interact about this project:

Join Data Boutique

Zoom, GPTbot, and Google “fair use” of copyrighted data

Three pieces of news inherent to what we do at Data Boutique came out recently:

1. Videoconferencing app Zoom Terms of Service updates caused users’ resentment, as they feared their personal conversations might be (or might have been) used to train AI. Zoom later said users need to opt-in to the service, but nonetheless, it caused a heated debate;

2. OpenAI released the specifics of the GPTbot, a web crawler aimed at collecting data for LLMs. OpenAI states it allows websites to opt-out of being scraped, but the question remains on what the incentive would be for websites to stay in, since they would be feeding for free a paid service to Microsoft with no reward;

3. In an Australian court, Google is claiming a “fair use” of copyrighted data collection. Basically, Google admits unlawful crawling of copyrighted data, claiming it shouldn’t be considered unlawful. Like the OpenAI case, Google claims that websites can opt out of the service, triggering the same questions that arise for the GPTbot case (given that they can separate it from the Google bot aimed at the search engine).

While this is not the place to discuss the details of this news, I am leaving a link to a podcast I recommend by Nathaniel Whittemore covering Zoom and the OpenAI case (looking forward to a new one on Google).

Why this is relevant to us

What is interesting for our case is the rising disclosure of data collection methods for AI, driven by users’ debate, regulators, and court filings. In summary:

1. Public opinion and regulators are paying more attention to what data is used in AI and are increasingly skeptical about providing their own for free;

2. Companies are forced to move out of stealth and disclose their data collection operations;

3. Current solutions seem to be aligned on adding an opt-in/out feature, which suggests a transition of sourcing models from a “complete but unethical” to a “clean but partial” one, as we can expect many to opt-out.

The Data Market Question

While the Zoom case refers to collecting data on how users interact with their application, which cannot be collected in other manners (yet does not imply users are willing to adhere), the OpenAI and Google cases refer to collecting generic purpose data, and they represent just a tiny portion of the entire AI ecosystem using web-scraped data to feed their business models which is in the same situation.

In the eyes of the consumer, they stand not accused of the quality and innovation of their algorithms and models but of the way they source data.

There is an obvious decoupling of the business of collecting data from the business of using the data, especially since progress in technology made the tools for using this data available to everyone.

In our vision, this separation between data collection and usage, and the increasing attention from public opinion and regulators on the first one, sets the stage for the general adoption of data markets: Markets on data that exists regardless of its use cases, data whose collection methods need to adhere to accepted rules, and requires quality assurance to prevent selection bias on the applications that will be developed on top of it.

A data market is where data collection is brought to light, adopts common rules, performs professionally and reliably, and is offered at fair market prices.

The Value of Accountability

When AI projects keep data collection in-house, they are onboarding a commoditized portion of the value chain with an embedded responsibility on a critical aspect in today’s society: Where and how did you get the data? Users, public opinion, regulators, and investors will sooner or later pose that question.

Data marketplaces address this. By being accountable for this, they relieve data and AI projects from the burden of a time- and resource-consuming activity loaded with reputational and operational risks.

Although not all data needs can be addressed by marketplaces, why on earth would you embark on something that burns cash, takes time, brings on risks, and raises eyebrows when you could have found it in a store? Because data is critical to your project? That is the exact reason why you should not do it in-house.

Join the Project

That was it for this week!

Data Boutique is a community for sustainable, ethical, high-quality web data exchanges. You can browse the current catalog and add your request if a website is not listed. Saving datasets to your interest list will allow sellers to correctly size the demand for datasets and onboard the platform.

More on this project can be found on our Discord channels.

Thanks for reading and sharing this.

Iscriviti adesso