Automated Data Warehousing for Marketplace Data (Without Hiring Engineers)

Every brand that sells across more than a handful of channels eventually tries to build one view of the business. It usually starts with someone exporting CSVs on a Monday morning.

Then it becomes a shared drive. Then a set of increasingly baroque spreadsheets. Then, if the brand is doing well enough, a request for a data warehouse and someone to maintain it.

The interesting thing is that the hard part is almost never the extraction.

Extraction is a solved problem. Agreement is not.

You can get data out of most seller portals. APIs where they exist, RPA and scheduled exports where they do not, email parsing for the stragglers. It is fiddly and it breaks when a portal redesigns, but it is tractable.

What breaks the project is that twenty channels have twenty definitions of the same concept.

• One marketplace reports an order at checkout. Another reports it at dispatch. A third reports it at delivery. Sum them naively and your order count is meaningless.
• Returns may appear as negative orders, as a separate return feed, or as a settlement adjustment weeks later with no link back to the original order.
• Ad spend is reported gross on one platform and net of credits on another, on different attribution windows.
• Inventory might mean units in the fulfilment centre, units sellable, or units including inbound — and the label rarely tells you which.
• SKU identifiers are the worst of it. Your internal code, the channel's SKU, the ASIN or listing ID, and the barcode are four different things, and the mapping between them is maintained by hand in most brands.

Load all of that into a warehouse without reconciling the definitions and you have built something worse than spreadsheets: a single source of confidently wrong truth.

What normalisation actually involves

Three layers, and skipping any of them causes problems later.

Raw. Land the data exactly as the source provided it, timestamped, immutable. This is your audit trail and your recovery position when a transformation turns out to be wrong. Do not skip it because storage feels wasteful.

Canonical. Map every source into a common schema with explicit definitions. One definition of an order, one of a return, one SKU key that everything resolves to. This is where the actual thinking happens, and it is a business decision as much as a technical one — somebody has to decide what an order means for your brand.

Serving. The aggregated tables and metrics analysts and dashboards actually query.

The mapping layer is where most in-house attempts stall, because it is never finished. Channels change their exports. New marketplaces get added. A category team starts selling on a platform nobody told data about.

The maintenance question

This is the part worth being honest about. A pipeline built once is a pipeline that breaks quietly.

Portals change field names without notice. An export that returned twelve columns returns thirteen. A rate limit tightens. None of these produce an error message that reaches the right person; they produce a number that is slightly wrong in a dashboard, and it stays wrong until somebody notices the trend looks odd.

Which is why the useful question about any data pipeline is not "can we build it" but "who fixes it on a Tuesday when Flipkart changes a column name". If the answer is a single analyst who also has a day job, the pipeline has a shelf life.

What this looks like working

Opptra sells across 20+ marketplaces in India, the Middle East and Southeast Asia, managing a portfolio of brands. Their reporting was scattered across seller portals, scraping feeds and ad dashboards, with no normalisation between them.

Automating extraction and standardising it into their warehouse gave them same-day data freshness across 100k+ SKUs, and returned 100+ hours a month that had been going into manual pulls and cleanup.

The freshness number is the one that changed behaviour. Weekly data supports reporting. Same-day data supports decisions — noticing a buybox loss on Tuesday rather than in next week's review.

A reasonable starting scope

Do not begin with everything. Begin with the two or three tables that decisions actually depend on.

For most consumer brands that means orders, inventory and settlements — in that order. Ads and competitive data are valuable but rarely urgent, and they are the ones most likely to change format.

Get those three landing reliably, normalised, with a defined SKU key, and you will have covered most of what people were opening spreadsheets for. Everything after that is an addition rather than a rebuild.

Questions we get asked