Importing bulk data into Odoo using S3 fast load setup
This guide will walk you through creating importable files and getting them picked up for load into your Odoo database hosted by Steersman.
This will allow you to update products, create price lists, add vendor prices, add and link images, and more, without needing to obtain external IDs for products. Odoo updates data in the background, without locking records, causing concurrent update issues or tying up resources used to run your database processes. If you need to update a million products simultaneously, this is the way to do it. You can also schedule these updates to run on a date and time of your choice.
Preliminary setup required
You will need to have your own AWS account and provide Steersman with information about the AWS users who must have access to bulk data updates in your Odoo. Steersman will set up permissions for these users to load files into a partition of our AWS S3 file storage, from which Steersman’s scripts will perform fast and stable data loading into your database.
Uploading your data files
After Steersman provides access to the AWS S3 location, access your AWS location and navigate to your S3 folder.
- Your folder for data files within Steersman’s S3 will be:
s3://sbs-uploads/file_engine/"company name"/csv/- Here you will be able to add new files and see file statuses, with files showing within sub-folders
- ‘processing/’ - these are files that have been picked up and are scheduled for processing or being processed
- ‘done/’ - these are files that were successfully uploaded
- A multi-line file may have some lines fail to import/update - in such cases, we skip those specific lines, but the import keeps running for other lines
- ‘error/’ - these are files that had a critical error upon import
- Here you will be able to add new files and see file statuses, with files showing within sub-folders
- When uploading a file, place your CSV file into a subfolder ‘new/’
- New files are picked up every minute, but could be processed later based on your requested processing data and the processing queue
- The name of your file will define how it will be ingested – files named not per the required format will not be ingested or may be ingested incorrectly, potentially causing damage to your data
Uploading new images and PDF documents to be added to your PIM in Odoo
There are two steps to loading images and documents – you need to upload them to make them available to Odoo, and then provide a file specifying the relationships of images or documents to products.
- Your folder for adding images/documents to Steersman’s S3 will be:
s3://sbs-uploads/file_engine/"company name"/media/- Here you will be able to add new files and see file statuses, with files showing within sub-folders
- ‘processing/’ - these are images that have been picked up and are scheduled for processing or being processed
- ‘done/’ - these are files that were successfully uploaded
- A multi-line file may have some lines fail to import/update - in such cases, we skip those specific lines, but the import keeps running for other lines
- ‘error/’ - these are files that had a critical error upon import
- Here you will be able to add new files and see file statuses, with files showing within sub-folders
- When uploading new images or documents, place your image/document files into a subfolder ‘new/’
- Your file names will be used to map files to products via a CSV import
Supported data files and what they update.
Click on individual file types to see naming and field requirements for each file/data type.
- Vendor costs
- Adds new vendor prices and auto-disables prior price for the same product from the same vendor.
- Product scheduled sale price
- Adds scheduled price lines into the “sale price” (“list price”) field and sets previously active price lines to expire on the “start date” of the new price.
- Product scheduled minimum advertised price (MAP)
- Adds scheduled price lines into the “minimum advertised price” (“MAP”) field and sets previously active price lines to expire on the “start date” of the new price.
- Product scheduled normal market cost
- Adds scheduled price lines into the “normal market cost” field and sets previously active price lines to expire on the “start date” of the new price.
- Static pricelist (pricelist with numeric prices for specific products without logic or formulas)
- Adds manual prices for specific products into a designated “pricelist” that must be configured as a “static” pricelist. Ok to add 1,000,000+ price entries into a single pricelist that way.
- Product images
- Specifies which images in the media library should be assigned to products and allows determining the sequence for the images on each product.
- Product documents
- Specifies which documents in the media library should be assigned to products and allows determining the sequence for the documents on each product. Example documents include MSDS and spec sheets, which are displayed on product pages as PDFs.
- Product website availability flags
- Specifies whether a product should be published to the website, and if so, what availability profile it should be assigned to (allow add to cart or block, display in categories, or leave published for indexed pages only, stock availability messaging, etc)
- Product reorder rules
- Adds or updates reorder rules for specific products in specific locations. Automatically updates an existing record in case of matching product and location instead of adding a new rule.
- Product information
- Creates or updates products and various product identifiers, dimensions, and descriptions.
File specifications
Vendor costs
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-vendor-cost.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| min_quantity | True | |
| cost | False | empty to delete |
| sequence | False | will push other lines down, last by default |
| lead_time_days | False | integer, ‘delay’ field in Odoo |
| date_start | False | YYYY-MM-DD |
| date_end | False | YYYY-MM-DD |
| code | False | vendor code |
| name | False | vendor product name |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Product scheduled sale price
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-price-schedule-list.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| price | True | |
| date_start | False | YYYY-MM-DD |
| date_end | False | YYYY-MM-DD |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Product scheduled minimum advertised price (MAP)
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-price-schedule-map.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| price | True | |
| date_start | False | YYYY-MM-DD |
| date_end | False | YYYY-MM-DD |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Product scheduled normal market cost
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-price-schedule-normal-market-cost.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| price | True | |
| date_start | False | YYYY-MM-DD |
| date_end | False | YYYY-MM-DD |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Static pricelist
pricelist with numeric prices for specific products without logic or formulas.
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-pricelist-static.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| pricelist_id | True | database product.pricelist id |
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| price | False | empty to delete |
*It is required to have one of the product identifiers (product_variant_id, orinternal_reference, orproduct_template_id`), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Product images
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-product-image.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| url | True | S3 or public HTTP URL |
| name | False | generated from the URL if missing |
| slug | False | generated from name if missing |
| tag_ids | False | comma separated values, whole cell in quotes |
| sequence | False | will push other lines down, last by default |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Product documents
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-product-document.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| url | True | S3 or public HTTP URL |
| name | False | generated from the URL if missing |
| slug | False | generated from name if missing |
| tag_ids | False | comma separated values, whole cell in quotes |
| sequence | False | will push other lines down, last by default |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Product website availability flags
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-product-website.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| website_id | True | odoo database id |
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| availability_id | False | empty to disable |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Product reorder rules
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-product-reorder-rule.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| location_id | True | odoo database id |
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| active | True | boolean |
| trigger | True | “auto” or “manual” |
| min_quantity | True | |
| max_quantity | True | |
| quantity_multiple | True |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Product information
- File name format:
- YYYY-MM-DD-hh-mm-seq-type.csv
- The date and time set are the earliest times to process the file in UTC
- 2024-11-18-14-28-001-product.csv
- YYYY-MM-DD-hh-mm-seq-type.csv
| Column header (field) | Required | Description |
|---|---|---|
| product_variant_id | True* | |
| internal_reference | True* | |
| product_template_id | True* | |
| name | False | |
| model_name | False | model |
| barcode | False | |
| mpn | False | |
| oempn | False | alternate part numbers, comma separated |
| nsn | False | national stock number |
| unspsc | False | |
| hs_code | False | |
| asin | False | |
| weight | False | |
| length | False | |
| width | False | |
| height | False | |
| description_preview_md | False | preview description (markdown) |
| description_short_md | False | summary (markdown) |
| description_long_md | False | full description (markdown) |
| slug | False | website slug |
| meta_title | False | |
| meta_description | False | |
| meta_og_title | False | OpenGraph title (Facebook/Meta) |
| meta_og_description | False | OpenGraph description (Facebook/Meta) |
| meta_twitter_title | False | card title (Twitter/X) |
| meta_twitter_description | False | card description (Twitter/X) |
| public_brand_ids | False | database brand ids, comma separated |
| public_category_ids | False | database category ids, comma separated |
*It is required to have one of the product identifiers (product_variant_id, or internal_reference, or product_template_id), if more than one of these identifiers is provided - the first one will be used in the order listed here, and others will be disregarded.
Related posts
How do Odoo E-Commerce Sites from Steersman Rank?
Odoo e-commerce sites from Steersman will get you the best performance, structure, and SEO in the world. See the ratings for yourself.
Receiving Invoices using EDI in Odoo
The EDI 810 inbound integration by Steersman enables seamless import and processing of electronic invoices into Odoo. By automating invoice receipt and validation, businesses can streamline their accounts payable operations, ensuring accuracy, compliance, and efficiency across the financial workflow.
Receiving Purchase Order Acknowledgments(POAs) using EDI in Odoo
Steersman’s EDI 855 inbound integration enhances the order management process by automating the receiving and processing of purchase order acknowledgments (POAs) within Odoo.
How Great PIM Systems Improve Online Catalog Management
A good catalog management system should offer control, usability, efficiency, and help boost sales. Here are some things that good PIM systems should have.