Feature #92
openCourier Bill of Entry PDF-to-JSON Converter Wizard
Description
Courier Bill of Entry PDF-to-JSON Converter Wizard¶
Frappe-based wizard for converting Courier Bill of Entry PDF documents into structured, application-ready JSON.
1. Project Overview¶
The Courier Bill of Entry PDF-to-JSON Converter Wizard is a Frappe-based document extraction and conversion solution developed to automate the processing of Courier Bill of Entry (BOE) PDF documents.
The user uploads a Courier Bill of Entry PDF through the wizard. The system analyzes the document, extracts the required document-level, invoice-level, and item-level information, normalizes the extracted values, validates the resulting structure, and generates structured JSON.
The implementation is based on the document-extraction approach developed for the Shipping Bill PDF Extractor, while introducing Courier BOE-specific parsing and JSON mapping logic.
The main objective is not simply to extract text from a PDF. The important requirement is to correctly understand the document structure and place every extracted value at the correct level in the final JSON.
2. Business Problem¶
Courier Bills of Entry contain multiple types of information within the same document.
The information can include:
- Bill of Entry details.
- Importer information.
- Supplier information.
- Invoice information.
- Item information.
- Quantity and rate.
- Assessable values.
- IGST and other duty-related values.
When this information is processed manually, users need to:
- Open the Courier BOE PDF.
- Locate the required fields.
- Identify the corresponding values.
- Identify invoice sections.
- Identify item rows.
- Associate items with the correct invoice.
- Format numeric values.
- Build the required JSON structure manually.
This creates unnecessary manual effort and can result in incorrect mappings, missing values, and incorrect JSON hierarchy.
The wizard automates this PDF-to-JSON conversion process.
3. Project Objectives¶
The main objectives of the project are:
- Provide a Frappe wizard for Courier BOE PDF processing.
- Accept Courier Bill of Entry PDF documents.
- Analyze multi-page PDF documents.
- Extract Courier BOE document-level information.
- Extract invoice-level information.
- Extract item-level information.
- Maintain the relationship between invoices and their items.
- Normalize numeric and monetary values.
- Generate structured Courier BOE JSON.
- Provide JSON for copying and downloading.
- Prevent invoice-level and item-level fields from being mixed.
- Reuse the common PDF analysis functionality.
- Keep Courier-specific parsing separate from the Shipping Bill parser.
- Provide a reusable foundation for different Courier BOE PDFs.
4. Solution Overview¶
The complete workflow is:
Courier BOE PDF
│
▼
Upload in Wizard
│
▼
PDF Validation
│
▼
PDF Analysis
│
▼
Courier BOE Parser
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Header Invoices Items
│ │ │
└─────────────┼─────────────┘
▼
Data Normalization
│
▼
Structure Validation
│
▼
Courier BOE JSON
│
┌─────┴─────┐
▼ ▼
Copy Download
5. Wizard Workflow¶
The Courier wizard follows the established document-extraction wizard approach.
Step 1 — Upload PDF¶
The user uploads the Courier Bill of Entry PDF.
The uploaded document becomes the input to the backend processing pipeline.
The wizard is designed so that the document values are discovered from the uploaded PDF rather than being hardcoded into the frontend.
Step 2 — Analyse & Review¶
After upload, the document is analyzed and parsed.
The system processes the PDF and extracts:
- Courier BOE header information.
- Supplier information.
- Invoice information.
- Item information.
- Relevant monetary and quantity values.
The extracted information can then be reviewed before generating the final JSON.
Step 3 — JSON Output¶
The final structured Courier BOE JSON is displayed to the user.
The wizard provides the ability to:
- Review the JSON.
- Copy the JSON.
- Download the JSON.
This eliminates the need for users to manually construct the JSON structure.
6. Application Architecture¶
The implementation follows a layered architecture.
┌─────────────────────────────────────┐
│ Courier BOE Wizard UI │
│ JavaScript / CSS / JSON │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Courier API Layer │
│ Upload / Processing Flow │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ PDF Analyzer │
│ PDF / Page / Text Analysis │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Courier BOE Parser │
│ Header / Invoice / Item Parsing │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Normalization & Validation │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Courier BOE JSON │
└─────────────────────────────────────┘
7. Reuse of PDF Analysis¶
The Courier implementation is designed to reuse the common PDF analysis layer instead of duplicating PDF-processing functionality.
The architecture can be represented as:
PDF Analyzer
│
┌────────┴────────┐
▼ ▼
Shipping Bill Parser Courier BOE Parser
│ │
▼ ▼
Shipping Bill JSON Courier BOE JSON
This provides a common foundation for PDF analysis while allowing each document type to have its own parsing rules.
The Shipping Bill parser and Courier BOE parser remain separate because the two document types have different structures and field requirements.
8. Courier-Specific Components¶
The Courier implementation contains Courier-specific frontend and backend components.
Typical components include:
courier_bill_extrac.js
courier_bill_extrac.css
courier_bill_extrac.json
courier_bill_parser.py
Courier-specific API
Component Responsibilities¶
| Component | Responsibility |
|---|---|
courier_bill_extrac.js |
Courier wizard UI and frontend interactions |
courier_bill_extrac.css |
Courier wizard styling |
courier_bill_extrac.json |
Frappe page configuration |
courier_bill_parser.py |
Courier BOE extraction and parsing |
| Courier API | Courier-specific processing and orchestration |
pdf_analyzer.py |
Common PDF analysis functionality |
9. Dynamic Courier BOE Extraction¶
The Courier parser is designed to process values from the uploaded document rather than using fixed sample values.
The extraction process is:
Uploaded Courier BOE
│
▼
Analyze PDF Structure
│
▼
Identify Relevant Labels
│
▼
Locate Corresponding Values
│
▼
Identify Document Sections
│
▼
Build Document / Invoice / Item Data
│
▼
Normalize Values
│
▼
Generate JSON
The same processing logic should be capable of handling Courier BOEs containing different:
- BOE numbers.
- Dates.
- Importers.
- Suppliers.
- Invoice numbers.
- Quantities.
- Rates.
- Assessable values.
- Duty values.
- Item descriptions.
10. Document-Level Information¶
The main Courier BOE object contains information that belongs to the complete Bill of Entry document.
Examples can include:
- Bill of Entry information.
- Importer details.
- Supplier-related information.
- Dates.
- Customs-related information.
- Overall document values.
These fields must remain at the document level.
They should not be duplicated into invoice or item dictionaries unless they are explicitly part of those structures.
11. Invoice-Level Information¶
Courier BOEs can contain one or more invoices.
Invoice information is maintained separately from item information.
Conceptually:
Courier BOE
│
└── invoices
│
├── Invoice 1
│ ├── Invoice-level fields
│ └── items
│
└── Invoice 2
├── Invoice-level fields
└── items
The parser must preserve the invoice structure so that invoice-level fields remain associated with the correct invoice.
12. Item-Level Information¶
Items represent the individual goods associated with an invoice.
Depending on the Courier BOE structure, item-level information can include:
- Item description.
- Quantity.
- Unit.
- Rate.
- Assessable value.
- IGST amount.
- Other item-level monetary values.
The item fields must remain inside the appropriate items collection.
13. Invoice and Item Data Separation¶
A key implementation requirement is maintaining a strict separation between invoice-level and item-level fields.
The intended structure is:
Courier BOE
│
├── Document-level fields
│
└── invoices
│
└── Invoice
│
├── Invoice-level fields
│
└── items
│
├── Item-level fields
├── Quantity
├── Rate
├── Assessable Value
└── IGST Amount
This prevents unrelated fields from being added to the wrong dictionary.
14. Mapping Issue Identified During Development¶
During development, a field-mapping issue was identified in which fields belonging to one hierarchy were being injected into another hierarchy.
For example, item-level fields such as:
cour_be_item_assessable_value
cour_be_item_igst_amt
cour_be_item_qty
were being incorrectly added to the invoice dictionary with default 0 values.
At the same time, an invoice-level field such as:
cour_be_inv_value
was being incorrectly inserted into the item dictionary.
This created JSON structures containing unrelated fields and default values.
15. Mapping Correction¶
The parser mapping was corrected so that every field is assigned only to its intended hierarchy.
Correct Structure¶
Invoice
├── Invoice-level fields
│
└── items
├── Item-level fields
├── Quantity
├── Rate
├── Assessable Value
└── IGST Amount
Incorrect Structure¶
Invoice
├── Invoice fields
├── Item fields ❌
└── Item fields with default values ❌
Item
├── Item fields
└── Invoice fields ❌
The correction ensures that:
- Invoice dictionaries contain invoice fields.
- Item dictionaries contain item fields.
- Unrelated default fields are not injected.
- The generated JSON remains aligned with the intended schema.
16. Numeric and Monetary Value Normalization¶
Numeric fields extracted from the PDF are normalized before JSON generation.
This includes values such as:
- Quantity.
- Rate.
- Assessable value.
- Duty values.
- IGST amounts.
- Other monetary values.
The processing flow is:
PDF Text
│
▼
Raw Extracted Value
│
▼
Numeric Parsing
│
▼
Normalized Python Value
│
▼
JSON Number
The objective is to provide consistent numeric values in the final JSON rather than inconsistent string representations.
17. Supplier Information Handling¶
Supplier information can be difficult to extract reliably because PDF text may be split across multiple elements.
The Courier parser includes supplier-name reconstruction logic to correctly represent supplier information.
During development, supplier information was validated against the expected extracted value:
ENDRESS HAUSER AUTOMATION
The parser also preserves the required raw suppliers_address artifact string where it forms part of the expected extraction output.
This handling is necessary because PDF text extraction does not always preserve the visual grouping of supplier information.
18. PDF-to-JSON Processing Pipeline¶
The complete backend processing flow is:
Courier BOE PDF
│
▼
PDF Analyzer
│
▼
PDF Text / Structure
│
▼
Courier Parser
│
├── Header Parsing
│
├── Supplier Parsing
│
├── Invoice Parsing
│
└── Item Parsing
│
▼
Value Normalization
│
▼
Invoice / Item Structure Validation
│
▼
Courier BOE JSON
19. JSON Structure¶
The final output follows the Courier Bill of Entry hierarchy.
Conceptually:
{
"courier_bill_of_entry_json": {
"doctype": "Courier Bill Of Entry",
"...document_level_fields": "...",
"invoices": [
{
"...invoice_level_fields": "...",
"items": [
{
"...item_level_fields": "..."
}
]
}
]
}
}
The actual field names and values depend on the Courier BOE schema and the uploaded document.
The critical structural rule is:
Document
└── Invoice
└── Item
Each field must remain at its correct level.
20. JSON Copy and Download¶
The final wizard output is designed for practical downstream use.
Users can:
Copy¶
Copy the generated JSON directly from the wizard.
Download¶
Download the JSON as a file.
This makes the extracted data immediately usable for:
- API requests.
- Frappe document creation.
- Testing.
- Integration workflows.
- Further automation.
21. User Interface¶
The Courier wizard follows the existing document-extraction UI approach.
The frontend provides:
- PDF upload.
- Processing controls.
- Analysis results.
- Extracted JSON display.
- Copy functionality.
- Download functionality.
Courier-specific frontend files are kept separate from the Shipping Bill frontend implementation.
This prevents changes made for Courier processing from unnecessarily affecting the existing Shipping Bill wizard.
22. Validation Strategy¶
Validation is performed at multiple levels.
PDF Validation¶
- File exists.
- PDF is readable.
- PDF can be analyzed.
- Pages are processed correctly.
Field Validation¶
- Required fields are extracted.
- Values are associated with the correct labels.
- Numeric values are normalized.
- Supplier information is reconstructed correctly.
Structure Validation¶
- Document-level fields remain at document level.
- Invoice-level fields remain at invoice level.
- Item-level fields remain inside items.
- Invoice fields do not appear inside item dictionaries.
- Item fields do not appear inside invoice dictionaries.
- Unrelated default fields are not introduced.
23. Testing and Debugging¶
The Courier parser was iteratively tested during development.
Testing focused on:
- PDF analysis.
- Header extraction.
- Supplier extraction.
- Invoice extraction.
- Item extraction.
- Numeric conversion.
- JSON structure.
- Invoice/item hierarchy.
- Default-value contamination.
- Output consistency.
A significant debugging issue was the accidental injection of fields from one hierarchy into another.
The parser was refined so that field ownership is respected during JSON construction.
24. Key Technical Considerations¶
PDF Text Is Not Enough¶
Extracting text from a PDF does not automatically provide the correct business mapping.
The parser must understand the relationship between:
Label
+
Value
+
Position
+
Document Section
Field Ownership Must Be Explicit¶
Every field should have a clearly defined level:
Document
Invoice
Item
Default Values Must Not Create Incorrect Fields¶
A default value such as 0 should not cause an invoice-only field to appear in an item dictionary or an item-only field to appear in an invoice dictionary.
Numeric Values Must Be Normalized¶
Quantities, rates, duties, and monetary values should be represented consistently.
Supplier Information May Require Reconstruction¶
PDF extraction can split supplier names and addresses, requiring parser-specific reconstruction logic.
25. Separation from Shipping Bill Implementation¶
The Courier functionality is implemented as a separate document-specific parser.
The architecture is:
Common PDF Analyzer
│
┌──────────┴──────────┐
│ │
▼ ▼
Shipping Bill Parser Courier BOE Parser
│ │
▼ ▼
Shipping Bill JSON Courier BOE JSON
This allows both document types to share common PDF processing while maintaining independent extraction rules.
The Courier parser does not replace the Shipping Bill parser.
26. Current Project Baseline¶
The Courier implementation follows the existing document-extraction architecture.
The baseline approach is:
Existing PDF Extraction Architecture
│
├── Reuse PDF analysis
│
├── Preserve existing wizard pattern
│
└── Add Courier-specific processing
│
├── Courier UI
├── Courier CSS
├── Courier Page Configuration
├── Courier Parser
└── Courier API
The approach extends the existing project rather than redesigning the complete PDF extraction framework.
27. Technology Stack¶
| Technology | Purpose |
|---|---|
| Frappe Framework | Wizard and application platform |
| Python | PDF parsing and Courier BOE extraction |
| JavaScript | Wizard UI and frontend interactions |
| CSS | Wizard styling |
| JSON | Page configuration and structured output |
| PDF Processing | Document analysis and text extraction |
28. Benefits¶
The Courier Bill of Entry PDF-to-JSON wizard provides:
- Automated Courier BOE extraction.
- Reduced manual data entry.
- Faster document processing.
- Structured JSON output.
- Correct document/invoice/item hierarchy.
- Consistent numeric values.
- Supplier information reconstruction.
- Copy and download functionality.
- Reusable PDF analysis.
- Courier-specific parsing.
- Separation from the Shipping Bill parser.
- Easier debugging and validation.
- Foundation for downstream automation.
29. Future Improvements¶
Potential improvements include:
Enhanced Layout Detection¶
Improve extraction across different Courier BOE layouts.
Better Invoice Detection¶
Improve identification of invoice boundaries when multiple invoices are present.
Improved Item Table Parsing¶
Improve extraction of item rows and their relationship with the correct invoice.
Confidence Scoring¶
Assign confidence levels to extracted fields and highlight uncertain mappings.
Schema Validation¶
Validate the complete JSON against the expected Courier BOE schema before download.
Additional Layout Support¶
Support additional Courier BOE formats without introducing hardcoded document-specific values.
30. End-to-End Architecture¶
┌─────────────────────┐
│ Courier BOE PDF │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Wizard - Upload │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ PDF Analyzer │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Courier BOE Parser │
└──────────┬──────────┘
│
┌─────────────────────┼─────────────────────┐
│ │ │
▼ ▼ ▼
Document Fields Invoices Items
│ │ │
└─────────────────────┼─────────────────────┘
│
▼
┌─────────────────────┐
│ Normalization & │
│ Structure Validation│
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Courier BOE JSON │
└──────────┬──────────┘
│
┌─────┴─────┐
▼ ▼
Copy Download
31. Project Outcome¶
The Courier Bill of Entry PDF-to-JSON Converter Wizard transforms the processing flow from:
Manual PDF Reading
↓
Manual Field Entry
↓
Manual Invoice/Item Mapping
↓
Manual JSON Creation
into:
Courier BOE PDF
↓
Automatic PDF Analysis
↓
Courier-Specific Parsing
↓
Document / Invoice / Item Mapping
↓
Normalization
↓
Validation
↓
Structured JSON
↓
Copy / Download
The major implementation achievement is not simply reading the PDF text.
The important part is converting the extracted content into the correct business hierarchy, ensuring that:
Document fields → Document
Invoice fields → Invoice
Item fields → Item
The parser was refined to prevent cross-level field injection, normalize numeric values, handle supplier information, and preserve the intended invoice-to-item relationship.
32. Conclusion¶
The Courier Bill of Entry PDF-to-JSON Converter Wizard provides a structured Frappe-based solution for automating Courier BOE document processing.
The implementation combines:
- A guided three-step wizard.
- PDF upload and validation.
- Reusable PDF analysis.
- Courier-specific parsing.
- Document-level extraction.
- Supplier information handling.
- Invoice-level extraction.
- Item-level extraction.
- Numeric and monetary normalization.
- Invoice/item hierarchy validation.
- Structured JSON generation.
- JSON copy and download functionality.
The architecture deliberately reuses the common PDF analysis layer while keeping Courier-specific parsing separate from the Shipping Bill implementation.
The resulting solution provides a foundation for converting Courier Bill of Entry PDFs into reliable, structured JSON that can be consumed by downstream Frappe processes, APIs, integrations, and business automation workflows.
Files