Actions

Copy link

Feature #91

open

RG RG

Shipping Bill PDF Extractor Wizard

Feature #91: Shipping Bill PDF Extractor Wizard

Added by Rajesh Gotur 10 days ago. Updated 9 days ago.

Status:

New

Priority:

Normal

Assignee:

Rajesh Gotur

Target version:

Backlogs

Start date:

08/17/2026

Due date:

% Done:

90%

Estimated time:

72:00 h

Spent time:

19:00 h

Description

Shipping Bill PDF Extractor Wizard¶

Frappe-based document extraction wizard for converting Shipping Bill PDFs into structured, application-ready JSON.

📌 Project Overview¶

The Shipping Bill PDF Extractor Wizard is a Frappe-based document-processing application developed to reduce the manual effort involved in reading Shipping Bill PDFs and entering their information into the application.

A user uploads a Shipping Bill PDF through the wizard. The system then analyzes the PDF, extracts the available document information, identifies the required Shipping Bill fields, normalizes the extracted values, and generates a structured JSON response.

The most important design requirement is that the extractor should work with different Shipping Bill PDFs containing different values. The extraction logic should therefore identify values from the document structure, labels, layout, text positions, and coordinates, rather than depending on fixed values from one sample PDF.

🎯 Objectives¶

The project was developed with the following objectives:

Automate Shipping Bill information extraction from PDF documents.
Reduce manual data entry and document reading.
Provide a simple guided wizard for users.
Analyze multi-page Shipping Bill PDFs.
Dynamically identify Shipping Bill fields.
Associate labels with their corresponding values.
Use document layout and coordinates where required.
Normalize extracted values into application-friendly formats.
Generate structured Shipping Bill JSON.
Provide extraction information useful for validation and debugging.
Create a reusable foundation that can process different Shipping Bill PDFs.

🏢 Business Problem¶

Shipping Bills contain a large amount of structured information, but that information is stored inside PDF documents.

In a manual process, a user would typically:

Open the Shipping Bill PDF.
Search for each required field.
Read the value associated with that field.
Enter the value into the application.
Repeat the process for every Shipping Bill.
Prepare the information in the required structured format.

This becomes inefficient when multiple Shipping Bills need to be processed.

Problems with the manual approach¶

Repetitive data entry.
Increased processing time.
Higher possibility of human error.
Difficult to maintain consistency.
Time spent searching through multi-page documents.
Additional effort required to convert extracted information into JSON or application fields.

The Shipping Bill Extractor Wizard was created to automate this document-to-data conversion.

💡 Solution¶

The solution introduces a guided wizard that acts as the entry point for Shipping Bill PDF processing.

                    Shipping Bill PDF
                           │
                           ▼
                    Upload in Wizard
                           │
                           ▼
                     PDF Validation
                           │
                           ▼
                      PDF Analysis
                           │
                           ▼
               Text / Layout Information
                           │
                           ▼
                Shipping Bill Field Detection
                           │
                           ▼
                  Value Extraction
                           │
                           ▼
                     Normalization
                           │
                           ▼
                Structured JSON Output
                           │
                           ▼
                 Review / Downstream Use

The wizard separates the user interaction from the document-processing logic, making the solution easier to maintain and improve.

🧭 Wizard Workflow¶

Step 1 — Upload Shipping Bill PDF¶

The user uploads a Shipping Bill PDF through the Frappe wizard.

The uploaded document becomes the input to the extraction pipeline.

The wizard is designed around the document rather than a specific sample file.

For example, the system should not assume that every document contains:

Shipping Bill No = 5632009
Port Code = INBOM4
Gross Weight = 1143

Those are values from one document and must be discovered dynamically from the uploaded PDF.

Step 2 — Validate the PDF¶

Before extraction begins, the input document is validated.

The processing layer checks that:

A file has been provided.
The document is available for processing.
The input is a PDF.
The PDF can be passed into the analysis pipeline.

The API layer controls the processing flow and passes the validated document to the PDF analyzer.

🔍 PDF Analysis¶

The PDF Analyzer is responsible for inspecting the uploaded PDF before the Shipping Bill parser attempts to extract fields.

The analysis provides information such as:

Total number of pages.
Page-level information.
Extractable text.
Text-based pages.
Image-based pages.
Content required by the parser.

Example test result¶

The extractor was tested using a multi-page Shipping Bill PDF:

Total Pages : 6
Text Pages  : 6
Image Pages : 0

This confirms that the tested document contained extractable text across all six pages.

🧩 Shipping Bill Parsing¶

Once the PDF has been analyzed, the extracted document information is passed to the Shipping Bill Parser.

The parser is responsible for identifying the required Shipping Bill fields and their corresponding values.

The parser considers relationships between:

Field labels.
Extracted values.
Text positions.
Coordinates.
Page information.
Document layout.
Nearby document elements.

The purpose is to determine which value belongs to which field, rather than simply searching the PDF for a value that happens to look correct.

🧠 Dynamic Field Extraction¶

Dynamic extraction is one of the most important requirements of this project.

A hardcoded implementation would look like:

Shipping Bill Number → 5632009
Port Code            → INBOM4
Gross Weight         → 1143

This approach is not acceptable for the final extractor because the next Shipping Bill can contain completely different values.

Instead, the system should work like:

Shipping Bill A
      │
      ▼
Read document structure
      │
      ▼
Identify labels
      │
      ▼
Locate corresponding values
      │
      ▼
Generate structured output

and:

Shipping Bill B
      │
      ▼
Read document structure
      │
      ▼
Identify labels
      │
      ▼
Locate corresponding values
      │
      ▼
Generate structured output

The extraction logic remains the same while the document values change.

📐 Label, Layout & Coordinate-Based Extraction¶

Shipping Bills can contain multiple values with similar formats.

For example, several numbers may appear on the same page. A simple search for a number is therefore not enough to determine which field it belongs to.

The parser uses the relationship between:

Label
  +
Position
  +
Coordinates
  +
Nearby Value
  +
Page Layout

to determine the correct field/value relationship.

Example¶

Field                  Extracted Value
------------------------------------------------
Port Code              INBOM4
Shipping Bill No       5632009
Shipping Bill Date     2026-08-04
IEC                    0300014953
CB Code                AAACM6824HCH032
Packages               2
Gross Weight           1143 KGS

The values shown above represent data extracted during testing. They are not hardcoded extraction values.

📋 Target Shipping Bill Information¶

The parser is intended to extract the fields required by the Shipping Bill structure.

Examples include:

Header Information¶

Port Code
Shipping Bill Number
Shipping Bill Date
IEC
CB Code
Port of Loading
Destination
Package Count
Gross Weight
Weight Unit

Transaction / Invoice Information¶

Invoice information
Invoice-related references
Item information

Item Information¶

Product/item details
Quantities
Relevant Shipping Bill item fields

The exact values depend on the uploaded document.

🔄 Data Normalization¶

Raw PDF text is not always immediately suitable for application use.

After extraction, the parser normalizes the values into the expected application format.

Normalization can include:

Date formatting.
Numeric conversion.
Weight handling.
Package count conversion.
Text cleanup.
Empty-value handling.
Field formatting.

For example:

Raw PDF Value
      │
      ▼
Extracted Value
      │
      ▼
Normalized Value
      │
      ▼
Application JSON

This ensures that downstream processing receives predictable data types and formats.

📦 Shipping Bill JSON Generation¶

After parsing and normalization, the extracted information is converted into structured JSON.

A representative output structure is:

{
    "shipping_bill_json": {
        "doctype": "Shipping Bill",
        "port_code": "...",
        "gwt": 0,
        "inv": 0,
        "shipping_bill_no": "...",
        "port_of_loading": "...",
        "shipping_bill_date": "...",
        "iec": "...",
        "cb_code": "...",
        "packages": 0
    }
}

The actual values are dynamically populated from the uploaded Shipping Bill PDF.

The JSON structure provides a consistent interface between the PDF extraction layer and downstream application processing.

🔎 Extraction Trace¶

The extraction process also maintains extraction trace information.

The trace is useful when reviewing how a field was detected.

It can help with:

Debugging incorrect extraction.
Reviewing field detection.
Understanding label/value relationships.
Identifying incorrect neighboring values.
Improving parser rules.
Testing new Shipping Bill layouts.

This is particularly important because Shipping Bill PDFs may contain complex layouts where a nearby value can otherwise be incorrectly associated with a label.

🏗️ Application Architecture¶

The implementation separates the major responsibilities into different layers.

┌─────────────────────────────────────┐
│          Shipping Bill Wizard       │
│        JavaScript / HTML / CSS      │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│              API Layer              │
│      File handling / orchestration  │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│           PDF Validation            │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│            PDF Analyzer             │
│     Pages / Text / PDF Structure    │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│       Shipping Bill Parser          │
│ Labels / Values / Layout / Coords   │
└──────────────────┬──────────────────┘
                   │
          ┌────────┴────────┐
          ▼                 ▼
┌─────────────────┐  ┌─────────────────┐
│  Normalization  │  │ Extraction Trace│
└────────┬────────┘  └─────────────────┘
         │
         ▼
┌─────────────────────────────────────┐
│       Shipping Bill JSON            │
└─────────────────────────────────────┘

Architectural Principle¶

The UI does not contain the complete extraction logic.

Instead:

The Wizard handles user interaction.
The API coordinates processing.
The PDF Analyzer understands the PDF structure.
The Shipping Bill Parser extracts fields.
Normalization prepares the values.
JSON generation produces application-ready output.

This separation allows each component to be improved independently.

📁 Project Structure¶

The current implementation includes the following core files:

shipping_bill_extractor/
│
├── shipping_bill_extrac.js
├── shipping_bill_extrac.css
├── shipping_bill_extrac.json
│
├── pdf_analyzer.py
└── shipping_bill_parser.py

File Responsibilities¶

File	Responsibility
`shipping_bill_extrac.js`	Wizard UI and frontend interaction
`shipping_bill_extrac.css`	Wizard styling and layout
`shipping_bill_extrac.json`	Frappe page configuration
`pdf_analyzer.py`	PDF analysis and document-level information
`shipping_bill_parser.py`	Shipping Bill field extraction and parsing

The current project baseline treats these files as the core implementation components.

🖥️ Wizard User Interface¶

The frontend is implemented using:

JavaScript
HTML
CSS
Frappe Page API

The UI is responsible for providing a simple workflow for the user.

Main responsibilities¶

Display the wizard.
Accept the Shipping Bill PDF.
Start document analysis.
Display processing results.
Present extracted information.
Display structured output for further use.

The intention is to keep the user experience simple while the backend performs the complex document-processing work.

🔌 API Processing Pipeline¶

The backend processing flow can be summarized as:

PDF Upload
    │
    ▼
File Validation
    │
    ▼
PDF Analyzer
    │
    ▼
Shipping Bill Parser
    │
    ▼
Value Normalization
    │
    ▼
Shipping Bill JSON
    │
    ▼
Extraction Trace

The API coordinates these stages without placing all processing responsibilities into a single function or the frontend.

🛡️ Validation & Error Handling¶

Validation is performed throughout the processing pipeline.

The implementation considers:

File availability.
PDF validity.
PDF structure.
Extractable content.
Required Shipping Bill fields.
Parsed values.
Normalized values.
Label/value relationships.

The extraction trace provides additional information when the parser produces an unexpected mapping.

🧪 Testing¶

The extractor was tested using an actual multi-page Shipping Bill PDF.

PDF Analysis Result¶

Total Pages : 6
Text Pages  : 6
Image Pages : 0

Extracted Test Values¶

The parser was tested against values including:

Port Code           : INBOM4
Shipping Bill No    : 5632009
Shipping Bill Date  : 2026-08-04
IEC                 : 0300014953
CB Code             : AAACM6824HCH032
Packages            : 2
Gross Weight        : 1143 KGS

These values were used to verify the extraction pipeline against the sample document.

They should not be treated as constants in the parser.

⚠️ Extraction Challenge Identified During Testing¶

Testing revealed an important challenge with Shipping Bill extraction.

Some labels can appear close to other labels or values on the same page. If extraction is based only on raw text matching, the parser may associate a field with the wrong neighboring value.

For example:

Label A     Value A     Label B     Value B

A basic text-search implementation may incorrectly associate:

Label A → Value B

instead of:

Label A → Value A

This is why the project requires layout- and coordinate-aware extraction.

The parser must understand not only what text exists, but also where that text exists in relation to other document elements.

🚫 Avoiding Hardcoded Document Values¶

One of the core project rules is:

Do not hardcode values from the sample Shipping Bill into the extraction logic.

For example, the parser should never contain business values such as:

shipping_bill_no = "5632009"
port_code = "INBOM4"
gross_weight = 1143

Instead, the parser should discover these values from the uploaded document.

Correct Approach¶

Uploaded PDF
     │
     ▼
Identify Label
     │
     ▼
Find Related Value
     │
     ▼
Validate Value
     │
     ▼
Normalize Value
     │
     ▼
Return Field

This makes the implementation reusable across different Shipping Bills.

🔁 Reusability Requirement¶

The same extraction pipeline should be capable of processing:

Shipping Bill A
      │
      ├── Different Port
      ├── Different Number
      ├── Different Date
      ├── Different Weight
      └── Different Items
             │
             ▼
      Same Extraction Engine

and:

Shipping Bill B
      │
      ├── Different Port
      ├── Different Number
      ├── Different Date
      ├── Different Weight
      └── Different Items
             │
             ▼
      Same Extraction Engine

The document values change; the extraction logic remains reusable.

📊 Current Processing Capabilities¶

The current implementation establishes the following processing capabilities:

Shipping Bill PDF upload.
PDF validation.
Multi-page PDF analysis.
Text-page identification.
Shipping Bill parsing.
Dynamic field extraction approach.
Label/value relationship handling.
Layout and coordinate-aware extraction approach.
Value normalization.
Structured JSON generation.
Extraction trace generation.
Frappe wizard-based user interface.

🔮 Future Improvements¶

The extraction engine can be further enhanced to improve reliability across more Shipping Bill layouts.

Potential improvements include:

1. Improved Label-to-Value Matching¶

Strengthen the logic used to determine which value belongs to each label.

2. Coordinate-Based Field Boundaries¶

Define field regions using PDF coordinates instead of relying only on nearby text.

3. Multi-Column Layout Handling¶

Improve extraction for documents where multiple fields are arranged horizontally or across columns.

4. Table Extraction¶

Improve extraction of invoice and item-level tabular information.

5. Repeated Field Handling¶

Handle fields that appear multiple times across different pages or sections.

6. Confidence Scoring¶

Assign confidence levels to extracted fields so uncertain values can be reviewed.

7. Field Validation¶

Validate extracted values against expected formats such as:

Dates
Numeric values
Package counts
Weight values
Shipping Bill numbers
Codes

8. Additional Shipping Bill Layout Support¶

Extend the parser to handle additional document layouts while keeping the same extraction architecture.

📈 Benefits¶

The Shipping Bill PDF Extractor Wizard provides several operational benefits:

Reduced manual effort — Users do not need to manually read and enter every field.
Faster processing — PDF information is processed automatically.
Improved consistency — Extracted values are normalized into a standard structure.
Reusable extraction logic — The parser is designed for different Shipping Bills.
Better traceability — Extraction trace information supports debugging and validation.
Scalable architecture — UI, API, PDF analysis, and parsing are separated.
Application-ready output — Extracted information is returned as structured JSON.
Better user experience — Users interact through a guided Frappe wizard.

🛠️ Technology Stack¶

Technology	Purpose
Frappe Framework	Application and wizard platform
Python	PDF analysis and document parsing
JavaScript	Wizard frontend and user interaction
HTML	Wizard structure
CSS	Wizard styling
PDF Processing	Text and document structure extraction
JSON	Structured Shipping Bill output

🔐 Design Principles¶

The project follows these core principles:

1. Document-Driven Extraction¶

The uploaded PDF is the source of truth for extracted values.

2. No Sample-Value Hardcoding¶

Values from a test document must not become constants in the parser.

3. Separation of Responsibilities¶

UI, API, PDF analysis, parsing, and normalization are handled separately.

4. Reusable Processing¶

The extraction engine should work across different Shipping Bill documents.

5. Structured Output¶

The final result must be predictable and suitable for downstream application processing.

6. Debuggable Extraction¶

Extraction trace information should make incorrect field mappings easier to investigate.

📌 End-to-End Architecture¶

                     ┌─────────────────────┐
                     │   Shipping Bill PDF  │
                     └──────────┬──────────┘
                                │
                                ▼
                     ┌─────────────────────┐
                     │   Wizard Interface  │
                     └──────────┬──────────┘
                                │
                                ▼
                     ┌─────────────────────┐
                     │      API Layer      │
                     └──────────┬──────────┘
                                │
                                ▼
                     ┌─────────────────────┐
                     │    PDF Analyzer     │
                     └──────────┬──────────┘
                                │
                                ▼
                     ┌─────────────────────┐
                     │ Shipping Bill Parser│
                     └──────────┬──────────┘
                                │
                    ┌───────────┴───────────┐
                    │                       │
                    ▼                       ▼
          ┌──────────────────┐   ┌──────────────────┐
          │   Normalization  │   │ Extraction Trace │
          └────────┬─────────┘   └──────────────────┘
                   │
                   ▼
          ┌─────────────────────┐
          │ Shipping Bill JSON  │
          └──────────┬──────────┘
                     │
                     ▼
          ┌─────────────────────┐
          │ Downstream Processing│
          └─────────────────────┘

🚀 Project Outcome¶

The project establishes a reusable Frappe-based framework for converting Shipping Bill PDF documents into structured application data.

The implementation moves the process from:

Manual PDF Reading
       ↓
Manual Data Entry
       ↓
Manual Formatting

to:

PDF Upload
     ↓
Automatic Analysis
     ↓
Dynamic Extraction
     ↓
Normalization
     ↓
Structured JSON

The key achievement is not simply extracting values from one Shipping Bill. The core objective is to build an extraction engine that can understand the document structure and dynamically extract the correct values from different Shipping Bill PDFs.

📄 Conclusion¶

The Shipping Bill PDF Extractor Wizard provides the foundation for automating Shipping Bill document processing within Frappe.

It combines a guided user interface with a backend processing pipeline consisting of:

PDF validation
PDF analysis
Text and layout processing
Shipping Bill field detection
Dynamic value extraction
Label/value relationship handling
Coordinate and layout awareness
Data normalization
Structured JSON generation
Extraction tracing

The architecture is designed to be reusable and maintainable, with the most important requirement being that the system should extract values dynamically from the uploaded Shipping Bill rather than relying on hardcoded values from a sample document.

This provides a strong foundation for integrating Shipping Bill PDF extraction into larger Frappe-based business automation workflows.

Files

Download all files

Frappe Shipping Bill PDF Extraction to JSON — Master Development Prompt.md (35.6 KB) Frappe Shipping Bill PDF Extraction to JSON — Master Development Prompt.md	Prompt Used for Creating a Wizard	Rajesh Gotur, 08/19/2026 10:34 AM
shipping_bill_parser_v44.py (152 KB) shipping_bill_parser_v44.py	Main Code to convert PDF to Json	Rajesh Gotur, 08/19/2026 10:35 AM
shipping_bill_parser_dynamic_schema_complete_v11.py (223 KB) shipping_bill_parser_dynamic_schema_complete_v11.py	Robust and dynamic version of final code	Rajesh Gotur, 08/20/2026 11:03 AM

RG Updated by Rajesh Gotur 10 days ago Actions
Copy link
#1

Description updated (diff)

RG Updated by Rajesh Gotur 10 days ago Actions
Copy link
#2

Description updated (diff)

RG Updated by Rajesh Gotur 9 days ago Actions
Copy link
#3

File shipping_bill_parser_dynamic_schema_complete_v11.py shipping_bill_parser_dynamic_schema_complete_v11.py added