Feature #91
Updated by Rajesh Gotur 10 days ago
from pathlib import Path readme = r'''# Shipping Bill PDF Extractor Wizard > **Frappe-based document extraction wizard for converting Shipping Bill PDFs into structured, application-ready JSON.** --- ## 📌 Project Overview The **Shipping Shipping Bill PDF Extractor Wizard** Wizard is a Frappe-based document-processing application solution developed to reduce automate the manual effort involved in reading extraction of Shipping Bill PDFs and entering their information into from uploaded PDF documents. The wizard provides a guided workflow in which the application. A user uploads a Shipping Bill PDF through the wizard. The system then PDF, analyzes the PDF, document, extracts the available document information, identifies the required Shipping Bill relevant fields, normalizes the extracted values, and generates a produces structured Shipping Bill JSON response. for further processing within the application. The most important design requirement implementation is that the extractor should work with **different designed to support different Shipping Bill PDFs containing and different values**. The extraction logic should therefore identify document values from the **document structure, labels, layout, text positions, and coordinates**, rather than depending relying on fixed hardcoded values from one sample a single PDF. --- Objective ## 🎯 Objectives The project was developed with primary objectives of the following objectives: Shipping Bill PDF Extractor Wizard are: - Automate Shipping Bill information data extraction from PDF documents. - Reduce manual data entry and document reading. - entry. Provide a simple guided wizard for users. - Analyze multi-page Shipping Bill PDFs. - Dynamically identify Shipping Bill fields. - Associate labels with their corresponding values. - wizard-based user experience. Extract fields dynamically from the uploaded document. Use document layout labels, layout, text positioning, and coordinates where required. - to identify values. Normalize extracted values into application-friendly formats. - the required application format. Generate structured Shipping Bill JSON. - Provide extraction information useful for validation that can be reviewed and debugging. - debugged. Create a reusable foundation that can process for processing different Shipping Bill PDFs. --- # 🏢 Business Problem Shipping Bills contain a large amount of structured information, but that Bill information is stored available inside PDF documents. documents and contains multiple fields that need to be captured before the information can be used in the application. In a manual process, a user would typically: Manual extraction requires users to: 1. Open the Shipping Bill PDF. 2. Search for each Locate the required field. 3. fields. Read the value associated with that field. 4. corresponding values. Enter the value values manually. Repeat the process for different documents. Convert the collected information into a structured format. This process is repetitive and can result in data-entry errors. The objective of the application. wizard is to automate this process and provide structured output from the uploaded PDF. Proposed Solution The solution introduces a dedicated Shipping Bill PDF Extractor Wizard. The overall workflow is: Upload Shipping Bill PDF │ ▼ 5. Repeat Validate PDF │ ▼ Analyze PDF │ ▼ Extract Text / Layout Information │ ▼ Identify Shipping Bill Fields │ ▼ Parse and Normalize Values │ ▼ Generate Shipping Bill JSON │ ▼ Review Extracted Output Wizard Workflow Step 1 – Upload Shipping Bill PDF The wizard provides an interface for the user to upload a Shipping Bill PDF. The uploaded document becomes the input for the extraction pipeline. The extraction process is not intended to depend on a fixed Shipping Bill number, port code, invoice number, or other hardcoded document values. Step 2 – PDF Validation Before processing begins, the uploaded document is validated to ensure that the required PDF input is available. The API layer handles the initial file-processing flow and passes the document into the PDF analysis pipeline. Step 3 – PDF Analysis The PDF Analyzer processes the uploaded document and identifies the available document structure and content. The analyzer examines information such as: Number of pages. Page-level information. Extractable text. Text-based pages. Image-based pages. Document content required for every parsing. The analyzer was tested against a multi-page Shipping Bill. 6. Prepare Bill PDF and successfully identified the available text pages. Step 4 – Shipping Bill Parsing After the PDF is analyzed, the document information in is passed to the Shipping Bill Parser. The parser identifies the required structured format. Shipping Bill fields from the extracted document information. The parsing process considers: Field labels. Nearby values. Text positions. Document layout. Coordinates. Page information. Relationships between labels and values. This becomes inefficient when multiple approach is intended to support different Shipping Bills need to Bill documents instead of depending on fixed values. Dynamic Field Extraction A key requirement of the implementation is dynamic field extraction. The parser should not use fixed values such as: Shipping Bill Number = 5632009 Port Code = INBOM4 Gross Weight = 1143 Instead, these values must be processed. detected from the uploaded PDF. ### Problems For example, the target output can contain fields such as: Port Code Shipping Bill Number Shipping Bill Date IEC CB Code Package Count Gross Weight Weight Unit Destination Invoice Information Item Information The actual values depend on the uploaded Shipping Bill. Label and Coordinate-Based Extraction Shipping Bills may contain multiple fields with similar formats or values. Therefore, simple text searching is not sufficient for reliable extraction. The parser uses the manual approach relationship between: - Repetitive data entry. Labels Values Coordinates Text position Page layout Nearby document elements For example: Label Extracted Value - Increased processing time. --------------------------------------- - Higher possibility of human error. Port Code INBOM4 - Difficult to maintain consistency. Shipping Bill No 5632009 - Time spent searching through multi-page documents. Shipping Bill Date 2026-08-04 - Additional effort required IEC 0300014953 CB Code AAACM6824HCH032 Packages 2 Gross Weight 1143 KGS The objective is to convert extracted information determine which value belongs to which field based on the document structure. Data Normalization After raw values are extracted, the parser normalizes them into JSON or the expected application format. Normalization can include handling: Dates. Numeric values. Weight values. Package counts. Text fields. The Empty values. Field formatting. This ensures that the extracted information has a consistent structure before being returned to the application. Shipping Bill Extractor Wizard was created to automate this document-to-data conversion. JSON Generation --- The final extraction result is converted into structured JSON. # 💡 Solution A representative structure is: { "shipping_bill_json": { "doctype": "Shipping Bill", "port_code": "...", "gwt": 0, "inv": 0, "shipping_bill_no": "...", "port_of_loading": "...", "shipping_bill_date": "...", "iec": "...", "cb_code": "...", "packages": 0 } } The solution introduces a guided wizard that acts as actual values are populated dynamically from the entry point for uploaded PDF. Extraction Trace The processing flow also provides extraction trace information. The trace is useful for: Debugging extraction results. Reviewing field detection. Understanding label/value relationships. Identifying incorrect mappings. Improving parser accuracy. The final API response is focused on the required structured Shipping Bill output rather than unnecessary raw page information. Application Architecture The implementation separates the user interface, API layer, PDF processing. analysis, and Shipping Bill parsing responsibilities. ```text Shipping Bill Wizard UI │ ▼ API Layer │ ▼ PDF Validation │ ▼ PDF Analyzer │ ▼ Shipping Bill Parser │ ┌───┴────┐ ▼ ▼ Normalization Extraction Trace │ ▼ Shipping Bill JSON This separation allows the extraction logic to be improved independently from the wizard interface. Project Components The current implementation includes the following core project files: shipping_bill_extrac.js shipping_bill_extrac.css shipping_bill_extrac.json pdf_analyzer.py shipping_bill_parser.py These files cover the wizard interface, page configuration, styling, PDF analysis, and Shipping Bill parsing functionality. Wizard User Interface The wizard frontend is implemented using: JavaScript HTML CSS Frappe Page API The interface provides the user with the required document-processing workflow. The UI is responsible for: Accepting the Shipping Bill PDF. Starting document analysis. Displaying processing results. Presenting extracted information. Providing the generated structured output. API Processing Pipeline The backend processing sequence is: PDF Upload in Wizard │ ▼ PDF File Validation │ ▼ PDF Analysis Analyzer │ ▼ Text / Layout Information Shipping Bill Parser │ ▼ Value Normalization │ ▼ Shipping Bill Field Detection JSON │ ▼ Value Extraction Trace The API coordinates the processing stages while keeping the extraction logic inside the dedicated processing components. Error Handling and Validation Validation is performed throughout the processing pipeline. The implementation considers: Uploaded file availability. PDF validity. PDF structure. Extractable content. Required Shipping Bill fields. Parsed values. Data normalization. Extraction trace information helps identify cases where a label is incorrectly associated with a neighboring value. Testing The Shipping Bill extractor was tested using an actual multi-page Shipping Bill PDF. The PDF analysis successfully identified: Total Pages: 6 Text Pages: 6 Image Pages: 0 The parser was also tested against Shipping Bill information including: Port Code: INBOM4 Shipping Bill Number: 5632009 Shipping Bill Date: 2026-08-04 IEC: 0300014953 CB Code: AAACM6824HCH032 Packages: 2 Gross Weight: 1143 KGS Testing also highlighted an important extraction challenge: header labels can sometimes be associated with neighboring labels or values. This reinforces the requirement for accurate label, layout, and coordinate-based extraction rather than simple text matching. Key Technical Requirement The extractor must not be hardcoded for a single Shipping Bill. The expected behavior is: Shipping Bill A │ ▼ Normalization Different Document Values │ ▼ Same Extraction Logic and: Shipping Bill B │ ▼ Different Document Values │ ▼ Same Extraction Logic The parser should dynamically identify the required values based on the structure and layout of the uploaded document. Benefits The Shipping Bill PDF Extractor Wizard provides: Automated PDF data extraction. Reduced manual data entry. Structured Shipping Bill output. Guided wizard workflow. Reusable extraction architecture. Separation between UI and backend processing. Dynamic field extraction. Label/layout/coordinate-based processing. Normalized extracted values. Extraction trace for debugging. Support for multi-page Shipping Bills. Future Improvements Potential improvements to the extraction engine include: More accurate label-to-value matching. Improved coordinate-based field boundaries. Better neighboring-label detection. Multi-column layout handling. Improved table extraction. Repeated-field handling. Confidence scoring for extracted fields. Validation against expected field formats. Support for additional Shipping Bill layouts. The main objective of future improvements is to increase extraction accuracy while continuing to support different Shipping Bill PDFs without hardcoded document values. Conclusion The Shipping Bill PDF Extractor Wizard provides a Frappe-based solution for converting Shipping Bill PDFs into structured application-ready data. The implementation combines a guided wizard interface with a backend processing pipeline consisting of PDF validation, PDF analysis, Shipping Bill parsing, value normalization, structured JSON Output │ ▼ Review / Downstream Use generation, and extraction tracing. The core design requirement is to dynamically identify Shipping Bill fields using document labels, layout, text positioning, and coordinates rather than relying on hardcoded values. This provides the foundation for automating Shipping Bill data capture and reducing the manual effort required to enter Shipping Bill information into the application.