# Frappe Shipping Bill PDF → Structured JSON Extraction System You are acting as a: - Senior Frappe Framework Solution Architect - Senior Frappe/Python Developer - PDF Processing Engineer - OCR/PDF Extraction Engineer - Document Processing Architect I am building a **PDF-to-JSON extraction system in the Frappe Framework**. The immediate target is specifically a **Shipping Bill PDF** such as the sample PDF provided with this project. The system must eventually be generic enough to support other document types, but **V1 must focus on Shipping Bill extraction and produce JSON in the exact structure required by my Frappe Shipping Bill DocType and its child tables.** --- # 1. ACTUAL PRODUCT WE ARE BUILDING We are NOT building a simple PDF text extractor. We are building: ```text Shipping Bill PDF | v PDF Analysis | v Detect PDF Type | +----------------------+ | | v v Text-based PDF Image/Scanned PDF | | v v PDF Text Parser OCR | | +----------+-----------+ | v Process ALL Pages | v Shipping Bill Field Extraction | v Rule / Pattern Engine | v Structured Data | v Validation | v User Review | v APPROVE | v Final JSON | +-----+------+ | | v v Copy JSON Download JSON ``` Future phase: ```text Final JSON | v Frappe Mapping Engine | v Shipping Bill DocType | +---- Manifest Detail | +---- Invoice Detail | +---- Item Detail | +---- Single Window Declaration | +---- Other Child Tables ``` --- # 2. VERY IMPORTANT — V1 DOES NOT REQUIRE PAID AI Do NOT use: - OpenAI API - Gemini API - Claude API - Azure Document Intelligence - AWS Textract - Any paid cloud OCR/API - Any paid external document-processing API The first implementation must work using: ```text Python Frappe PDF processing libraries OCR if required Regex / pattern matching Table extraction JSON processing ``` Possible open-source libraries: ```text PyMuPDF pdfplumber pypdf PaddleOCR Tesseract Docling Camelot Tabula Pydantic JSON Schema ``` Do not install all of these blindly. Select only what is actually necessary. Before adding a dependency, explain: 1. Why it is needed. 2. What problem it solves. 3. Whether an existing dependency can already solve the problem. The architecture should allow a local AI/LLM fallback in the future, but **AI is not part of the V1 dependency.** --- # 3. IMPORTANT: USE THE PROVIDED SHIPPING BILL PDF AS THE PRIMARY TEST DOCUMENT The project includes a sample Shipping Bill PDF. Use it as the primary development/test document. The PDF contains **6 pages**. The system must process: ```text Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 ``` Do NOT process only the first page. Do NOT assume that the document ends after the first page. The sample contains: ```text Page 1 PART - I - SHIPPING BILL SUMMARY ``` ```text Page 2 PART - II - INVOICE DETAILS ``` ```text Page 3 PART - III - ITEM DETAILS ``` ```text Page 4 PART - IV - EXPORT SCHEME DETAILS ``` ```text Page 5 PART - IV - EXPORT SCHEME DETAILS Other Additional Information / Reexport Details ``` ```text Page 6 PART - V - DECLARATIONS ``` The source PDF confirms the six-page structure. --- # 4. FRAPPE PAGE UI Create a custom Frappe page. The page must behave as a 3-step wizard. ```text STEP 1 Upload PDF ↓ STEP 2 Analyse + Review ↓ STEP 3 JSON Output ``` The user must be able to clearly see which step they are currently on. --- # 5. STEP 1 — UPLOAD PDF When the user opens the page: ```text +--------------------------------------------------+ | SHIPPING BILL PDF EXTRACTOR | | | | Upload Shipping Bill PDF | | | | [ Choose PDF ] | | | | Selected File: None | | | | [ NEXT ] | +--------------------------------------------------+ ``` Requirements: - Allow PDF upload. - Accept PDF only. - Display file name. - Validate file type. - Do not start extraction immediately. - Start processing when the user clicks `NEXT`. After selecting: ```text Selected File: 563200904082026INBOM4SB22050820261955 (1).pdf [ NEXT ] ``` --- # 6. STEP 2 — ANALYSE THE PDF When the user clicks `NEXT`: The backend must analyze the uploaded PDF. First determine: ```text PDF Type ``` Possible values: ```text Text-based PDF Image-based / Scanned PDF Mixed PDF ``` For the determination, inspect every page. Example: ```text Page 1 → Text Page 2 → Text Page 3 → Text Page 4 → Text Page 5 → Text Page 6 → Text ``` The overall document can therefore be identified as: ```text Text-based PDF ``` if all pages contain meaningful extractable text. If some pages contain no meaningful text and require OCR: ```text Mixed PDF ``` If the entire document requires OCR: ```text Image-based / Scanned PDF ``` --- # 7. STEP 2 — SHOW PDF INFORMATION After processing: ```text PDF ANALYSIS File Name: 563200904082026INBOM4SB22050820261955 (1).pdf Document: Shipping Bill PDF Type: Text-based PDF Total Pages: 6 Pages Processed: 6 / 6 Extraction Status: Completed ``` Also show processing time. Example: ```text Processing Time: 40.91 seconds ``` The actual processing time must be generated by the application. Do NOT hardcode it. --- # 8. PROCESS EVERY PAGE The system must process the PDF page-by-page. Internally: ```text PDF | +--- Page 1 | +--- Page 2 | +--- Page 3 | +--- Page 4 | +--- Page 5 | +--- Page 6 ``` Each page must produce extracted information. The system must preserve: ```text page_number section field value ``` where possible. Example: ```json { "page": 1, "section": "Shipping Bill Summary", "field": "shipping_bill_no", "value": "5632009" } ``` --- # 9. DO NOT JUST RETURN RAW PDF TEXT This is extremely important. The following is NOT the desired result: ```text AIR CARGO COMPLEX... Port Code... SB No... SB Date... IEC... ... ``` Instead, convert the document into structured business information. For example: ```text SB No 5632009 ``` must become: ```json { "shipping_bill_no": "5632009" } ``` Similarly: ```text Port Code INBOM4 ``` becomes: ```json { "port_code": "INBOM4" } ``` --- # 10. SHIPPING BILL JSON IS THE TARGET The final JSON must follow the structure expected by the Frappe Shipping Bill DocType. The target top-level structure is: ```json { "shipping_bill_json": { "doctype": "Shipping Bill", "port_code": "", "gwt": 0, "inv": 0, "shipping_bill_no": "", "port_of_loading": "", "cntry_of_finaldstn": "", "state_of_origin": "", "port_of_finaldstn": "", "port_of_discharge": "", "cntry_of_discharge": "", "gwt_unit": "", "item": 0, "shipping_bill_date": "", "pkg": 0, "cont": 0, "iec": "", "iec_branch_code": "", "cb_code": "", "p_1a_mode": "", "p_1a_assess": "", "p_1a_dbk": "", "p_1a_re_exp": null, "p_1a_exam": "", "p_1a_rodtp": "", "p_1a_lut": "", "p_1a_job": "", "p_1a_licence": "", "p_1a_dfrc": "", "p_1b_exporter_name": "", "p_1b_exporter_address": "", "p_1b_cb_name": "", "p_1b_consignee_name": "", "p_1b_consignee_address": "", "p_1b_type": "", "p_1b_gstin": "", "p_1b_ad_code": "", "p_1b_forex_ac_no": "", "p_1c_fob_val": 0, "p_1c_com": 0, "p_1c_freight": 0, "p_1c_deductions": 0, "p_1c_insurance": 0, "p_1c_p_c": 0, "p_1c_discount": 0, "p_1d_dbk": 0, "p_1d_rodtep_amt": 0, "p_1d_rosctle_amt": 0, "p_1j_subm_dt": "", "p_1i_subm_time": "", "p_1i_leo_no": "", "p_1i_leo_dt": "", "p_1i_exmn_dt": "", "p_1i_exmn_time": "", "p_1i_leo_time": "", "p_1i_brc_realzn_dt": "", "manifest_details": [], "challan_details": [], "annex_details": [], "invoice_details": [], "aa__dfia_licence_details": [], "item_details": [], "drawback_and_rosl_claim": [], "job_details": [], "single_window_declaration": [], "single_window_declaration_constituents": [], "single_window_declaration_control": [], "supporting_documents": [], "ar4_details": [], "third_party_details": [], "item_manufacturerproducergrower_details": [], "rodtep_details": [], "container_details": [] }, "file_id": "", "processing_time": "" } ``` This is the **target structure**. The actual values must be extracted from the PDF. --- # 11. IMPORTANT — FIELD NAMES MUST MATCH THE TARGET Do not rename fields. For example: Use: ```text shipping_bill_no ``` NOT: ```text shipping_bill_number ``` Use: ```text p_1b_exporter_name ``` NOT: ```text exporter_name ``` Use: ```text p_1c_fob_val ``` NOT: ```text fob_value ``` The field names are intentionally aligned with the Frappe DocType. Preserve them exactly. --- # 12. SHIPPING BILL HEADER FIELDS The system must extract fields such as: ```text port_code gwt inv shipping_bill_no shipping_bill_date pkg cont iec iec_branch_code cb_code gwt_unit item ``` For the sample PDF, these values are visible in the header: ```text Port Code: INBOM4 SB No: 5632009 SB Date: 04-AUG-26 IEC/Br: 0300014953 CB Code: AAACM6824HCH032 G.WT: 1143 KGS ``` The source document shows these values on page 1. --- # 13. SHIPPING BILL STATUS FIELDS Extract: ```text p_1a_mode p_1a_assess p_1a_dbk p_1a_re_exp p_1a_exam p_1a_rodtp p_1a_lut p_1a_job p_1a_licence p_1a_dfrc ``` The values must be taken from the actual Shipping Bill table. Do not infer values from general knowledge. The source PDF contains the status fields in the Part-I Shipping Bill Summary. --- # 14. EXPORTER / CONSIGNEE / DECLARANT DETAILS Extract: ```text p_1b_exporter_name p_1b_exporter_address p_1b_cb_name p_1b_consignee_name p_1b_consignee_address p_1b_type p_1b_gstin p_1b_ad_code p_1b_forex_ac_no ``` Do not merge exporter and consignee data. Keep them in their respective fields. The sample PDF contains exporter information beginning on page 1 and consignee information in the declarant section. --- # 15. VALUE SUMMARY Extract: ```text p_1c_fob_val p_1c_com p_1c_freight p_1c_deductions p_1c_insurance p_1c_p_c p_1c_discount p_1d_dbk p_1d_rodtep_amt p_1d_rosctle_amt ``` Convert numeric values into appropriate JSON numbers. Example: ```json { "p_1c_fob_val": 18956370.0, "p_1c_com": 0.0 } ``` Do not return numeric values as strings unless the target field specifically requires a string. The sample's value summary appears in Part-I on page 1. --- # 16. PROCESS DETAILS Extract: ```text p_1j_subm_dt p_1i_subm_time p_1i_leo_no p_1i_leo_dt p_1i_exmn_dt p_1i_exmn_time p_1i_leo_time p_1i_brc_realzn_dt ``` The sample contains these events in the Process Details section on page 1. Normalize dates to: ```text YYYY-MM-DD ``` Normalize times to: ```text HH:MM:SS ``` For example: ```text 04-AUG-26 ``` becomes: ```text 2026-08-04 ``` --- # 17. CHILD TABLES The JSON must support Frappe child tables. This is critical. Do not flatten child-table data into the parent. For example: ```json "manifest_details": [ { "doctype": "Manifest Detail", ... } ] ``` This structure must be preserved. --- # 18. MANIFEST DETAILS Extract the E-Manifest information. Target structure: ```json "manifest_details": [ { "doctype": "Manifest Detail", "p_1e_mawb_no": "", "p_1e_cin_no": "", "p_1e_cin_dt": "", "p_1e_cin_site_id": "" } ] ``` The sample PDF contains: ```text MAWB NO CIN NO CIN DT CIN SITE ID ``` on page 1. If multiple manifest rows exist, create multiple objects. --- # 19. CHALLAN DETAILS Target: ```json "challan_details": [] ``` If rows exist, populate them. If the section is empty: ```json "challan_details": [] ``` Do not invent records. --- # 20. ANNEX DETAILS Target: ```json "annex_details": [ { "doctype": "Annex Detail", "p_1i_seal_typ": "", "p_1i_loose_pkts": 0, "p_1i_nature_cargo": "", "p_1i_marks_numbers": "", "p_1i_no_of_pkgs": 0, "p_1i_no_of_containers": 0 } ] ``` The sample contains Annex Details on page 1, including: ```text SEAL TYPE NATURE OF CARGO NO. OF PACKETS NO. OF CONTAINERS LOOSE PACKETS MARKS & NUMBERS ``` --- # 21. INVOICE DETAILS The Invoice Details section begins on page 2. Target: ```json "invoice_details": [ { "doctype": "Invoice Detail", "inv_sn": 1, "p_2a_inv_no": "", "p_2a_inv_dt": "", "p_2a_ad_code": "", "p_2a_invterm": "", "p_2b_exporter_name": "", "p_2b_buyer_addr": "", "p_2b_exporter_addr": "", "p_2b_buyer_name": "", "p_2c_invoice_value": 0, "p_2c_freight": 0, "p_2c_discount": 0, "p_2c_deduct": 0, "p_2c_exchng_rate_desc": "", "p_2c_invoice_curr": "", "p_2c_frieght_curr": "", "p_2c_discount_curr": "", "p_2c_deduct_curr": null, "p_2c_fob_val": 0, "p_2c_insurance": 0, "p_2c_commison": 0, "p_2c_p_c": 0, "p_2c_fob_curr": "", "p_2c_insurance_curr": "", "p_2c_commison_curr": null, "p_2c_p_c_curr": null } ] ``` The sample invoice information is visible on page 2, including invoice number/date, exporter, buyer, invoice value, FOB value, exchange rate and currency. --- # 22. ITEM DETAILS This is one of the most important sections. The sample Item Details are spread across pages 2 and 3. The system must understand that: ```text Page 2 Invoice item information begins ↓ Page 3 Part III Item Details contains the complete item information ``` Do not create duplicate item rows simply because the same item appears on multiple pages. The system must combine the information belonging to the same item. Target: ```json "item_details": [ { "p_3a_invsno": "", "p_3a_itemsn": "", "p_3a_cth": "", "p_3a_item_desc": "", "p_3a_qty": "", "p_3a_uqc": "", "p_3a_rate": "", "p_3a_value": "", "p_3a_fob": "", "p_3a_pmv": "", "p_3a_duty_amt": "", "p_3a_cess_rate": "", "p_3a_cess_amt": "", "p_3a_dbk_claimed": "", "p_3a_igststat": "", "p_3a_igst_val": "", "p_3a_igst_amt": "", "p_3a_schcod": "", "p_3a_scheme_desc": "", "p_3a_sqc_mst": "", "p_3a_sqc_uqc": "", "p_3a_state_of_origin": "", "p_3a_district_of_origin": "", "p_3a_pt_abroad": "", "p_3a_comp_cess": 0, "p_3a_end_use": "", "p_3a_benefit_availd": "", "p_3a_reward_benefit": "", "p_3a_third_party_item": "", "doctype": "Item Detail" } ] ``` The sample's Part III contains fields including HS code, description, quantity, UQC, rate, value, FOB, PMV, duty, IGST status, scheme, state/district of origin, end use, FTA benefit and third-party item. --- # 23. SINGLE WINDOW DECLARATION This is another child table. Target: ```json "single_window_declaration": [ { "p_4d_invsn": "", "p_4d_itmsn": "", "p_4d_info": "", "p_4d_qualifier": "", "p_4d_info_cd": "", "p_4d_info_text": "", "p_4d_info_msr": "", "p_4d_uqc": "", "doctype": "Single Window Declaration" } ] ``` The sample contains multiple rows: ```text CHR DTY / GCESS DTY / RDT ORC / DOO ORC / EPT ORC / STO ``` These must become separate JSON objects. The source table appears on page 4. --- # 24. OTHER CHILD TABLES Preserve the following target keys: ```text aa__dfia_licence_details drawback_and_rosl_claim job_details single_window_declaration_constituents single_window_declaration_control supporting_documents ar4_details third_party_details item_manufacturerproducergrower_details rodtep_details container_details ``` If the corresponding section is empty: ```json [] ``` Do not remove the key. Do not generate fake records. The sample PDF contains several empty sections in Part IV, so the final JSON should preserve those arrays as empty arrays. --- # 25. EMPTY VALUES Use appropriate empty values. For fields that are genuinely absent: ```json null ``` can be used where appropriate. For child tables: ```json [] ``` must be used when there are no records. Do not fabricate values. For example, if no container exists: ```json "container_details": [] ``` not: ```json "container_details": [ { "container": "UNKNOWN" } ] ``` --- # 26. DO NOT CONFUSE PDF HEADER REPEATS WITH NEW DATA The Shipping Bill repeats common header information on every page. For example: ```text Port Code SB No SB Date IEC CB Code GSTIN G.WT ``` appear repeatedly. Do NOT create duplicate parent fields because the same header is repeated on pages 2–6. The repeated header should be recognized as document-level metadata. --- # 27. DO NOT DUPLICATE CHILD RECORDS This is critical. For example, an item may appear on: ```text Page 2 Page 3 ``` The system must understand that it is the same item. Expected: ```json "item_details": [ { ... } ] ``` NOT: ```json "item_details": [ { ... }, { ... } ] ``` unless there are genuinely two different items. Use invoice serial number + item serial number or other available identifiers to correlate records. --- # 28. DATA NORMALIZATION Normalize values appropriately. ### Dates Convert: ```text 04-AUG-26 29/07/2026 05-AUG-26 ``` to: ```text 2026-08-04 2026-07-29 2026-08-05 ``` ### Times Convert: ```text 11:40 19:01 19:55 ``` to: ```text 11:40:00 19:01:00 19:55:00 ``` ### Numbers Where the target field is numeric: ```text 18,956,370 ``` should become: ```json 18956370.0 ``` Do not include commas in numeric JSON values. --- # 29. IMPORTANT: DO NOT CHANGE BUSINESS VALUES The PDF is the source of truth. Do not "correct" values using external assumptions. For example: If the PDF says: ```text INBOM4 ``` return: ```json "port_code": "INBOM4" ``` If the PDF says: ```text FOB ``` return: ```json "p_2a_invterm": "FOB" ``` Do not replace values because they appear unusual. If OCR produces an uncertain value, flag it rather than silently changing it. --- # 30. SOURCE TRACEABILITY Internally maintain source information wherever possible. For example: ```json { "field": "shipping_bill_no", "value": "5632009", "page": 1 } ``` For child records: ```json { "item_serial": "1", "page": 3 } ``` The user-facing final JSON does not necessarily need to include all provenance metadata unless required by the target DocType. However, the backend should preserve enough information for debugging. --- # 31. STEP 2 UI — ORGANIZED TABLE The extracted information must NOT be displayed as a huge JSON block in Step 2. Step 2 should be a human-review interface. Use organized sections. Example: ```text ================================================== PDF ANALYSIS ================================================== Document Type: Shipping Bill PDF Type: Text-based PDF Pages: 6 Processed: 6 / 6 Status: Completed ================================================== SHIPPING BILL SUMMARY ================================================== | Field | Value | Page | |-------|-------|------| | Port Code | INBOM4 | 1 | | SB No | 5632009 | 1 | | SB Date | 04-08-2026 | 1 | | IEC | 0300014953 | 1 | | CB Code | AAACM6824HCH032 | 1 | | G.WT | 1143 KGS | 1 | ================================================== EXPORTER / CONSIGNEE ================================================== | Field | Value | Page | |-------|-------|------| | Exporter | ... | 1 | | Consignee | ... | 1 | | CB Name | ... | 1 | ================================================== INVOICE DETAILS ================================================== | Field | Value | Page | |-------|-------|------| | Invoice No | ... | 2 | | Invoice Date | ... | 2 | | Invoice Value | ... | 2 | ================================================== ITEM DETAILS ================================================== | Item | HS Code | Description | Qty | UQC | Rate | Value | |------|---------|-------------|-----|-----|------|-------| ================================================== SINGLE WINDOW DECLARATION ================================================== | Info | Qualifier | Code | Text | Measure | UQC | |------|-----------|------|------|---------|-----| [ APPROVE ] ``` The exact UI can use Frappe's styling and components. --- # 32. STEP 2 SHOULD SHOW ALL EXTRACTED DATA Do not only display the fields that are convenient. The objective is to allow the user to review the data that will become JSON. The review screen should cover: ```text Parent fields + Child tables + Empty sections where relevant ``` For large tables, use collapsible sections or tabs. --- # 33. APPROVE BUTTON The button should be: ```text [ APPROVE & GENERATE JSON ] ``` When clicked: 1. Validate the extracted structure. 2. Confirm there are no fatal extraction errors. 3. Build final JSON. 4. Store the approved JSON. 5. Move to Step 3. Do not navigate to Step 3 before successful validation. --- # 34. STEP 3 — JSON OUTPUT After approval: ```text STEP 3 — JSON OUTPUT ``` Display the final JSON in a code editor / formatted code block. The JSON should look like: ```json { "shipping_bill_json": { "doctype": "Shipping Bill", "port_code": "INBOM4", "gwt": 1143.0, "inv": 1, "shipping_bill_no": "5632009", "port_of_loading": "INBOM4 (Mumbai (Ex Bombay))", "cntry_of_finaldstn": "SAUDI ARABIA", ... }, "file_id": "generated-file-id", "processing_time": "40.91s" } ``` The actual JSON must contain all required fields and child tables. --- # 35. COPY JSON Step 3 must contain: ```text [ COPY JSON ] ``` When clicked: ```text JSON copied successfully. ``` The copied content must be valid JSON. --- # 36. DOWNLOAD JSON Also provide: ```text [ DOWNLOAD JSON ] ``` The downloaded file should be: ```text shipping_bill_.json ``` For example: ```text shipping_bill_5632009.json ``` --- # 37. FILE ID The: ```json "file_id": "" ``` value is application metadata. Do not attempt to extract it from the PDF. It should be generated by the application when the PDF is uploaded/processed. --- # 38. PROCESSING TIME The: ```json "processing_time": "" ``` value must be measured by the backend. Example: ```text Start timer ↓ PDF processing ↓ Extraction ↓ Validation ↓ End timer ``` Then: ```json "processing_time": "40.91s" ``` Do not hardcode this. --- # 39. FINAL JSON IS THE CONTRACT Treat the final JSON as the contract between: ```text PDF Extraction Engine ``` and: ```text Frappe DocType Mapping Engine ``` Therefore: ```text PDF ↓ Extraction ↓ Structured JSON ↓ Validation ↓ APPROVAL ↓ FINAL JSON ``` must be independent from the future DocType creation logic. --- # 40. FUTURE DOCType MAPPING Do not implement this immediately unless requested. Future flow: ```text Approved JSON | v Shipping Bill Mapper | v Shipping Bill DocType | +--- Manifest Detail | +--- Invoice Detail | +--- Annex Detail | +--- Item Detail | +--- Single Window Declaration | +--- Other child tables ``` The final JSON structure is intentionally designed so that it can later be mapped directly to Frappe. --- # 41. EXPECTED TARGET JSON The following is the example structure that the system should reproduce from the provided Shipping Bill PDF. ```json { "shipping_bill_json": { "doctype": "Shipping Bill", "port_code": "INBOM4", "gwt": 1143.0, "inv": 1, "shipping_bill_no": "5632009", "port_of_loading": "INBOM4 (Mumbai (Ex Bombay))", "cntry_of_finaldstn": "SAUDI ARABIA", "state_of_origin": "Maharashtra", "port_of_finaldstn": "JED (JEDDAH )", "port_of_discharge": "JED (JEDDAH )", "cntry_of_discharge": "SAUDI ARABIA", "gwt_unit": "KGS", "item": 1, "shipping_bill_date": "2026-08-04", "pkg": 2, "cont": 0, "iec": "0300014953", "iec_branch_code": "0", "cb_code": "AAACM6824HCH032", "p_1a_mode": "AIR", "p_1a_assess": "N", "p_1a_dbk": "N", "p_1a_re_exp": null, "p_1a_exam": "Y", "p_1a_rodtp": "N", "p_1a_lut": "Y", "p_1a_job": "N", "p_1a_licence": "N", "p_1a_dfrc": "N", "p_1b_exporter_name": "ENDRESS + HAUSER (INDIA) PRIVATE LIMITED", "p_1b_exporter_address": "7B,7TH FLR.GODREJ ONE,PIROJSHANAGAR\nEASTERN EXPRESS HIGHWAY Contact No:\nVIKHROLI EAST", "p_1b_cb_name": "M/S.DHL LOGISTICS PVT LTD.", "p_1b_consignee_name": "NEOM GREEN HYDROGEN COMPANY LTD", "p_1b_consignee_address": "KINGDOM OF SAUDI ARABIA CR# 3550139\n610 P.O BOX 8244 BUILDING NO. 4758 KHURAYBAH 49643", "p_1b_type": "Private", "p_1b_gstin": "27AAACE5283C1ZV GSN", "p_1b_ad_code": "6550001", "p_1b_forex_ac_no": "51XXXX000", "p_1c_fob_val": 18956370.0, "p_1c_com": 0.0, "p_1c_freight": 0.0, "p_1c_deductions": 0.0, "p_1c_insurance": 0.0, "p_1c_p_c": 0.0, "p_1c_discount": 0.0, "p_1d_dbk": 0.0, "p_1d_rodtep_amt": 0.0, "p_1d_rosctle_amt": 0.0, "p_1j_subm_dt": "2026-08-04", "p_1i_subm_time": "11:40:00", "p_1i_leo_no": "2/838", "p_1i_leo_dt": "2026-08-05", "p_1i_exmn_dt": "2026-08-05", "p_1i_exmn_time": "19:01:00", "p_1i_leo_time": "19:55:00", "p_1i_brc_realzn_dt": "2027-05-31", "manifest_details": [ { "doctype": "Manifest Detail", "p_1e_mawb_no": "06547093771", "p_1e_cin_no": "26PCEG08053702299400", "p_1e_cin_dt": "05-AUG-26", "p_1e_cin_site_id": "INBOM4" } ], "challan_details": [], "annex_details": [ { "doctype": "Annex Detail", "p_1i_seal_typ": "WAREHOUSE SEALED", "p_1i_loose_pkts": 2, "p_1i_nature_cargo": "PACKAGED", "p_1i_marks_numbers": "AS PER INVOICE & PACKING LIST: SUPPLY MEANT FOR EXPORT FOR AUTHORISED OPERATIONS UNDER LETTER OF UNDERTAKING WITHOUT PAYMENT OF INTEGRATED TAX. ARN No. AD2703260673407 DT: 24/03/2026. AEO NO: INAAACE5283C3F263 DT: 29.05.2026 VALID UPTO 28.05.2031", "p_1i_no_of_pkgs": 2, "p_1i_no_of_containers": 0 } ], "invoice_details": [ { "doctype": "Invoice Detail", "inv_sn": 1, "p_2a_inv_no": "6031962610", "p_2a_inv_dt": "2026-07-29", "p_2a_ad_code": "6550001", "p_2a_invterm": "FOB", "p_2b_exporter_name": "ENDRESS + HAUSER (INDIA) PRIVATE LIMITED", "p_2b_buyer_addr": "KINGDOM OF SAUDI ARABIA CR# 3550139\n610 P.O BOX 8244 BUILDING NO. 4758\nKHURAYBAH 49643", "p_2b_exporter_addr": "7B,7TH FLR.GODREJ ONE,PIROJSHANAGAR\nEASTERN EXPRESS HIGHWAY Contact No:\n400079", "p_2b_buyer_name": "NEOM GREEN HYDROGEN COMPANY LTD", "p_2c_invoice_value": 198600.0, "p_2c_freight": 0.0, "p_2c_discount": 0.0, "p_2c_deduct": 0.0, "p_2c_exchng_rate_desc": "1 USD INR 95.45", "p_2c_invoice_curr": "USD", "p_2c_frieght_curr": "USD", "p_2c_discount_curr": "USD", "p_2c_deduct_curr": null, "p_2c_fob_val": 198600.0, "p_2c_insurance": 0.0, "p_2c_commison": 0.0, "p_2c_p_c": 0.0, "p_2c_fob_curr": "USD", "p_2c_insurance_curr": "USD", "p_2c_commison_curr": null, "p_2c_p_c_curr": null } ], "aa__dfia_licence_details": [], "item_details": [ { "p_3a_invsno": "1", "p_3a_itemsn": "1", "p_3a_cth": "90268090", "p_3a_item_desc": "IN GM GASMETER- FLOW METER FOR GAS MEASUREMENT, E&H FLOWMETER: TAG: 761-FE/FIT-8 818 (COO: GERMANY) AS PER PAC LIST SET", "p_3a_qty": "2", "p_3a_uqc": "SET", "p_3a_rate": "99300", "p_3a_value": "198600", "p_3a_fob": "18956370", "p_3a_pmv": "10426003.5", "p_3a_duty_amt": "", "p_3a_cess_rate": "", "p_3a_cess_amt": "", "p_3a_dbk_claimed": "N", "p_3a_igststat": "LUT", "p_3a_igst_val": "", "p_3a_igst_amt": "", "p_3a_schcod": "00", "p_3a_scheme_desc": "Free SB Involving Remittance O", "p_3a_sqc_mst": "2", "p_3a_sqc_uqc": "NOS", "p_3a_state_of_origin": "Maharashtra", "p_3a_district_of_origin": "MUMBAI", "p_3a_pt_abroad": "NCPTI", "p_3a_comp_cess": 0.0, "p_3a_end_use": "GNX100", "p_3a_benefit_availd": "Y", "p_3a_reward_benefit": "No", "p_3a_third_party_item": "N", "doctype": "Item Detail" } ], "drawback_and_rosl_claim": [], "job_details": [], "single_window_declaration": [ { "p_4d_invsn": "1", "p_4d_itmsn": "1", "p_4d_info": "CHR", "p_4d_qualifier": "SQC", "p_4d_info_cd": "", "p_4d_info_text": "", "p_4d_info_msr": "2", "p_4d_uqc": "NOS", "doctype": "Single Window Declaration" }, { "p_4d_invsn": "1", "p_4d_itmsn": "1", "p_4d_info": "DTY", "p_4d_qualifier": "GCESS", "p_4d_info_cd": "", "p_4d_info_text": "", "p_4d_info_msr": "0", "p_4d_uqc": "INR", "doctype": "Single Window Declaration" }, { "p_4d_invsn": "1", "p_4d_itmsn": "1", "p_4d_info": "DTY", "p_4d_qualifier": "RDT", "p_4d_info_cd": "RODTEPN", "p_4d_info_text": "Not Claimed", "p_4d_info_msr": "", "p_4d_uqc": "", "doctype": "Single Window Declaration" }, { "p_4d_invsn": "1", "p_4d_itmsn": "1", "p_4d_info": "ORC", "p_4d_qualifier": "DOO", "p_4d_info_cd": "482", "p_4d_info_text": "", "p_4d_info_msr": "", "p_4d_uqc": "", "doctype": "Single Window Declaration" }, { "p_4d_invsn": "1", "p_4d_itmsn": "1", "p_4d_info": "ORC", "p_4d_qualifier": "EPT", "p_4d_info_cd": "NCPTI", "p_4d_info_text": "", "p_4d_info_msr": "", "p_4d_uqc": "", "doctype": "Single Window Declaration" }, { "p_4d_invsn": "1", "p_4d_itmsn": "1", "p_4d_info": "ORC", "p_4d_qualifier": "STO", "p_4d_info_cd": "27", "p_4d_info_text": "", "p_4d_info_msr": "", "p_4d_uqc": "", "doctype": "Single Window Declaration" } ], "single_window_declaration_constituents": [], "single_window_declaration_control": [], "supporting_documents": [], "ar4_details": [], "third_party_details": [], "item_manufacturerproducergrower_details": [], "rodtep_details": [], "container_details": [] }, "file_id": "generated-file-id", "processing_time": "40.91s" } ``` **Important:** The above JSON is a target structure/example. The actual application must always take the **PDF as the source of truth** and extract the values from the uploaded document. Do not blindly copy example values into every document. --- # 42. DEVELOPMENT RULE Do not try to implement everything at once. Work in this order: ```text 1. Frappe Page ↓ 2. PDF Upload ↓ 3. PDF Detection ↓ 4. Page Count ↓ 5. Text Extraction ↓ 6. OCR fallback ↓ 7. Shipping Bill Section Detection ↓ 8. Parent Field Extraction ↓ 9. Child Table Extraction ↓ 10. JSON Construction ↓ 11. JSON Validation ↓ 12. Step 2 Review UI ↓ 13. Approve ↓ 14. Step 3 JSON Output ↓ 15. Copy JSON ↓ 16. Download JSON ↓ 17. Future DocType Creation ``` --- # 43. HOW TO GUIDE ME I am not asking for a theoretical architecture only. I want to actually build this application. When I provide my existing Frappe files/code: 1. Analyze the existing implementation first. 2. Do not replace working code unnecessarily. 3. Tell me exactly which files need modification. 4. Tell me whether a new file is required. 5. Give me complete replacement code where appropriate. 6. Keep code compatible with my existing Frappe application. 7. Do not assume files/classes exist unless I provide them. 8. Do not invent DocType fields. 9. Preserve the exact JSON field names. 10. Implement one step at a time. 11. Test each stage logically before moving to the next. 12. If an extraction problem is caused by the PDF layout, explain the exact problem and modify the extraction logic accordingly. 13. Do not solve extraction problems by introducing paid AI. 14. Do not use AI unless I explicitly approve it. --- # 44. FIRST IMPLEMENTATION TARGET Do NOT start with DocType creation. Our immediate milestone is: ```text Frappe Page ↓ Upload Shipping Bill PDF ↓ Next ↓ Detect Text/Image/Mixed ↓ Process ALL 6 pages ↓ Extract Shipping Bill fields ↓ Display organized review tables ↓ User clicks APPROVE ↓ Generate exact Shipping Bill JSON ↓ Step 3 ↓ Display formatted JSON ↓ Copy JSON ↓ Download JSON ``` Once this works correctly with the supplied 6-page Shipping Bill PDF, we will move to the next phase: ```text Approved JSON ↓ Frappe Shipping Bill DocType ↓ Child Tables ↓ Create Document ``` Treat this architecture and workflow as the **canonical project plan** for this implementation.