Project

General

Profile

Download (6.33 KB)

Bug #146 » README.md

README: Make automated DTA Sale and BOE Import records match the manually created ones - Akshath Shettigara, 10/15/2026 04:35 PM

 

type: redmine-readme
project: Canofy Process Standardization
subject: Make automated DTA Sale and BOE Import records match the manually created ones
story_key: nsdl-data-parity-dta-be
generated_at: 2026-10-15T16:16:51+05:30
story_type: bug
attachments:

  • file: nsdl-data-parity-dta-be_20261015-1616__verify_fields.py
    description: Compares live Frappe documents with the NSDL export, field by field (DTA); reads credentials from AWS at run time
  • file: nsdl-data-parity-dta-be_20261015-1616__verify_new_parser.py
    description: Runs the changed DTA parser on stored exports and compares with manually created documents
  • file: nsdl-data-parity-dta-be_20261015-1616__verify_be_parity.py
    description: Same comparison for FTWZ BOE Import documents

Make automated DTA Sale and BOE Import records match the manually created ones

Fields that the manual process fills but the automation left empty are now sent, based on a comparison with hundreds of manually created documents.

1. Summary

The user compared an automated FTWZ DTA Sale document with a manual one and found empty fields such as Buyer Address. The records were compared field by field against NSDL's export and against all available manually created documents. The DTA parser now maps two address columns and defaults two item fields to "0" when NSDL leaves them blank. The same check was then done for FTWZ BOE Import, which needed one address field and one default. After each change the parser output equals the manual records for every compared field.

2. Requirement

# Requirement Source Status
R1 Review all DTA Sale fields and find what is missing in automated records user (2026-10-13) Done
R2 A default is accepted only if supported by many records, not one user: "we can't conclude based on one record" Done
R3 Apply the address and default changes; check the PAN and dealer fields user Done
R4 Fix the BE parser the same way user (2026-10-14) Done

3. Decisions

Decision Evidence Date
Map BUYER ADDRESS and CLIENT ADDRESS (DTA) NSDL has both in all 954 rows of that export; manual documents have them in 257 of 257 2026-10-13
Send "0" for blank freight and blank BCD-to-be-considered; keep NSDL's value when present 257 manual documents, 1,861 item rows: blank freight is "0" in 114 of 114 cases, BCD-to-be-considered is "0" in 1,861 of 1,861 2026-10-13
Do not map buyer PAN, authorised dealer code and name, place of origin (DTA) Blank or "NA" in NSDL for every matched document; empty on the manual records 2026-10-13
Map CLIENT ADDRESS and default BCD-to-be-considered to "0" (BE) 105 manual BOE documents, 1,079 item rows: address present 105 of 105, "0" in 1,079 of 1,079 2026-10-14
Keep sending BE authorised dealer code and name Manual records lack them in 7 of 105 documents; the user did not decide 2026-10-14

4. Logic & Workflow

Check method: stored NSDL export -> parser -> payload compared with manually created Frappe document for the same request ID, field by field, per item row.

Field DTA BE
buyer_address Copied from BUYER ADDRESS Not part of the BE comparison
client_address Copied from CLIENT ADDRESS Copied from CLIENT ADDRESS
Item freight_in_inr NSDL value, "0" when blank Unchanged; BE freight showed 0.000 from NSDL
Item bcd_to_be_considered_for_calculating_igst "0" when blank "0" when blank

NSDL numbers are kept verbatim (a value such as 15100.740 is sent as NSDL wrote it).

5. Files & Components Changed

File / resource Type Change Purpose
app/phase_a_fetch/nsdl_parser.py Code DTA: address mapping and "0" defaults (24 lines); BE: address and default (7 lines) Field parity

6. Configuration & Deployment

No configuration change. Deployed through the pipeline after each merge (2026-10-13 and 2026-10-14). Documents that already exist in Frappe are found by request ID and reused, so they do not receive new fields: they must be deleted and re-created, or back-filled.

7. Testing

Test Data / Site Expected Actual Status
Changed DTA parser vs manual documents 257 documents, 1,861 item rows, two sites Equal Addresses 257/257; BCD-to-be-considered 1,861/1,861; freight 1,857/1,861 (the other 4 are the same number in another format) Pass
Changed BE parser vs manual documents 105 documents, 1,079 item rows Equal; no regression on DTA No field missing, no item row differs; DTA unchanged Pass
Ruff, compile, sam validate, earlier test suites Repo Pass Pass Pass
Live DTA re-stage and delivery (wud) Real NSDL export, TEST site New fields present 3 new documents with both addresses and "0" defaults Pass
Live BE delivery (eap) 44 BE records New fields present 44 documents (536 items) with client_address and "0" Pass
Live reuse of existing documents wud 3, eap 43 Fields updated Not updated (by design) Known limitation

8. Known Limitations & Open Items

  • Existing wud documents FTWZ-DTA-S-00102, 00108 and 00112 and 43 existing eap DTA documents lack the new fields. Whether to delete and re-create or back-fill them: not confirmed.
  • BE authorised dealer code and name are still sent; the user did not answer whether to drop them.
  • The BE comparison covered only the eap site; the DTA comparison covered two sites. 1,810 of the 4,182 eap DTA documents belong to the Administrator user, probably a script import, so that sample is not purely hand-entered (the result was the same either way).
  • On eap, the API user is now a named person, so automated documents show that person as owner. Whether this is intended: not confirmed.
  • wud documents were removed by the user before the re-run; the 6 DynamoDB rows were backed up and deleted with approval.

9. Status

Item Value
Branch feature/AI-TBD-nsdl-otp-login
Commits a557399, b7d7f08
PR Created and merged by the user for each; numbers not confirmed
Deployed to TEST stack

10. References

Related stories: nsdl-boe-pdf-print-layout (same time period), nsdl-pdf-attachment-be-dta, nsdl-login-otp-captcha-update.

(1-1/7)