Bug #146
openMake automated DTA Sale and BOE Import records match the manually created ones
0%
Description
Make automated DTA Sale and BOE Import records match the manually created ones¶
Fields that the manual process fills but the automation left empty are now sent, based on a comparison with hundreds of manually created documents.
1. Summary¶
The user compared an automated FTWZ DTA Sale document with a manual one and found empty fields such as Buyer Address. The records were compared field by field against NSDL's export and against all available manually created documents. The DTA parser now maps two address columns and defaults two item fields to "0" when NSDL leaves them blank. The same check was then done for FTWZ BOE Import, which needed one address field and one default. After each change the parser output equals the manual records for every compared field.
2. Requirement¶
| # | Requirement | Source | Status |
|---|---|---|---|
| R1 | Review all DTA Sale fields and find what is missing in automated records | user (2026-10-13) | Done |
| R2 | A default is accepted only if supported by many records, not one | user: "we can't conclude based on one record" | Done |
| R3 | Apply the address and default changes; check the PAN and dealer fields | user | Done |
| R4 | Fix the BE parser the same way | user (2026-10-14) | Done |
3. Decisions¶
| Decision | Evidence | Date |
|---|---|---|
Map BUYER ADDRESS and CLIENT ADDRESS (DTA) |
NSDL has both in all 954 rows of that export; manual documents have them in 257 of 257 | 2026-10-13 |
Send "0" for blank freight and blank BCD-to-be-considered; keep NSDL's value when present |
257 manual documents, 1,861 item rows: blank freight is "0" in 114 of 114 cases, BCD-to-be-considered is "0" in 1,861 of 1,861 | 2026-10-13 |
| Do not map buyer PAN, authorised dealer code and name, place of origin (DTA) | Blank or "NA" in NSDL for every matched document; empty on the manual records | 2026-10-13 |
Map CLIENT ADDRESS and default BCD-to-be-considered to "0" (BE) |
105 manual BOE documents, 1,079 item rows: address present 105 of 105, "0" in 1,079 of 1,079 | 2026-10-14 |
| Keep sending BE authorised dealer code and name | Manual records lack them in 7 of 105 documents; the user did not decide | 2026-10-14 |
4. Logic & Workflow¶
Check method: stored NSDL export -> parser -> payload compared with manually created Frappe document for the same request ID, field by field, per item row.
| Field | DTA | BE |
|---|---|---|
buyer_address |
Copied from BUYER ADDRESS
|
Not part of the BE comparison |
client_address |
Copied from CLIENT ADDRESS
|
Copied from CLIENT ADDRESS
|
Item freight_in_inr
|
NSDL value, "0" when blank |
Unchanged; BE freight showed 0.000 from NSDL |
Item bcd_to_be_considered_for_calculating_igst
|
"0" when blank |
"0" when blank |
NSDL numbers are kept verbatim (a value such as 15100.740 is sent as NSDL wrote it).
5. Files & Components Changed¶
| File / resource | Type | Change | Purpose |
|---|---|---|---|
app/phase_a_fetch/nsdl_parser.py |
Code | DTA: address mapping and "0" defaults (24 lines); BE: address and default (7 lines) |
Field parity |
6. Configuration & Deployment¶
No configuration change. Deployed through the pipeline after each merge (2026-10-13 and 2026-10-14). Documents that already exist in Frappe are found by request ID and reused, so they do not receive new fields: they must be deleted and re-created, or back-filled.
7. Testing¶
| Test | Data / Site | Expected | Actual | Status |
|---|---|---|---|---|
| Changed DTA parser vs manual documents | 257 documents, 1,861 item rows, two sites | Equal | Addresses 257/257; BCD-to-be-considered 1,861/1,861; freight 1,857/1,861 (the other 4 are the same number in another format) | Pass |
| Changed BE parser vs manual documents | 105 documents, 1,079 item rows | Equal; no regression on DTA | No field missing, no item row differs; DTA unchanged | Pass |
Ruff, compile, sam validate, earlier test suites |
Repo | Pass | Pass | Pass |
Live DTA re-stage and delivery (wud) |
Real NSDL export, TEST site | New fields present | 3 new documents with both addresses and "0" defaults |
Pass |
Live BE delivery (eap) |
44 BE records | New fields present | 44 documents (536 items) with client_address and "0"
|
Pass |
| Live reuse of existing documents |
wud 3, eap 43 |
Fields updated | Not updated (by design) | Known limitation |
8. Known Limitations & Open Items¶
- Existing
wuddocumentsFTWZ-DTA-S-00102,00108and00112and 43 existingeapDTA documents lack the new fields. Whether to delete and re-create or back-fill them: not confirmed. - BE authorised dealer code and name are still sent; the user did not answer whether to drop them.
- The BE comparison covered only the
eapsite; the DTA comparison covered two sites. 1,810 of the 4,182eapDTA documents belong to theAdministratoruser, probably a script import, so that sample is not purely hand-entered (the result was the same either way). - On
eap, the API user is now a named person, so automated documents show that person as owner. Whether this is intended: not confirmed. -
wuddocuments were removed by the user before the re-run; the 6 DynamoDB rows were backed up and deleted with approval.
9. Status¶
| Item | Value |
|---|---|
| Branch | feature/AI-TBD-nsdl-otp-login |
| Commits |
a557399, b7d7f08
|
| PR | Created and merged by the user for each; numbers not confirmed |
| Deployed to | TEST stack |
10. References¶
Related stories: nsdl-boe-pdf-print-layout (same time period), nsdl-pdf-attachment-be-dta, nsdl-login-otp-captcha-update.
Pushed to Redmine on 15 Oct 2026, 16:35 from nsdl-data-parity-dta-be_20261015-1616_20261015-163531.md (README created 15 Oct 2026, 16:35).
Files
No data to display