clqms-be/docs/MIGRATION_VALUESET_VID_TO_VVALUE.md

# Migration Plan: Valueset VID → VValue

## Overview

Transition from using database `valueset` table (VID as primary key) to the new `App\Libraries\ValueSet` library (VValue as key). This eliminates database joins for lookup values and uses JSON-based static lookup files.

## Current State

- Database `valueset` table with columns: `VID` (PK, INT), `VValue` (VARCHAR), `VDesc` (VARCHAR)
- 30+ places using `->join('valueset', 'valueset.VID = ...')`
- Selects use `valueset.VValue` and `valueset.VDesc` for display text

## Target State

- Use `App\Libraries\ValueSet` library with `getLabel(lookupName, key)` method
- Lookup names use table-prefixed PascalCase (e.g., `patient_Sex`, `test_TestType`, `container_ContainerCapColor`)
- All fields store `VValue` codes directly (e.g., '1', '2', 'M', 'F', 'TEST')
- Remove all `valueset` table joins from queries
- Keep raw field values for codes; use `ValueSet::getLabel()` for display text
- JSON files in `app/Libraries/Data/valuesets/` are already populated

---

## Phase 1: JSON Files Rename

Rename all JSON files in `app/Libraries/Data/valuesets/` to use table-prefixed PascalCase format:

| Old Name | New Name | Source Table | Field |
|----------|----------|--------------|-------|
| `gender.json` | `patient_Sex.json` | patient | Gender |
| `country.json` | `patient_Country.json` | patient | Country |
| `race.json` | `patient_Race.json` | patient | Race |
| `religion.json` | `patient_Religion.json` | patient | Religion |
| `ethnic.json` | `patient_Ethnic.json` | patient | Ethnic |
| `marital_status.json` | `patient_MaritalStatus.json` | patient | MaritalStatus |
| `death_indicator.json` | `patient_DeathIndicator.json` | patient | DeathIndicator |
| `test_type.json` | `test_TestType.json` | testdefsite | TestType |
| `container_cap_color.json` | `container_ContainerCapColor.json` | containerdef | Color |
| `container_class.json` | `container_ContainerClass.json` | containerdef | ConClass |
| `additive.json` | `container_Additive.json` | containerdef | Additive |
| `location_type.json` | `location_LocationType.json` | location | LocType |
| `ws_type.json` | `organization_WorkstationType.json` | workstation | Type |
| `enable_disable.json` | `organization_EnableDisable.json` | workstation | Enable |
| `site_type.json` | `organization_SiteType.json` | site | SiteTypeID |
| `site_class.json` | `organization_SiteClass.json` | site | SiteClassID |
| `numeric_ref_type.json` | `ref_NumericRefType.json` | refnum | NumRefType |
| `range_type.json` | `ref_RangeType.json` | refnum | RangeType |
| `text_ref_type.json` | `ref_TextRefType.json` | reftxt | TxtRefType |
| `reference_type.json` | `test_ReferenceType.json` | testdeftech | RefType |
| `math_sign.json` | `ref_MathSign.json` | refnum | LowSign, HighSign |
| `country.json` | `account_Country.json` | account | Country |
| ... | ... | ... | ... |

All lookup names use `{table}_{Field}` format for clarity and namespace isolation.

---

## Phase 2: Database Schema Migration

### Files to DELETE

| File | Action |
|------|--------|
| `app\Database\Seeds\ValueSetSeeder.php` | DELETE |
| `app\Database\Seeds\ValueSetCountrySeeder.php` | DELETE |
| `app\Database\Seeds\MinimalMasterDataSeeder.php` | DELETE |
| `app\Database\Seeds\PatientSeeder.php` | DELETE |
| `app\Database\Migrations\2025-09-15-130122_ValueSet.php` | DELETE |

### Migration: Modify Columns INT → VARCHAR(10)

**File:** `app\Database\Migrations\2026-01-12-000001_ValuesetVidToVvalue.php`

```php
<?php

namespace App\Database\Migrations;

use CodeIgniter\Database\Migration;

class ValuesetVidToVvalue extends Migration
{
    public function up()
    {
        // patient table
        $this->forge->modifyColumn('patient', [
            'Gender' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Country' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Race' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Religion' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Ethnic' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'MaritalStatus' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'DeathIndicator' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // testdefsite table
        $this->forge->modifyColumn('testdefsite', [
            'TestType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => false],
        ]);

        // containerdef table
        $this->forge->modifyColumn('containerdef', [
            'Additive' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'ConClass' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Color' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // location table
        $this->forge->modifyColumn('location', [
            'LocType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // workstation table
        $this->forge->modifyColumn('workstation', [
            'Type' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Enable' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // site table
        $this->forge->modifyColumn('site', [
            'SiteTypeID' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'SiteClassID' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // account table
        $this->forge->modifyColumn('account', [
            'Country' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // refnum table
        $this->forge->modifyColumn('refnum', [
            'Sex' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'NumRefType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'RangeType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'LowSign' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'HighSign' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // reftxt table
        $this->forge->modifyColumn('reftxt', [
            'Sex' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'TxtRefType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // orderstatus table
        $this->forge->modifyColumn('orderstatus', [
            'OrderStatus' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => false],
        ]);
    }

    public function down()
    {
        // Revert to INT
        $this->forge->modifyColumn('patient', [
            'Gender' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Country' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Race' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Religion' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Ethnic' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'MaritalStatus' => ['type' => 'TINYINT', 'null' => true],
            'DeathIndicator' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('testdefsite', [
            'TestType' => ['type' => 'INT', 'null' => false],
        ]);

        $this->forge->modifyColumn('containerdef', [
            'Additive' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'ConClass' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Color' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('location', [
            'LocType' => ['type' => 'INT', 'null' => true],
        ]);

        $this->forge->modifyColumn('workstation', [
            'Type' => ['type' => 'TINYINT', 'null' => true],
            'Enable' => ['type' => 'INT', 'null' => true],
        ]);

        $this->forge->modifyColumn('site', [
            'SiteTypeID' => ['type' => 'INT', 'null' => true],
            'SiteClassID' => ['type' => 'INT', 'null' => true],
        ]);

        $this->forge->modifyColumn('account', [
            'Country' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('refnum', [
            'Sex' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'NumRefType' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'RangeType' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'LowSign' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'HighSign' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('reftxt', [
            'Sex' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'TxtRefType' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('orderstatus', [
            'OrderStatus' => ['type' => 'INT', 'null' => false],
        ]);
    }
}
```

**Note:** No data migration needed - dummy data will be lost. This is acceptable for development/testing environments.

---

## Phase 3: Library & Model Updates

### ValueSet Library - Update to read from JSON files

**File:** `app/Libraries/ValueSet.php`

Ensure the library reads from JSON files in `app/Libraries/Data/valuesets/`:

```php
<?php

namespace App\Libraries;

class ValueSet
{
    private static $cache = [];

    private static function loadFile(string $name): array
    {
        if (!isset(self::$cache[$name])) {
            $path = APPPATH . 'Libraries/Data/valuesets/' . $name . '.json';
            if (file_exists($path)) {
                $content = file_get_contents($path);
                self::$cache[$name] = json_decode($content, true)['values'] ?? [];
            } else {
                self::$cache[$name] = [];
            }
        }
        return self::$cache[$name];
    }

    public static function getLabel(string $lookupName, string $key): ?string
    {
        $values = self::loadFile($lookupName);
        foreach ($values as $item) {
            if (($item['key'] ?? $item['value'] ?? null) === $key) {
                return $item['value'] ?? $item['label'] ?? null;
            }
        }
        return null;
    }

    public static function getOptions(string $lookupName): array
    {
        $values = self::loadFile($lookupName);
        return array_map(function ($item) {
            return [
                'key' => $item['key'] ?? '',
                'value' => $item['value'] ?? $item['label'] ?? '',
            ];
        }, $values);
    }

    public static function transformLabels(array $data, array $fieldMappings): array
    {
        foreach ($data as &$row) {
            foreach ($fieldMappings as $field => $lookupName) {
                if (isset($row[$field]) && $row[$field] !== null) {
                    $row[$field . 'Text'] = self::getLabel($lookupName, $row[$field]) ?? '';
                }
            }
        }
        return $data;
    }
}
```

### ValueSetModel - Deprecate or Repurpose

**File:** `app/Models/ValueSet/ValueSetModel.php`

Options:
1. **Deprecate entirely** - No longer needed after migration
2. **Repurpose for JSON file management** - Read/write to JSON files
3. **Keep as-is for backward compatibility** - If database valuesets are still needed

Recommended: Deprecate and remove references after migration.

---

## Phase 4: Model Changes

### Pattern for Model Updates

**Before:**
```php
$this->select("..., gender.VValue as Gender, gender.VDesc as GenderText")
     ->join('valueset gender', 'gender.VID = patient.Gender', 'left')
```

**After:**
```php
use App\Libraries\ValueSet;

$this->select("..., patient.Gender");
// After fetching:
$rows = ValueSet::transformLabels($rows, [
    'Gender' => 'patient_Sex',
    'Country' => 'patient_Country',
    'Race' => 'patient_Race',
    'Religion' => 'patient_Religion',
    'Ethnic' => 'patient_Ethnic',
    'DeathIndicator' => 'patient_DeathIndicator',
    'MaritalStatus' => 'patient_MaritalStatus',
]);
```

### Models to Modify (8 files)

#### 1. `app/Models/Patient/PatientModel.php`

**Remove:**
- Line 27: `$this->join('valueset vs', 'vs.vid = Gender', 'left');`
- Lines 52-64: All `*.VID as *VID` aliases
- Lines 75-81: All `valueset.*` joins

**Add transformation in `getPatient()`:**
```php
$patient = ValueSet::transformLabels([$patient], [
    'Gender' => 'patient_gender',
    'Country' => 'patient_country',
    'Race' => 'patient_race',
    'Religion' => 'patient_religion',
    'Ethnic' => 'patient_ethnic',
    'DeathIndicator' => 'patient_death_indicator',
    'MaritalStatus' => 'patient_marital_status',
])[0];
```

#### 2. `app/Models/Location/LocationModel.php`

**Remove:**
- Lines 18, 30: `->join("valueset v", "v.VID=location.loctype", ...)`

**Add transformation:**
```php
$rows = ValueSet::transformLabels($rows, [
    'LocType' => 'location_LocationType',
]);
```

#### 3. `app/Models/Test/TestDefSiteModel.php`

**Remove:**
- Lines 42, 75, 103: `->join("valueset", "valueset.VID=...")`

**Add transformation:**
```php
$rows = ValueSet::transformLabels($rows, [
    'TestType' => 'test_TestType',
]);
```

#### 4. `app/Models/Test/TestDefGrpModel.php`

**Remove:**
- Line 32: `->join('valueset vs', 'vs.VID=t.TestType', 'left')`

#### 5. `app/Models/Specimen/ContainerDefModel.php`

**Remove:**
- Lines 20-22, 37-39: All 6 `valueset.*` joins

**Add transformation:**
```php
$rows = ValueSet::transformLabels($rows, [
    'Color' => 'container_ContainerCapColor',
    'ConClass' => 'container_ContainerClass',
    'Additive' => 'container_Additive',
]);
```

#### 6. `app/Models/Organization/SiteModel.php`

**Remove:**
- Lines 38-39: `->join('valueset sitetype'...)` and `->join('valueset siteclass'...)`

**Add transformation:**
```php
$row = ValueSet::transformLabels([$row], [
    'SiteTypeID' => 'organization_SiteType',
    'SiteClassID' => 'organization_SiteClass',
])[0];
```

#### 7. `app/Models/Organization/AccountModel.php`

**Remove:**
- Line 41: `->join('valueset country'...)`

**Remove from select:**
- Line 36: `country.VID as country`

**Add transformation in controller if needed:**
```php
$rows = ValueSet::transformLabels($rows, [
    'Country' => 'account_Country',
]);
```

#### 8. `app/Models/Organization/WorkstationModel.php`

**Remove:**
- Lines 36-37: `->join('valueset wstype'...)` and `->join('valueset enable'...)`

**Add transformation:**
```php
$row = ValueSet::transformLabels([$row], [
    'Type' => 'organization_WorkstationType',
    'Enable' => 'organization_EnableDisable',
])[0];
```

---

## Phase 5: Controller Changes

### `app/Controllers/TestsController.php`

**Remove:**
- Line 69: `->join("valueset", "valueset.VID=testdefsite.TestType", "left")`
- Line 111: `->join("valueset", "valueset.VID=testdefsite.TestType", "left")`
- Line 140: `->join('valueset vs', 'vs.VID=t.TestType', 'left')`

**Replace `getVValue()` method:**
```php
private function getVValue($vsetID, $vid) {
    // DEPRECATED - Use ValueSet::getLabel() instead
    return null;
}
```

**Update references from `getVValue()` to `ValueSet::getLabel()`:**
```php
// Before:
'NumRefTypeVValue' => $this->getVValue(46, $r['NumRefType']),

// After:
'NumRefTypeVValue' => \App\Libraries\ValueSet::getLabel('ref_NumericRefType', $r['NumRefType']),
```

**VSetID to Lookup Name Mapping:**
| VSetID | Constant | Lookup Name |
|--------|----------|-------------|
| 44 | `VALUESET_REF_TYPE` | `test_ReferenceType` |
| 45 | `VALUESET_RANGE_TYPE` | `ref_RangeType` |
| 46 | `VALUESET_NUM_REF_TYPE` | `ref_NumericRefType` |
| 47 | `VALUESET_TXT_REF_TYPE` | `ref_TextRefType` |
| 3 | `VALUESET_SEX` | `patient_Sex` |
| 41 | `VALUESET_MATH_SIGN` | `ref_MathSign` |

**Update `getValuesetOptions()` to use JSON:**
```php
private function getValuesetOptions($lookupName)
{
    return \App\Libraries\ValueSet::getOptions($lookupName);
}
```

---

## Phase 6: API Endpoints - Replace with JSON-based endpoints

### New API Controller: `app/Controllers/ValueSetApiController.php`

```php
<?php

namespace App\Controllers;

use App\Libraries\ValueSet;

class ValueSetApiController extends \CodeIgniter\Controller
{
    use \CodeIgniter\API\ResponseTrait;

    public function index(string $lookupName)
    {
        $data = ValueSet::getOptions($lookupName);
        return $this->respond([
            'status' => 'success',
            'data' => $data
        ], 200);
    }

    public function all()
    {
        $dir = APPPATH . 'Libraries/Data/valuesets/';
        $files = glob($dir . '*.json');
        $result = [];
        foreach ($files as $file) {
            $name = basename($file, '.json');
            $result[] = [
                'name' => $name,
                'options' => ValueSet::getOptions($name)
            ];
        }
        return $this->respond([
            'status' => 'success',
            'data' => $result
        ], 200);
    }
}
```

### Update Routes: `app/Config/Routes.php`

```php
$routes->group('api', function ($routes) {
    $routes->get('valueset/(:segment)', 'ValueSetApiController::index/$1');
    $routes->get('valueset', 'ValueSetApiController::all');
});
```

---

## Phase 7: View Updates

### 1. `app/Views/v2/master/valuesets/valuesets_index.php`

Repurpose to manage JSON-based valuesets instead of database table.

| Before | After |
|--------|-------|
| `fetch(...api/valueset...)` | `fetch(...api/valueset/lookupName...)` |
| Database CRUD operations | File-based CRUD operations |

### 2. `app/Views/v2/master/valuesets/valueset_nested_crud.php`

Repurpose for JSON file management.

### 3. `app/Views/v2/master/valuesets/valueset_dialog.php`

Update for JSON file format.

### 4. `app/Views/v2/master/tests/tests_index.php`

| Before | After |
|--------|-------|
| `type?.VID` | `type?.key` |
| `type?.VValue` | `type?.value` |
| `type?.VDesc` | `type?.label` |
| `{ VID: 1, VValue: 'TEST', ... }` | `{ key: 'TEST', value: 'Test', ... }` |
| `getTypeName(vid)` | `getTypeName(value)` |
| `api/valuesetdef/27` | `api/valueset/test_TestType` |
| Hardcoded fallback: `{ VID: 1, VValue: 'TEST', VDesc: 'Test' }` | `{ key: 'TEST', value: 'Test' }` |

### 5. Additional Views to Update

| View File | Fields to Update | Lookup Name |
|-----------|------------------|-------------|
| `app/Views/v2/patients/patients_index.php` | Gender, Country, Race, Religion, Ethnic, MaritalStatus, DeathIndicator | `patient_*` |
| `app/Views/v2/master/specimen/containers_index.php` | Color, ConClass, Additive | `container_*` |
| `app/Views/v2/master/organization/sites_index.php` | SiteTypeID, SiteClassID | `organization_*` |
| `app/Views/v2/master/organization/workstations_index.php` | Type, Enable | `organization_*` |
| `app/Views/v2/master/organization/accounts_index.php` | Country | `account_Country` |
| `app/Views/v2/master/organization/locations_index.php` | LocType | `location_LocationType` |

---

## Phase 8: Test Files Update

### `tests/feature/ValueSet/ValueSetApiControllerTest.php`

Update tests to use new JSON-based API endpoints.

### `tests/_support/v2/MasterTestCase.php`

Update any valueset-related test data setup.

---

## Testing Checklist

1. **Unit Tests**
   - Test `ValueSet::getLabel('patient_Sex', '1')` returns 'Female'
   - Test `ValueSet::getLabel('test_TestType', 'TEST')` returns 'Test'
   - Test `ValueSet::getOptions('container_ContainerCapColor')` returns correct format
   - Test `ValueSet::transformLabels()` with table-prefixed field mappings

2. **Integration Tests**
   - Patient CRUD (create, read, update, delete)
   - Test definition CRUD
   - Location CRUD
   - Container definition CRUD
   - Organization (site, account, workstation) CRUD

3. **Manual Testing**
   - Verify all dropdowns display correct labels
   - Verify filtering by valueset fields works
   - Verify form submissions save correct VValue codes

---

## Rollback Plan

1. Run migration `down()` to revert column types
2. Restore deleted seeders from git if needed
3. Restore deleted migration file from git

---

## Files Summary

| Phase | Action | Files |
|-------|--------|-------|
| 1 | RENAME | ~50 JSON files in `app/Libraries/Data/valuesets/` |
| 2 | DELETE | 5 seeders, 1 migration |
| 2 | CREATE | 1 migration (column changes) |
| 3 | UPDATE | 1 library (ValueSet.php), ValueSetModel.php deprecation |
| 4 | UPDATE | 8 Model files |
| 5 | UPDATE | 1 Controller, Routes |
| 6 | CREATE | 1 Controller (ValueSetApiController.php) |
| 6 | UPDATE | ~6+ View files |
| 8 | UPDATE | 2+ Test files |

---

## Estimated Effort

- Phase 1 (JSON Rename): 15 minutes
- Phase 2 (Migration): 30 minutes
- Phase 3 (Library + Model): 30 minutes
- Phase 4 (Models): 1.5 hours
- Phase 5 (Controller): 30 minutes
- Phase 6 (API + Views): 2 hours
- Phase 8 (Tests): 30 minutes
- Testing: 1 hour

**Total Estimated Time: ~7 hours**