clqms-be/docs/MIGRATION_VALUESET_VID_TO_VVALUE.md

Migration Plan: Valueset VID → VValue

Overview

Transition from using database valueset table (VID as primary key) to the new App\Libraries\ValueSet library (VValue as key). This eliminates database joins for lookup values and uses JSON-based static lookup files.

Current State

• Database valueset table with columns: VID (PK, INT), VValue (VARCHAR), VDesc (VARCHAR)
• 30+ places using ->join('valueset', 'valueset.VID = ...')
• Selects use valueset.VValue and valueset.VDesc for display text

Target State

• Use App\Libraries\ValueSet library with getLabel(lookupName, key) method
• Lookup names use table-prefixed PascalCase (e.g., patient_Sex, test_TestType, container_ContainerCapColor)
• All fields store VValue codes directly (e.g., '1', '2', 'M', 'F', 'TEST')
• Remove all valueset table joins from queries
• Keep raw field values for codes; use ValueSet::getLabel() for display text
• JSON files in app/Libraries/Data/valuesets/ are already populated

---

Phase 1: JSON Files Rename

Rename all JSON files in app/Libraries/Data/valuesets/ to use table-prefixed PascalCase format:

Old Name	New Name	Source Table	Field
gender.json	patient_Sex.json	patient	Gender
country.json	patient_Country.json	patient	Country
race.json	patient_Race.json	patient	Race
religion.json	patient_Religion.json	patient	Religion
ethnic.json	patient_Ethnic.json	patient	Ethnic
marital_status.json	patient_MaritalStatus.json	patient	MaritalStatus
death_indicator.json	patient_DeathIndicator.json	patient	DeathIndicator
test_type.json	test_TestType.json	testdefsite	TestType
container_cap_color.json	container_ContainerCapColor.json	containerdef	Color
container_class.json	container_ContainerClass.json	containerdef	ConClass
additive.json	container_Additive.json	containerdef	Additive
location_type.json	location_LocationType.json	location	LocType
ws_type.json	organization_WorkstationType.json	workstation	Type
enable_disable.json	organization_EnableDisable.json	workstation	Enable
site_type.json	organization_SiteType.json	site	SiteTypeID
site_class.json	organization_SiteClass.json	site	SiteClassID
numeric_ref_type.json	ref_NumericRefType.json	refnum	NumRefType
range_type.json	ref_RangeType.json	refnum	RangeType
text_ref_type.json	ref_TextRefType.json	reftxt	TxtRefType
reference_type.json	test_ReferenceType.json	testdeftech	RefType
math_sign.json	ref_MathSign.json	refnum	LowSign, HighSign
country.json	account_Country.json	account	Country
...	...	...	...

All lookup names use {table}_{Field} format for clarity and namespace isolation.

---

Phase 2: Database Schema Migration

Files to DELETE

File	Action
app\Database\Seeds\ValueSetSeeder.php	DELETE
app\Database\Seeds\ValueSetCountrySeeder.php	DELETE
app\Database\Seeds\MinimalMasterDataSeeder.php	DELETE
app\Database\Seeds\PatientSeeder.php	DELETE
app\Database\Migrations\2025-09-15-130122_ValueSet.php	DELETE

Migration: Modify Columns INT → VARCHAR(10)

File: app\Database\Migrations\2026-01-12-000001_ValuesetVidToVvalue.php

<?php

namespace App\Database\Migrations;

use CodeIgniter\Database\Migration;

class ValuesetVidToVvalue extends Migration
{
    public function up()
    {
        // patient table
        $this->forge->modifyColumn('patient', [
            'Gender' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Country' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Race' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Religion' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Ethnic' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'MaritalStatus' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'DeathIndicator' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // testdefsite table
        $this->forge->modifyColumn('testdefsite', [
            'TestType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => false],
        ]);

        // containerdef table
        $this->forge->modifyColumn('containerdef', [
            'Additive' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'ConClass' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Color' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // location table
        $this->forge->modifyColumn('location', [
            'LocType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // workstation table
        $this->forge->modifyColumn('workstation', [
            'Type' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'Enable' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // site table
        $this->forge->modifyColumn('site', [
            'SiteTypeID' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'SiteClassID' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // account table
        $this->forge->modifyColumn('account', [
            'Country' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // refnum table
        $this->forge->modifyColumn('refnum', [
            'Sex' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'NumRefType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'RangeType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'LowSign' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'HighSign' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // reftxt table
        $this->forge->modifyColumn('reftxt', [
            'Sex' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
            'TxtRefType' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => true],
        ]);

        // orderstatus table
        $this->forge->modifyColumn('orderstatus', [
            'OrderStatus' => ['type' => 'VARCHAR', 'constraint' => 10, 'null' => false],
        ]);
    }

    public function down()
    {
        // Revert to INT
        $this->forge->modifyColumn('patient', [
            'Gender' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Country' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Race' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Religion' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Ethnic' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'MaritalStatus' => ['type' => 'TINYINT', 'null' => true],
            'DeathIndicator' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('testdefsite', [
            'TestType' => ['type' => 'INT', 'null' => false],
        ]);

        $this->forge->modifyColumn('containerdef', [
            'Additive' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'ConClass' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'Color' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('location', [
            'LocType' => ['type' => 'INT', 'null' => true],
        ]);

        $this->forge->modifyColumn('workstation', [
            'Type' => ['type' => 'TINYINT', 'null' => true],
            'Enable' => ['type' => 'INT', 'null' => true],
        ]);

        $this->forge->modifyColumn('site', [
            'SiteTypeID' => ['type' => 'INT', 'null' => true],
            'SiteClassID' => ['type' => 'INT', 'null' => true],
        ]);

        $this->forge->modifyColumn('account', [
            'Country' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('refnum', [
            'Sex' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'NumRefType' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'RangeType' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'LowSign' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'HighSign' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('reftxt', [
            'Sex' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
            'TxtRefType' => ['type' => 'INT', 'constraint' => 11, 'null' => true],
        ]);

        $this->forge->modifyColumn('orderstatus', [
            'OrderStatus' => ['type' => 'INT', 'null' => false],
        ]);
    }
}

Note: No data migration needed - dummy data will be lost. This is acceptable for development/testing environments.

---

Phase 3: Library & Model Updates

ValueSet Library - Update to read from JSON files

File: app/Libraries/ValueSet.php

Ensure the library reads from JSON files in app/Libraries/Data/valuesets/:

<?php

namespace App\Libraries;

class ValueSet
{
    private static $cache = [];

    private static function loadFile(string $name): array
    {
        if (!isset(self::$cache[$name])) {
            $path = APPPATH . 'Libraries/Data/valuesets/' . $name . '.json';
            if (file_exists($path)) {
                $content = file_get_contents($path);
                self::$cache[$name] = json_decode($content, true)['values'] ?? [];
            } else {
                self::$cache[$name] = [];
            }
        }
        return self::$cache[$name];
    }

    public static function getLabel(string $lookupName, string $key): ?string
    {
        $values = self::loadFile($lookupName);
        foreach ($values as $item) {
            if (($item['key'] ?? $item['value'] ?? null) === $key) {
                return $item['value'] ?? $item['label'] ?? null;
            }
        }
        return null;
    }

    public static function getOptions(string $lookupName): array
    {
        $values = self::loadFile($lookupName);
        return array_map(function ($item) {
            return [
                'key' => $item['key'] ?? '',
                'value' => $item['value'] ?? $item['label'] ?? '',
            ];
        }, $values);
    }

    public static function transformLabels(array $data, array $fieldMappings): array
    {
        foreach ($data as &$row) {
            foreach ($fieldMappings as $field => $lookupName) {
                if (isset($row[$field]) && $row[$field] !== null) {
                    $row[$field . 'Text'] = self::getLabel($lookupName, $row[$field]) ?? '';
                }
            }
        }
        return $data;
    }
}

ValueSetModel - Deprecate or Repurpose

File: app/Models/ValueSet/ValueSetModel.php

Options:

1. Deprecate entirely - No longer needed after migration
2. Repurpose for JSON file management - Read/write to JSON files
3. Keep as-is for backward compatibility - If database valuesets are still needed

Recommended: Deprecate and remove references after migration.

---

Phase 4: Model Changes

Pattern for Model Updates

Before:

$this->select("..., gender.VValue as Gender, gender.VDesc as GenderText")
     ->join('valueset gender', 'gender.VID = patient.Gender', 'left')

After:

use App\Libraries\ValueSet;

$this->select("..., patient.Gender");
// After fetching:
$rows = ValueSet::transformLabels($rows, [
    'Gender' => 'patient_Sex',
    'Country' => 'patient_Country',
    'Race' => 'patient_Race',
    'Religion' => 'patient_Religion',
    'Ethnic' => 'patient_Ethnic',
    'DeathIndicator' => 'patient_DeathIndicator',
    'MaritalStatus' => 'patient_MaritalStatus',
]);

Models to Modify (8 files)

1. app/Models/Patient/PatientModel.php

Remove:

• Line 27: $this->join('valueset vs', 'vs.vid = Gender', 'left');
• Lines 52-64: All *.VID as *VID aliases
• Lines 75-81: All valueset.* joins

Add transformation in getPatient():

$patient = ValueSet::transformLabels([$patient], [
    'Gender' => 'patient_gender',
    'Country' => 'patient_country',
    'Race' => 'patient_race',
    'Religion' => 'patient_religion',
    'Ethnic' => 'patient_ethnic',
    'DeathIndicator' => 'patient_death_indicator',
    'MaritalStatus' => 'patient_marital_status',
])[0];

2. app/Models/Location/LocationModel.php

Remove:

• Lines 18, 30: ->join("valueset v", "v.VID=location.loctype", ...)

Add transformation:

$rows = ValueSet::transformLabels($rows, [
    'LocType' => 'location_LocationType',
]);

3. app/Models/Test/TestDefSiteModel.php

Remove:

• Lines 42, 75, 103: ->join("valueset", "valueset.VID=...")

Add transformation:

$rows = ValueSet::transformLabels($rows, [
    'TestType' => 'test_TestType',
]);

4. app/Models/Test/TestDefGrpModel.php

Remove:

• Line 32: ->join('valueset vs', 'vs.VID=t.TestType', 'left')

5. app/Models/Specimen/ContainerDefModel.php

Remove:

• Lines 20-22, 37-39: All 6 valueset.* joins

Add transformation:

$rows = ValueSet::transformLabels($rows, [
    'Color' => 'container_ContainerCapColor',
    'ConClass' => 'container_ContainerClass',
    'Additive' => 'container_Additive',
]);

6. app/Models/Organization/SiteModel.php

Remove:

• Lines 38-39: ->join('valueset sitetype'...) and ->join('valueset siteclass'...)

Add transformation:

$row = ValueSet::transformLabels([$row], [
    'SiteTypeID' => 'organization_SiteType',
    'SiteClassID' => 'organization_SiteClass',
])[0];

7. app/Models/Organization/AccountModel.php

Remove:

• Line 41: ->join('valueset country'...)

Remove from select:

• Line 36: country.VID as country

Add transformation in controller if needed:

$rows = ValueSet::transformLabels($rows, [
    'Country' => 'account_Country',
]);

8. app/Models/Organization/WorkstationModel.php

Remove:

• Lines 36-37: ->join('valueset wstype'...) and ->join('valueset enable'...)

Add transformation:

$row = ValueSet::transformLabels([$row], [
    'Type' => 'organization_WorkstationType',
    'Enable' => 'organization_EnableDisable',
])[0];

---

Phase 5: Controller Changes

app/Controllers/TestsController.php

Remove:

• Line 69: ->join("valueset", "valueset.VID=testdefsite.TestType", "left")
• Line 111: ->join("valueset", "valueset.VID=testdefsite.TestType", "left")
• Line 140: ->join('valueset vs', 'vs.VID=t.TestType', 'left')

Replace getVValue() method:

private function getVValue($vsetID, $vid) {
    // DEPRECATED - Use ValueSet::getLabel() instead
    return null;
}

Update references from getVValue() to ValueSet::getLabel():

// Before:
'NumRefTypeVValue' => $this->getVValue(46, $r['NumRefType']),

// After:
'NumRefTypeVValue' => \App\Libraries\ValueSet::getLabel('ref_NumericRefType', $r['NumRefType']),

VSetID to Lookup Name Mapping:

VSetID	Constant	Lookup Name
44	VALUESET_REF_TYPE	test_ReferenceType
45	VALUESET_RANGE_TYPE	ref_RangeType
46	VALUESET_NUM_REF_TYPE	ref_NumericRefType
47	VALUESET_TXT_REF_TYPE	ref_TextRefType
3	VALUESET_SEX	patient_Sex
41	VALUESET_MATH_SIGN	ref_MathSign

Update getValuesetOptions() to use JSON:

private function getValuesetOptions($lookupName)
{
    return \App\Libraries\ValueSet::getOptions($lookupName);
}

---

Phase 6: API Endpoints - Replace with JSON-based endpoints

New API Controller: app/Controllers/ValueSetApiController.php

<?php

namespace App\Controllers;

use App\Libraries\ValueSet;

class ValueSetApiController extends \CodeIgniter\Controller
{
    use \CodeIgniter\API\ResponseTrait;

    public function index(string $lookupName)
    {
        $data = ValueSet::getOptions($lookupName);
        return $this->respond([
            'status' => 'success',
            'data' => $data
        ], 200);
    }

    public function all()
    {
        $dir = APPPATH . 'Libraries/Data/valuesets/';
        $files = glob($dir . '*.json');
        $result = [];
        foreach ($files as $file) {
            $name = basename($file, '.json');
            $result[] = [
                'name' => $name,
                'options' => ValueSet::getOptions($name)
            ];
        }
        return $this->respond([
            'status' => 'success',
            'data' => $result
        ], 200);
    }
}

Update Routes: app/Config/Routes.php

$routes->group('api', function ($routes) {
    $routes->get('valueset/(:segment)', 'ValueSetApiController::index/$1');
    $routes->get('valueset', 'ValueSetApiController::all');
});

---

Phase 7: View Updates

1. app/Views/v2/master/valuesets/valuesets_index.php

Repurpose to manage JSON-based valuesets instead of database table.

Before	After
fetch(...api/valueset...)	fetch(...api/valueset/lookupName...)
Database CRUD operations	File-based CRUD operations

2. app/Views/v2/master/valuesets/valueset_nested_crud.php

Repurpose for JSON file management.

3. app/Views/v2/master/valuesets/valueset_dialog.php

Update for JSON file format.

4. app/Views/v2/master/tests/tests_index.php

Before	After
type?.VID	type?.key
type?.VValue	type?.value
type?.VDesc	type?.label
{ VID: 1, VValue: 'TEST', ... }	{ key: 'TEST', value: 'Test', ... }
getTypeName(vid)	getTypeName(value)
api/valuesetdef/27	api/valueset/test_TestType
Hardcoded fallback: { VID: 1, VValue: 'TEST', VDesc: 'Test' }	{ key: 'TEST', value: 'Test' }

5. Additional Views to Update

View File	Fields to Update	Lookup Name
app/Views/v2/patients/patients_index.php	Gender, Country, Race, Religion, Ethnic, MaritalStatus, DeathIndicator	patient_*
app/Views/v2/master/specimen/containers_index.php	Color, ConClass, Additive	container_*
app/Views/v2/master/organization/sites_index.php	SiteTypeID, SiteClassID	organization_*
app/Views/v2/master/organization/workstations_index.php	Type, Enable	organization_*
app/Views/v2/master/organization/accounts_index.php	Country	account_Country
app/Views/v2/master/organization/locations_index.php	LocType	location_LocationType

---

Phase 8: Test Files Update

tests/feature/ValueSet/ValueSetApiControllerTest.php

Update tests to use new JSON-based API endpoints.

tests/_support/v2/MasterTestCase.php

Update any valueset-related test data setup.

---

Testing Checklist

1. Unit Tests

   • Test ValueSet::getLabel('patient_Sex', '1') returns 'Female'
   • Test ValueSet::getLabel('test_TestType', 'TEST') returns 'Test'
   • Test ValueSet::getOptions('container_ContainerCapColor') returns correct format
   • Test ValueSet::transformLabels() with table-prefixed field mappings

2. Integration Tests

   • Patient CRUD (create, read, update, delete)
   • Test definition CRUD
   • Location CRUD
   • Container definition CRUD
   • Organization (site, account, workstation) CRUD

3. Manual Testing

   • Verify all dropdowns display correct labels
   • Verify filtering by valueset fields works
   • Verify form submissions save correct VValue codes

---

Rollback Plan

1. Run migration down() to revert column types
2. Restore deleted seeders from git if needed
3. Restore deleted migration file from git

---

Files Summary

Phase	Action	Files
1	RENAME	~50 JSON files in app/Libraries/Data/valuesets/
2	DELETE	5 seeders, 1 migration
2	CREATE	1 migration (column changes)
3	UPDATE	1 library (ValueSet.php), ValueSetModel.php deprecation
4	UPDATE	8 Model files
5	UPDATE	1 Controller, Routes
6	CREATE	1 Controller (ValueSetApiController.php)
6	UPDATE	~6+ View files
8	UPDATE	2+ Test files

---

Estimated Effort

• Phase 1 (JSON Rename): 15 minutes
• Phase 2 (Migration): 30 minutes
• Phase 3 (Library + Model): 30 minutes
• Phase 4 (Models): 1.5 hours
• Phase 5 (Controller): 30 minutes
• Phase 6 (API + Views): 2 hours
• Phase 8 (Tests): 30 minutes
• Testing: 1 hour

Total Estimated Time: ~7 hours