Bio-Cat CRM Migration — Executive Overview

Entity Types

Migration Phases

150+

Fields Mapped

Safety Modes

What Is This Migration?

Bio-Cat is moving its sales pipeline data from the legacy Microsoft CRM 2011 on-premises system to Microsoft Dynamics 365 (D365) in the cloud. This automated migration transfers Accounts, Contacts, Leads, Opportunities, and Quotes — preserving all relationships, history, owner assignments, and business data.

Every migrated record is tagged with a legacy ID, a direct link back to the original CRM 2011 record, and a batch ID so that any migration run can be fully audited or reversed.

High-Level Migration Flow

flowchart LR
    subgraph Source["CRM 2011 (On-Premises)"]
        A1[Accounts]
        C1[Contacts]
        L1[Leads]
        O1[Opportunities]
        Q1[Quotes]
    end

    subgraph Engine["Migration Engine"]
        direction TB
        V[Validate Metadata]
        T[Transform & Map Fields]
        R[Resolve Relationships]
    end

    subgraph Target["Dynamics 365 (Cloud)"]
        A2[Accounts]
        C2[Contacts]
        L2[Leads]
        O2[Opportunities + Line Items]
        Q2[Quotes]
    end

    A1 --> Engine
    C1 --> Engine
    L1 --> Engine
    O1 --> Engine
    Q1 --> Engine

    Engine --> A2
    Engine --> C2
    Engine --> L2
    Engine --> O2
    Engine --> Q2

Migration Phases & Dependencies

Entities must be migrated in a specific order because later phases depend on records created in earlier phases.

flowchart TB
    P1["Phase 1 — Account Sync"]
    P2["Phase 2 — Contact Migration"]
    P3["Phase 3 — Lead Migration"]
    P4["Phase 4 — Opportunity Migration"]
    P5["Phase 5 — Quote Migration"]

    P1 -->|"Accounts must exist for contact linking"| P2
    P2 -->|"Contacts must exist for opportunity linking"| P4
    P1 -->|"Accounts must exist for lead linking"| P3
    P1 -->|"Accounts must exist for opportunity linking"| P4
    P3 -->|"Leads may be referenced by opportunities"| P4
    P4 -->|"Opportunities must exist for quote linking"| P5
    P1 -->|"Accounts must exist for quote linking"| P5

    style P1 fill:#3498db,stroke:#2c3e50,color:#fff
    style P2 fill:#2ecc71,stroke:#2c3e50,color:#fff
    style P3 fill:#e67e22,stroke:#2c3e50,color:#fff
    style P4 fill:#9b59b6,stroke:#2c3e50,color:#fff
    style P5 fill:#e74c3c,stroke:#2c3e50,color:#fff

Safety Model

Every migration script supports three execution modes to prevent accidental data changes:

Mode	Flag	Behavior
Verify Only	`--verify-only`	Validates field mappings against live metadata in both systems. No data is read or written.
Dry Run (default)	`--dry-run`	Reads source data, builds payloads, resolves relationships, and reports statistics — but writes nothing to D365.
Commit	`--commit`	Performs the actual migration, creating/updating records in D365. Supports `--limit N` to process a controlled number of records.

Rollback: Every migration batch is tagged with a unique Batch ID. The interactive cleanup mode can list all batches and selectively delete any batch — fully reversing that migration run.

Phase 1 — Account Sync

Purpose: Synchronize account data (company names, addresses, phone numbers, owners) from CRM 2011 into matching D365 accounts. Accounts that exist in both systems are updated; accounts that only exist in CRM 2011 and are needed by later phases are created on-demand.

flowchart TB
    F["Fetch all CRM 2011 Accounts"]
    M1{"Match by Account Number?"}
    M2{"Match by Account Name?"}
    S["Sync all fields to D365 — PATCH update"]
    SK["Skip — no D365 match"]
    OW["Resolve Owner via systemuser lookup"]

    F --> M1
    M1 -->|Yes| OW
    M1 -->|No| M2
    M2 -->|Yes| OW
    M2 -->|No| SK
    OW --> S

    style F fill:#3498db,color:#fff
    style S fill:#27ae60,color:#fff
    style SK fill:#95a5a6,color:#fff

Technical Details — Account Sync

Matching Logic

Each CRM 2011 account is matched to D365 using a two-step lookup:

Account Number — exact OData filter on accountnumber
Account Name — fallback exact filter on name

If neither matches, the account is skipped during the sync phase. It may be auto-created later during contact or opportunity migration if needed.

Owner Resolution

All active D365 system users are cached at startup (filtered by isdisabled eq false). The CRM 2011 owner name is matched case-insensitively. Known former employees are routed to the default owner (Phil Smoker) via a configurable override map.

Fields Synced

Category	Fields
Core	Name, AccountNumber, SIC, Description
Communication	Telephone1–3, Fax, EMailAddress1–3, WebSiteURL
Address 1 (Primary)	Line1–3, City, State, PostalCode, Country, County, Name, Phone, Fax
Address 2 (Other)	Line1–3, City, State, PostalCode, Country, County, Name, Phone, Fax
Preferences	DoNotEmail, DoNotPhone, DoNotFax, DoNotBulkEMail, DoNotPostalMail
Legacy Tracking	kl_legacyaccountid, kl_legacyuri, kl_migrationbatchid

Auto-Creation (On-Demand)

When a later phase references a CRM 2011 account that doesn't exist in D365:

Only active CRM 2011 accounts (StateCode = 0) are created
Created accounts are tagged with [Created by Bio-Cat Migration] in the description
A legacy ID and direct CRM 2011 URL are stored for auditability

Cleanup

Migration-created accounts can be found by the description tag and deleted in bulk. A corrective pass also clears any accidentally backfilled batch IDs on pre-existing accounts.

Phase 2 — Contact Migration

Purpose: Migrate all active contacts from CRM 2011 to D365, linking each contact to its parent account. If a contact's parent account doesn't yet exist in D365, it is auto-created.

flowchart TB
    F["Fetch active CRM 2011 Contacts — StateCode = 0"]
    V["Validate field mappings against D365 metadata"]
    PA["Resolve Parent Account by AccountNumber or Name"]
    CA{"Account in D365?"}
    CR["Auto-create Account"]
    OW["Resolve Owner"]
    B["Build D365 payload — 59 mapped fields"]
    W["Create Contact in D365"]

    F --> V --> PA
    PA --> CA
    CA -->|Yes| OW
    CA -->|No| CR --> OW
    OW --> B --> W

    style F fill:#2ecc71,color:#fff
    style W fill:#27ae60,color:#fff
    style CR fill:#f39c12,color:#fff

Technical Details — Contact Migration

Scope

Only active contacts are migrated (CRM 2011 StateCode/Value eq 0).

Parent Account Resolution

The contact's ParentCustomerId lookup is read. If it references an account, the shared account resolution logic finds or creates the D365 account (same as Phase 1). Results are cached within the run to avoid repeated API calls.

Key Fields (59 total)

Category	Fields
Name	FirstName, LastName, MiddleName, Salutation, Suffix, NickName, JobTitle
Communication	Telephone1–3, MobilePhone, Fax, EMailAddress1–3, WebSiteUrl
Address 1	Line1–3, City, StateOrProvince, PostalCode, Country, County
Address 2	Line1–3, City, StateOrProvince, PostalCode, Country, County
Preferences	DoNotEmail, DoNotPhone, DoNotFax, DoNotBulkEMail, DoNotPostalMail, DoNotSendMM
Company Info	Company, Department, AssistantName, AssistantPhone, ManagerName, ManagerPhone
Social	SpousesName, Anniversary, BirthDate, GenderCode, FamilyStatusCode
Legacy Tracking	kl_legacycontactid, kl_legacyuri, kl_migrationbatchid

Batch Tracking

Each run generates a unique batch ID (UUID). All created contacts are tagged. The batch ID is saved to contact_migration_last_batch.json for cleanup reference.

Phase 3 — Lead Migration

Purpose: Migrate all open leads from CRM 2011 to D365, linking each lead to its parent account and contact (if they exist in D365). Unlike earlier phases, leads do not auto-create missing accounts or contacts — they link only if the parent already exists.

flowchart TB
    F["Fetch open CRM 2011 Leads — StateCode = 0"]
    V["Validate 72 field mappings against D365 metadata"]
    PA["Look up Parent Account by legacy ID in D365"]
    PC["Look up Parent Contact by legacy ID in D365"]
    OW["Resolve Owner"]
    T["Transform and Coerce: type conversion, picklists, booleans, dates"]
    W["Create Lead in D365"]

    F --> V
    V --> PA
    V --> PC
    PA --> OW
    PC --> OW
    OW --> T --> W

    style F fill:#e67e22,color:#fff
    style W fill:#27ae60,color:#fff

Technical Details — Lead Migration

Scope

Only open/active leads (StateCode = 0) are migrated.

Parent Linking (Read-Only)

Leads attempt to link to existing D365 accounts and contacts by querying on the legacy ID fields (kl_legacyaccountid, kl_legacycontactid). If the parent record wasn't migrated in a previous phase, the lead is created without the link (a warning is logged).

Field Mappings (72 fields)

Category	Fields
Core	Subject, FirstName, LastName, MiddleName, Salutation, JobTitle, Company
Communication	Telephone1–3, MobilePhone, Fax, EMailAddress1–3, WebSiteUrl, Pager
Address 1 & 2	Full address sets (Line1–3, City, State, PostalCode, Country, County)
Business	Revenue, NumberOfEmployees, SIC, IndustryCode, LeadSourceCode, Description
Preferences	DoNotEmail, DoNotPhone, DoNotFax, DoNotBulkEMail, DoNotPostalMail, DoNotSendMM
Legacy Tracking	kl_legacyleadid, kl_legacyuri, kl_migrationbatchid

Value Type Handling

CRM 2011 Type	Conversion
Picklist / OptionSet	→ Integer (D365 picklist value)
Boolean	→ Python bool
DateTime	→ ISO 8601 string (handles `/Date(ms)/` format)
Money	→ Float (unwrapped from `{"Value": n}`)
String / Memo	→ Stripped strings

Error Recovery

If D365 rejects a field (invalid property or out-of-range value), the migration automatically removes that field from the payload and retries — up to 5 attempts per record. This ensures a single bad field doesn't block the entire record.

Phase 4 — Opportunity Migration

Purpose: Migrate all opportunities from CRM 2011 to D365, including their line items (products). This is the most complex phase — it resolves accounts, contacts, owners, transforms option sets, concatenates formula fields, and handles monetary values.

flowchart TB
    F["Fetch CRM 2011 Opportunities"]
    V["Validate mappings against both CRM metadata"]
    OW["Resolve Owner via systemuser cache"]
    AC["Resolve Account — auto-create if missing"]
    CO["Resolve Contact — auto-create if missing"]
    T["Transform Fields"]
    BP["Build D365 Payload"]
    CR["Create Opportunity in D365"]
    LI["Fetch and Migrate Line Items as write-in products"]

    F --> V --> OW
    OW --> AC --> CO
    CO --> T --> BP --> CR --> LI

    style F fill:#9b59b6,color:#fff
    style CR fill:#27ae60,color:#fff
    style LI fill:#8e44ad,color:#fff

Technical Details — Opportunity Field Mappings

Field Mapping Summary

CRM 2011 Field	D365 Field	Transform
CFSProduct_Name	name (Topic)	Appends 🔄️ suffix
CFPRequested_by	kl_requestedby	Picklist → systemuser GUID
CFSSample_Amount	kl_sampleamount	Direct
budgetamount	budgetamount	Money (unwrap)
new_allergen	kl_all_allergens	OptionSet (identical values)
CFPCertificate_of_Analysis	kl_examplecofa	Picklist → Boolean
CFPCF_Industry	kl_industry	Int remapped (e.g., 2→34)
CFPMSDS	kl_msds, kl_safetydatasheet	Picklist → Boolean
new_clinicaltrial	kl_clinicaltrial	Picklist → Boolean
new_sorametalstesting	kl_sorametalstesting	Picklist → Boolean
new_specificationsheet	kl_specificationsheet	Picklist → Boolean
CFPNew_Customer	kl_legacynewcustomer	Picklist → Label text
new_productstatus	kl_legacyproductstatus	Picklist → Label text
CFPUnit_of_Measure	kl_legacyuom	Picklist → Label text
CFPRequested_by	kl_legacyrequestedby	Picklist → Name text
CFPCF_Industry	kl_legacyindustry	Picklist → Label text
cfsformula_1..20 + cfmother	kl_legacyformula	Concatenated (newline-joined)

Lookup Bindings (OData @odata.bind)

Target Field	Navigation Property	Entity Set
transactioncurrencyid	transactioncurrencyid	transactioncurrencies
originatingleadid	originatingleadid	leads
kl_requestedby	kl_RequestedBy	systemusers

Always-Set Fields

D365 Field	Value
Price List	"Standard" (GUID from config)
Currency	US Dollar (GUID from config)
kl_legacyopportunityid	Source OpportunityId GUID
kl_legacyuri	Direct URL to CRM 2011 record
kl_migrationbatchid	Batch ID (hashed to int)

Technical Details — Line Items (Opportunity Products)

After each opportunity is created, its CRM 2011 line items are fetched and migrated as write-in products (not catalog items).

Line Item Configuration

Setting	Value	Purpose
isproductoverridden	true	Marks as write-in (custom) product
ispriceoverridden	true	Allows manual pricing
skippricecalculation	1 (Do Not Calculate)	Prevents D365 pricing engine errors

Line Item Fields

CRM 2011	D365
ProductId.Name	productdescription (write-in product name)
Quantity	quantity
PricePerUnit	priceperunit
Description	description
Tax	tax
ManualDiscountAmount	manualdiscountamount
LineItemNumber	lineitemnumber
UoMId.Name	kl_uom (picklist: KG→102770000, LB→102770001, etc.)

Technical Details — Data Transform Pipeline

Every field value passes through a multi-stage transform pipeline:

flowchart TB
    RAW["Raw CRM 2011 Value"]
    N["1. Normalize: Unwrap Money, OptionSet, preserve Lookups"]
    T["2. Transform: Option set remap, Picklist to Boolean, Picklist to Label, UoM crosswalk, Requested By to GUID"]
    C["3. Coerce to D365 Type: String, Int, Float, Bool, DateTime, Picklist, Money"]
    D365["D365 Payload Value"]

    RAW --> N --> T --> C --> D365

Industry Value Crosswalk (Example)

CRM 2011 Value	Label	D365 Value
2	Animal Nutrition	34
3	Baking	35
4	Brewing & Alcohol Production	36
5	Cane/Beet Sugar Processing	37
6	Cleaning	38
7	Dairy	39
8	Dietary Supplements	40
100000000	Probiotics	100000001
100000001	Aquaculture - Feed	100000000

Picklist → Boolean Conversion

CRM 2011 Picklist Value	Meaning	D365 Boolean
0	No (Pattern B)	false
1	Yes (Pattern B)	true
2	Yes (Pattern A)	true
3	No (Pattern A)	false
5	Deleted option	skip field

Food Grade Toggle

CRM 2011 Picklist Value	Meaning	D365 kl_foodgradetoggle
1	Default Value	skip (no equivalent)
2	Yes	true
3	No	false

Phase 5 — Quote Migration

Purpose: Migrate active quotes from CRM 2011 to D365, linking each quote to its parent account and originating opportunity. Quotes use a different industry option set than opportunities and include separate formula fields.

flowchart TB
    F["Fetch active CRM 2011 Quotes — StateCode = 1"]
    V["Validate 32 field mappings"]
    AC["Resolve Account — auto-create if missing"]
    OP["Resolve Opportunity by legacy ID in D365"]
    OW["Resolve Owner"]
    T["Transform Fields: Quote-specific Industry map, shared picklist/bool/label logic, formula concatenation"]
    W["Create Quote in D365"]

    F --> V --> OW
    OW --> AC
    OW --> OP
    AC --> T
    OP --> T
    T --> W

    style F fill:#e74c3c,color:#fff
    style W fill:#27ae60,color:#fff

Technical Details — Quote Migration

Quote-Specific Differences from Opportunities

Aspect	Opportunity	Quote
Industry Option Set	INDUSTRY_VALUE_MAP (int → int)	QUOTE_INDUSTRY_LABELS (int → label text)
Formula Fields	cfsformula_1..20 + cfmother	CFSFormula_1..16 + CFMFormula
Line Items	Migrated as write-in products	Not migrated (headers only)
State Filter	All states	Active only (StateCode = 1)
Opportunity Link	N/A	Resolved by legacy ID lookup
Food Grade	kl_foodgradetoggle (Yes/No boolean)	kl_foodgradetoggle (Yes/No boolean)

Key Fields (32 total)

Category	Fields
Summary	Name, RequestedActivity, MinimumOrderRequirement, EffectiveFrom/To, Description
Line Items	Allergens, FoodGradeToggle (Yes/No), KosherCertification, HalalCertification, CertificateOfAnalysis
Bill-To Address	Line1–3, City, State, PostalCode, Country
Ship-To Address	Line1–3, City, State, PostalCode, Country
Legacy	Contact, UoM, PricePerUnit, NewLead, Industry, RequestedActivity, MinOrderReq, CreatedBy
Tracking	kl_legacyquoteid, kl_legacyuri, kl_migrationbatchid

Opportunity Resolution

Quotes are linked to their originating D365 opportunity by querying kl_legacyopportunityid with the CRM 2011 opportunity GUID. If the opportunity hasn't been migrated, the quote is created without the link (warning logged).

Quote Industry Labels (Separate Option Set)

CRM 2011 Value	D365 Label
100000000	Animal Nutrition
100000001	Agriculture
100000002	Aquaculture - Feed
100000003	Baking
100000004	Brewing & Alcohol Production
100000005	Cane Beet Sugar Processing
100000006	Cleaning
100000007	Dairy
100000008	Dietary Supplements
100000009	Food, Beverage & Ingredients
100000010	Fruit/Vegetable Processing
100000011	Probiotics
100000012	Protein Processing
100000013	Septic & Drain Care
100000014	Waste Treatment
100000015	Other
100000016	Item
100000017	Aquaculture - Other
100000018	Silver Recovery

Audit Trail & Traceability

Every migrated record carries three audit fields, enabling full traceability between the old and new systems:

Audit Field	Purpose	Example
Legacy ID	The original CRM 2011 record GUID	`A1B2C3D4-E5F6-7890-...`
Legacy URI	Direct clickable link to the CRM 2011 record	`http://192.168.0.44/BioCat/main.aspx?etc=3&id=...`
Batch ID	Groups all records from a single migration run	`f47ac10b-58cc-4372-...`

Legacy Record Links: Every migrated D365 record contains a direct URL back to the original CRM 2011 record. This allows staff to quickly cross-reference historical data during the transition period.

Technical Details — Legacy Fields by Entity

Entity	Legacy ID Field	Legacy URI Field	Batch Tag Field	CRM 2011 Entity Code
Account	kl_legacyaccountid	kl_legacyuri	kl_migrationbatchid	etc=1
Contact	kl_legacycontactid	kl_legacyuri	kl_migrationbatchid	etc=2
Lead	kl_legacyleadid	kl_legacyuri	kl_migrationbatchid	etc=4
Opportunity	kl_legacyopportunityid	kl_legacyuri	kl_migrationbatchid	etc=3
Quote	kl_legacyquoteid	kl_legacyuri	kl_migrationbatchid	etc=1084

Error Handling & Resilience

Scenario	Response
D365 rejects a field (invalid property)	Field is automatically removed and the record is retried (up to 5 attempts)
Field value out of range	Field is removed and retried — remaining fields are preserved
Owner has no D365 security roles	Falls back to default owner (Phil Smoker) and tags description with intended owner
Owner no longer exists in D365	Mapped via known-former-employee overrides, or defaults to Phil Smoker
Parent account not found	Auto-created in D365 (tagged for cleanup tracking)
Parent contact not found	Auto-created during opportunity phase; leads skip unresolvable parents
Line item creation fails	Warning logged; opportunity is still created successfully
CRM 2011 field list rejected	Retries without `$select` clause (fetches all fields)

Technical Details — Owner Resolution

flowchart TB
    SRC["CRM 2011 OwnerId lookup"]
    NAME["Extract owner name"]
    CACHE{"Name in D365 systemuser cache?"}
    OVER{"Name in former-employee overrides?"}
    DEF["Use default: Phil Smoker"]
    FOUND["Use matched D365 systemuser GUID"]
    OVR["Use override GUID"]

    SRC --> NAME --> CACHE
    CACHE -->|Yes| FOUND
    CACHE -->|No| OVER
    OVER -->|Yes| OVR
    OVER -->|No| DEF

    style FOUND fill:#27ae60,color:#fff
    style OVR fill:#f39c12,color:#fff
    style DEF fill:#e74c3c,color:#fff

Known Former Employee Overrides

Former Employee	Assigned To
Charlie Pettygrove	Phil Smoker (default owner)
Marc Jensen	Phil Smoker (default owner)
Maggie Murphy	Phil Smoker (default owner)

How It Runs

The migration suite runs inside a Docker container for environment consistency. An interactive launcher (run.sh) presents a menu-driven interface:

flowchart LR
    L["Interactive Launcher — run.sh"]
    D["Docker Container"]
    M["Menu Options"]

    L --> D --> M

#	Menu Option
1	Verify CRM/D365 connections
2	Migrate contacts
3	Migrate opportunities
4	Migrate leads
5	Migrate quotes
6	Validate migrated opportunity data
7–11	Cleanup options (contacts, opportunities, leads, quotes, accounts)
12	Sync accounts from CRM 2011 to D365

Each option prompts for mode selection (Dry Run / Verify Only / Commit), optional record limits, and batch IDs. The fully-formed command is displayed before execution for transparency.

Connection Architecture

Technical Details — Authentication & Connectivity

System	Protocol	Authentication
CRM 2011	OData v2 (OrganizationData.svc)	Auto-detected: Basic Auth or NTLM
Dynamics 365	Web API v9.2 (OData v4)	MSAL Public Client flow (username/password with device code fallback)

D365 authentication mirrors the XrmToolBox public client configuration, using MSAL with client ID 51f81489-12ee-4a9e-aaae-a2591f45987d. Tokens are cached and refreshed silently when possible.

Connection Flow

sequenceDiagram
    participant Script as Migration Script
    participant CRM as CRM 2011
    participant AAD as Azure AD
    participant D365 as Dynamics 365

    Script->>CRM: OData request (Basic/NTLM auto-detect)
    CRM-->>Script: Account/Contact/Lead/Opp/Quote data

    Script->>AAD: MSAL token request (username/password)
    AAD-->>Script: Access token (cached for reuse)

    Script->>D365: Web API request (Bearer token)
    D365-->>Script: Created/Updated record confirmation