Transforms & Masking

Transformation happens in memory, after materialization— never in SQL. Filtering and sorting still run against the real values in the database; what reaches the caller is changed on the way out.

Detached, then transformed

Entities returned from a guarded query are detached before anything is changed. If they were tracked, the masked value would be a pending modification and the next unrelated SaveChanges on the same context would write asterisks over the real data — silently, with no exception and no way back without a restore.

A transform on a query the caller materializes itself is refused with TransformRequiresMaterialization rather than skipped, so a composable method cannot hand back an IQueryable that quietly never masks anything. Since 3.3.0 that refusal asks what a row of the type can hold as well as which paths the policy names: a type whose only transform sits where no path reaches it — on a member only a subtype declares, one five segments down, one of an object a dictionary holds — was handed the query, and its rows came back exactly as stored. With no named column to list, the refusal names the clause, FieldPath "*", in both tiers. A type nothing transforms anywhere still gets its query.

The nine mask strategies

[DwMask(MaskStrategy.Partial, KeepEnd = 4)]
public string CardNumber { get; set; }        // ************4242

[DwMask(MaskStrategy.Email)]
public string Email { get; set; }             // s*************@c******.com

[DwMask(MaskStrategy.Hash)]
public string NationalId { get; set; }        // stable per salt, useful for joins

[DwMask(MaskStrategy.Tokenize)]
public string PassportNumber { get; set; }    // stable per vault, useful for joins

Hashing against tokenizing

Both keep a column groupable and joinable while hiding what is in it, and both do it by mapping one value to one output. The difference is where the secret lives, and it decides what an attacker has to reach to undo the mask.

A hash is computed, so whoever holds the salt can recompute every digest the deployment has ever emitted, and a guessable salt is recovered offline. A token is drawn at random the first time a value is seen and written into a vault, so the only way back is to read that vault — a store you can lock, move and revoke separately from the data. Guard it as you would guard the column it protects, and give it a key.

new DwPolicyOptions
{
    HashSalt   = secret,                      // 16 characters or more
    TokenVault = new RedisTokenVault(redis)   // or EfTokenVault, or InMemoryTokenVault
}

Three vaults ship and all three pass one conformance suite. InMemoryTokenVault lives and dies with the process, which is right for a test and wrong for any column compared across restarts. RedisTokenVault and EfTokenVault keep the mapping outside the process and cache every mapping they resolve, which they can do safely because a token is written once and never rewritten.

Give a durable vault a key (3.3.0)

A vault stores its mapping under the scope and a digest of the value. Without a key that digest is a plain SHA-256, and a tokenized column is nearly always drawn from a space small enough to hash whole — phone numbers, national identifiers, card numbers. So a copy of the store, a backup or a replica or a dump, gives back every value in it, and with them the value behind every token ever issued. Under a key held where the store is not — configuration, a secret manager — the digest is an HMAC-SHA256, and the store and the key have to be taken together.

new DwPolicyOptions
{
    TokenVault = new RedisTokenVault(redis, key)              // 16 bytes or more
    // TokenVault = new EfTokenVault(() => new AppDbContext(opts), key)
}

The constructors that take no key are unchanged and unkeyed. InMemoryTokenVault draws a random 32-byte key of its own per instance — nothing to configure, and no API change — because its mappings die with the process anyway. The key must be at least DwToken.MinimumKeyLength (16) bytes; a shorter one is refused by the constructor, as a short hash salt is refused. A keyed mapping's key starts with DwToken.KeyedPrefix ("hmac:"), so an operator can tell the two kinds apart in a store holding both. The scope sits inside the digest as well as in front of it, so one value tokenized in two scopes is two unrelated keys.

The cost is small and there is no schema change. Redis reads both fields in one round trip, so a value new to the store costs two round trips instead of one; EF Core costs one more read for a new value, and in retire mode one more read per first-met value. A keyed key is at most 326 characters against the 512 the Key column already holds.

Tokens are namespaced by TokenScope, and where none is given the namespace falls back to the field's path relative to the entity being queried — with no type name in it. Two columns reached by different paths therefore get different tokens for the same value, and two entities whose member path is spelled identically share them, whether or not anybody decided they should. Name the scope explicitly on both sides when you want the match — at the cost of telling a caller the two rows concern the same subject — and name a distinct one on each when you do not.

The other five

What can be applied to what

Masking, truncation and formatting all emit text, so they cannot be assigned to a decimal or a DateTime. Startup validation reports this as an error and names the fix.

Employee.Salary: [DwMask] emits text, which cannot be assigned to Decimal.
Use [DwGeneralize] to reduce a number or a date while keeping its type, or
project into a type whose member is a string.

Chaining

Several transforms on one member compose into a chain, elected one stage at a time. A runtime rule can add a stage on top of a sealed one and cannot replace it.

[DwGeneralize(GeneralizeMode.Bucket, Step = 5000)]   // 45000-49999
[DwTruncate(5, Ellipsis = "+")]                      // 45000+
public string SalaryBand { get; set; } = string.Empty;

The member is a string because both stages emit text, and startup validation refuses a chain that emits text into a member that cannot hold it. Reduce a number while keeping its type with [DwGeneralize] alone.

Through the graph

The walk descends through reference navigations, collections, arrays, interfaces, structs and jagged collections, transforming every element it reaches. It is driven by the paths the policy names — the declared types, four segments deep — which is what makes it incapable of missing a path because it failed to recognise a navigation.

A value no path of the policy names

A value can sit in the materialized rows where none of those paths goes. A [DwMask] member five segments down an included or in-memory graph, one only a subtype of the row's type declares — Dog.Chip on rows typed Animal, in memory or in a TPH hierarchy — one on an object a dictionary holds, and the far side of a cycle: each came back exactly as stored, at the default caps, under Strict, with no Selects, with the navigation named whole in Selects, and in a dynamic projection holding a real object.

• Only members that declare a transform or an audit for Select, or that can lead to one, are read. A navigation whose type can reach neither is never touched, so a lazy loader behind it is not woken, and a model that declares neither anywhere pays for no second pass at all.
• The same pass reports each audited member it meets where the policy names no path to it, which the terminal records as a read — see breaking point 43.
• The transform is the member's own attributes. No rule can speak to such a member, since no path names it — the same answer as no runtime rule can unmask a field — and a resolver built over no AttributePolicyProvider reads no attribute here either.
• It obeys Selects as the first pass does, runs in a dry run as transforms always have, and fails the query with InvalidOperationException for a transformed member with no setter, exactly as one along a named path does.
• The trace records the path with its stages and the note (declared on the member; no path of the policy names it).
• Still a limit: a member typed object, or a collection that is not generic, says nothing about what it holds and is not read into.

A transform is applied to a member, which is why a path beneath one — Bonus.Value, a decimal where Bonus is what is rounded — is refused for Select, Group and Aggregate instead: there is no member there to apply the chain to, and the value it would hand back is the stored one. A transformed member past four segments, which only a raised MaxNavigationDepth lets a request name, is a member, so naming it in Selects returns it transformed; only a grouping key and an aggregated field are refused there, because a summary's own transform finds a generated row's columns by the type's list, and that list stops at four segments.