When building APIs, controlling what data gets sent back to clients is crucial for security and performance. We’ve entered a speed vs stability era of software development where it’s a constant trade-off between querying all the data you need to reduce the number of calls to the database which can result in more data than needed being returned to the client. One option is to allow for field optionality or pointers to allow for a nil state. Alternatively, you can manipulate the resulting object before, during or after the marshaling step. Let’s explore how to create a flexible sanitization system using reflection in Go.

The Problem

Consider this common scenario: You have a User struct with sensitive information that shouldn’t be exposed in certain cases. Different endpoints might need different views of the same data:

type User struct { 
    ID        int    `json:"id"` 
    Email     string `json:"email"` 
    Password  string `json:"password"` 
    APIKey    string `json:"api_key"` 
    Metadata  map[string]string `json:"metadata"` 
}

You might want to:

  • Hide the password and API key in public endpoints
  • Include the API key but not the password in developer endpoints
  • Remove specific metadata fields for certain clients
  • Sanitize the object before logging

Building a Dynamic Sanitizer

While there are options to handle this like using the data you need and then changing or setting the value, in some cases the result includes the object key with an empty or null value. This can result in more data being sent than necessary or result in data being rendered incorrectly depending on the calling client.

To solve this, we will introduce a scrubber.OfFields utility.

package scrubber 
 
import ( 
 "reflect" 
) 
 
// OfFields creates a map from a struct or map with specified fields removed, supporting dot-notation for nested fields. 
func OfFields(input any, fields ...string) map[string]any { 
 // Parse fields into a map for quick lookup 
 removeFields := make(map[string]struct{}) 
 for _, field := range fields { 
  removeFields[field] = struct{}{} 
 } 
 
 return process(input, removeFields, "") 
} 
 
// process handles both structs and maps recursively and removes specified fields. 
func process(input any, removeFields map[string]struct{}, parent string) map[string]any { 
 
 v := reflect.ValueOf(input) 
 if v.Kind() == reflect.Ptr { 
  v = v.Elem() 
 } 
 
 switch v.Kind() { 
 case reflect.Struct: 
  return processStruct(v, removeFields, parent) 
 case reflect.Map: 
  return processMap(v, removeFields, parent) 
 default: 
  panic("RemoveFields: input must be a struct or map") 
 } 
} 
 
// processStruct processes a struct recursively and removes specified fields. 
func processStruct(v reflect.Value, removeFields map[string]struct{}, parent string) map[string]any { 
 output := make(map[string]any) 
 
 for i := 0; i < v.NumField(); i++ { 
  field := v.Type().Field(i) 
  fieldValue := v.Field(i) 
  fieldName := field.Tag.Get("json") 
  if fieldName == "" { 
   fieldName = field.Name 
  } 
 
  fullFieldName := fieldName 
  if parent != "" { 
   fullFieldName = parent + "." + fieldName 
  } 
 
  if _, found := removeFields[fullFieldName]; found { 
   continue 
  } 
 
  if fieldValue.Kind() == reflect.Struct || (fieldValue.Kind() == reflect.Ptr && fieldValue.Elem().Kind() == reflect.Struct) { 
   output[fieldName] = processStruct(fieldValue, removeFields, fullFieldName) 
  } else if fieldValue.Kind() == reflect.Map { 
   output[fieldName] = processMap(fieldValue, removeFields, fullFieldName) 
  } else { 
   output[fieldName] = fieldValue.Interface() 
  } 
 } 
 
 return output 
} 
 
// processMap processes a map recursively and removes specified fields. 
func processMap(v reflect.Value, removeFields map[string]struct{}, parent string) map[string]any { 
 output := make(map[string]any) 
 
 for _, key := range v.MapKeys() { 
  keyStr, ok := key.Interface().(string) 
  if !ok { 
   panic("RemoveFields: map keys must be strings") 
  } 
 
  fieldValue := v.MapIndex(key).Interface() 
  fullFieldName := keyStr 
  if parent != "" { 
   fullFieldName = parent + "." + fullFieldName 
  } 
 
  if _, found := removeFields[fullFieldName]; found { 
   continue 
  } 
 
  // Check nested structures or maps 
  fieldValueReflect := reflect.ValueOf(fieldValue) 
  if fieldValueReflect.Kind() == reflect.Struct || fieldValueReflect.Kind() == reflect.Map { 
   nestedResult := process(fieldValue, removeFields, fullFieldName) 
   // Ensure the key exists in the output even if the nested map is empty 
   output[keyStr] = nestedResult 
  } else { 
   output[keyStr] = fieldValue 
  } 
 } 
 
 return output 
}

This package will take an input of a struct or map[string](any) type and loop through the object to remove the targeted fields and keys. It supports dotted.notation for nested object references, but mostly relies on the reflect package to identify struct fields and the matching of their JSON tags.

Implementing Into Your API

The scrubber is functional all on it’s own, however, we’ve proven the pattern when applying it as a middleware on your API or Web application. This this SanitizeResponse approach, we are able to JSON-decode the response from any route, sanitize the payload and re-encode it for response to the client.

func SanitizeResponse(fieldsToRemove ...string) func(http.Handler) http.Handler { 
    return func(next http.Handler) http.Handler { 
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { 
            // Create a response writer that captures the output 
            recorder := httptest.NewRecorder() 
             
            // Call the next handler 
            next.ServeHTTP(recorder, r) 
             
            // Get the response body 
            body := recorder.Body.Bytes() 
             
            // Parse the JSON response 
            var data interface{} 
            if err := json.Unmarshal(body, &data); err != nil { 
                // If not JSON, write original response and return 
                w.Write(body) 
                return 
            } 
             
            // Sanitize the response 
            sanitized := scrubber.OfFields(data, fieldsToRemove...) 
             
            // Write the sanitized response 
            w.Header().Set("Content-Type", "application/json") 
            json.NewEncoder(w).Encode(sanitized) 
        }) 
    } 
}

Using the Sanitizer

You can now apply different sanitization rules to different routes:

func main() { 
    r := chi.NewRouter() 
     
    // Public endpoint - remove sensitive fields 
    r.With(SanitizeResponse("password", "api_key", "metadata.internal")). 
        Get("/users/{id}", GetUser) 
     
    // Developer endpoint - only remove password 
    r.With(SanitizeResponse("password")). 
        Get("/dev/users/{id}", GetUser) 
     
    // Admin endpoint - no sanitization 
    r.Get("/admin/users/{id}", GetUser) 
}

Advanced Usage

The real power comes from dynamic field removal based on context. Here’s an example that adjusts sanitization based on the user’s role:

func DynamicSanitizer(next http.Handler) http.Handler { 
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { 
        // Get user role from context 
        role := GetUserRole(r.Context()) 
         
        // Define fields to remove based on role 
        fieldsToRemove := []string{} 
        switch role { 
        case "public": 
            fieldsToRemove = []string{"password", "api_key", "metadata"} 
        case "developer": 
            fieldsToRemove = []string{"password", "metadata.internal"} 
        case "admin": 
            // No fields removed 
        } 
         
        // Apply the sanitization 
        SanitizeResponse(fieldsToRemove...)(next).ServeHTTP(w, r) 
    }) 
}

Performance Considerations

While reflection-based operations are typically slower than hardcoded field access, the flexibility gained often outweighs the performance cost. For the best performance, it’s preferable to sanitize data as early as possible and not rely on an approach like this. When working in a distributed architecture, it can be hard to enforce standards across technical and organizational boundaries. Implementing data sanitization controls at the edge are a great way of enforcing standardization before the traffic leaves your environment.

Conclusion

Dynamic response sanitization is a powerful pattern that helps maintain clean, secure APIs. By leveraging Go’s reflection capabilities and middleware pattern, we can create flexible, maintainable solutions for controlling our API outputs.

The approach shown here can be extended to handle more complex scenarios like:

  • Field-level access control
  • Request-specific sanitization rules
  • Conditional field removal based on business logic

Remember, the goal is to find the right balance between flexibility and performance for your specific use case.

Share this post