data-expert

Data Expert data analysis initial exploration When reviewing or writing code, apply these guidelines: Begin analysis with data exploration and summary statistics. Implement data quality checks at the beginning of analysis. Handle missing data appropriately (imputation, removal, or flagging). data fetching rules for server components When reviewing or writing code, apply these guidelines: For data fetching in server components (in .tsx files): tsx async function getData() { const res = await fetch(' https://api.example.com/data ', { next: { revalidate: 3600 } }) if (!res.ok) throw new Error('Failed to fetch data') return res.json() } export default async function Page() { const data = await getData() // Render component using data } data pipeline management with dvc When reviewing or writing code, apply these guidelines: Data Pipeline Management: Employ scripts or tools like dvc to manage data preprocessing and ensure reproducibility. data synchronization rules When reviewing or writing code, apply these guidelines: Implement Data Synchronization: Create an efficient system for keeping the region grid data synchronized between the JavaScript UI and the WASM simulation. This might involve: a. Implementing periodic updates at set intervals. b. Creating an event-driven synchronization system that updates when changes occur. c. Optimizing large data transfers to maintain smooth performance, possibly using typed arrays or other efficient data structures. d. Implementing a queuing system for updates to prevent overwhelming the simulation with rapid changes. data tracking and charts rule When reviewing or writing code, apply these guidelines: There should be a chart page that tracks just about everything that can be tracked in the game. data validation with pydantic When reviewing or writing code, apply these guidelines: Data Validation: Use Pydantic models for rigorous Consolidated Skills This expert skill consolidates 1 individual skills: data-expert Iron Laws ALWAYS validate all external data at system boundaries using a schema validator (Zod, Pydantic, Joi) — never trust API responses, user input, or file contents without validation. NEVER load entire large datasets into memory — always stream, paginate, or batch-process data beyond a few thousand records to prevent memory spikes and timeouts. ALWAYS sanitize data before using it in downstream operations — HTML, SQL, and shell-injected content must be stripped or escaped before processing or storage. NEVER use string manipulation (regex, split, replace) as a primary parser for structured formats — use purpose-built parsers (JSON.parse, csv-parse, xml2js) for reliable type-safe results. ALWAYS make data transformation functions pure and idempotent — a function that mutates external state or produces different results for the same input cannot be safely tested or reused. Anti-Patterns Anti-Pattern Why It Fails Correct Approach Trusting API responses without validation API schemas change silently; unvalidated data causes downstream type errors Validate all responses with Zod/Pydantic schemas at the API boundary fs.readFileSync on large CSV/JSON files Loads entire file into memory; crashes on files > available RAM Use streaming parsers (csv-parse/stream, JSONStream) with backpressure Regex for parsing HTML or XML HTML/XML structure is not regular; regex breaks on nested tags and attributes Use proper DOM/XML parsers (cheerio, xml2js, DOMParser) Mutating input objects in transformations Caller still holds a reference to the mutated object; causes ghost bugs Return new objects ( { ...input, newField } ) instead of mutating Logging full request/response bodies with PII PII ends up in log aggregators readable by non-authorized users Redact PII fields before logging; log schemas and IDs only Memory Protocol (MANDATORY) Before starting: cat .claude/context/memory/learnings.md After completing: Record any new patterns or exceptions discovered. ASSUME INTERRUPTION: Your context may reset. If it's not in memory, it didn't happen.