Pandas Best Practices
Expert guidelines for Pandas development, focusing on data manipulation, analysis, and efficient DataFrame operations.
Code Style and Structure Write concise, technical responses with accurate Python examples Prioritize reproducibility in data analysis workflows Use functional programming; avoid unnecessary classes Prefer vectorized operations over explicit loops Use descriptive variable names reflecting data content Follow PEP 8 style guidelines DataFrame Creation and I/O Use pd.read_csv(), pd.read_excel(), pd.read_json() with appropriate parameters Specify dtype parameter to ensure correct data types on load Use parse_dates for automatic datetime parsing Set index_col when the data has a natural index column Use chunksize for reading large files incrementally Data Selection Use .loc[] for label-based indexing Use .iloc[] for integer position-based indexing Avoid chained indexing (e.g., df['col'][0]) - use .loc or .iloc instead Use boolean indexing for conditional selection: df[df['col'] > value] Use .query() method for complex filtering conditions Method Chaining Prefer method chaining for data transformations when possible Use .pipe() for applying custom functions in a chain Chain operations like .assign(), .query(), .groupby(), .agg() Keep chains readable by breaking across multiple lines Data Cleaning and Validation Missing Data Check for missing data with .isna() and .info() Handle missing data appropriately: .fillna(), .dropna(), or imputation Use pd.NA for nullable integer and boolean types Document decisions about missing data handling Data Quality Checks Implement data quality checks at the beginning of analysis Validate data types with .dtypes and convert as needed Check for duplicates with .duplicated() and handle appropriately Use .describe() for quick statistical overview Type Conversion Use .astype() for explicit type conversion Use pd.to_datetime() for date parsing Use pd.to_numeric() with errors='coerce' for safe numeric conversion Utilize categorical data types for low-cardinality string columns Grouping and Aggregation GroupBy Operations Use .groupby() for efficient aggregation operations Specify aggregation functions with .agg() for multiple operations Use named aggregation for clearer output column names Consider .transform() for broadcasting results back to original shape Pivot Tables and Reshaping Use .pivot_table() for multi-dimensional aggregation Use .melt() to convert wide to long format Use .pivot() to convert long to wide format Use .stack() and .unstack() for hierarchical index manipulation Performance Optimization Memory Efficiency Use categorical data types for low-cardinality strings Downcast numeric types when appropriate Use pd.eval() and .eval() for large expression evaluation Computation Speed Use vectorized operations instead of .apply() with row-wise functions Prefer built-in aggregation functions over custom ones Use .values or .to_numpy() for NumPy operations when faster Avoiding Common Pitfalls Avoid iterating with .iterrows() - use vectorized operations Don't modify DataFrames while iterating Be aware of SettingWithCopyWarning - use .copy() when needed Avoid growing DataFrames row by row - collect in list and create once Time Series Operations Use DatetimeIndex for time series data Leverage .resample() for time-based aggregation Use .shift() and .diff() for lag operations Use .rolling() and .expanding() for window calculations Merging and Joining Use .merge() for SQL-style joins Specify how parameter: 'inner', 'outer', 'left', 'right' Use validate parameter to check join cardinality Use .concat() for stacking DataFrames Key Conventions Import as import pandas as pd Use snake_case for column names when possible Document data sources and transformations Keep notebooks reproducible with clear cell execution order