When training or deploying large neural networks, it is essential to estimate how much memory a model will consume so you can choose appropriate hardware and avoid out-of-memory errors. The total memory needed depends not just on the model parameters themselves, but also on whether you need to store gradients and optimizer states during training.
Implement a function estimate_model_memory(num_params, bytes_per_param, optimizer, include_gradients, unit) that computes the total memory footprint of a model.
Rules:
num_params * bytes_per_param bytes.include_gradients is True, add another num_params * bytes_per_param bytes for the gradient buffer (gradients are stored at the same precision as parameters).'none': 0 extra bytes'sgd': num_params * 4 bytes (one momentum buffer)'adam': num_params * 8 bytes (first and second moment buffers)unit, where 'MB' uses a divisor of 1024**2 and 'GB' uses 1024**3.