Not at all! Check out the kernel config section of the Handbook.
Note: We recommend that you make a dated snapshot of your new /kernel called /kernel.YYMMDD after you get it working properly. Also back up your new /modules directory to /modules.YYMMDD. That way, if you make a mistake the next time you play with your configuration you can boot the backup kernel instead of having to fall back to kernel.GENERIC. This is particularly important if you are now booting from a controller that GENERIC does not support.