OK. I have been very frustrated with the GMP for the past couple of hours. It seems that there is no default support for GMP library for Windows as they tend to target for Unix based systems. And also the tutorials that are found on the net are somewhat outdated, wasn't written keeping the noobs, like me, in mind or the newer tutorials are not on the first page of Google. -_-"
That said, I've finally been able to prepare it for windows. And I'm writing this just to help the others like me that may be searching solution to this problem and also for a future reference to myself for such other libraries as well. Let's do it ,shall we?

Getting the Library

First of all, we should get the library from gmplib.org. Current version is 5.0.4. The file may have been compressed twice. Extract it with a archive software like 7-zip. We have got the files that we need. We assume that we have extracted the folder here:

D:/C_Library/gmp-5.0.4

Now the next step should be to get MinGW.

Get MinGW

We have to get MSYS. Since current version of MSYS can be obtained along with the MinGW. We shall download MinGW from their SourceForge Page. Let's download it from there clicking the big green button. Now, when installing it we should be careful to check MSYS so that we can use it. Otherwise the whole point of downloading MinGW becomes worthless. For me I checked everything. Since 401MB of space for a software seems too little these days. -_-" Default installation directory for this is thought to be

C:\MinGW

Using MSYS to build GMP

Now, from the directory where we have installed MinGW, we now look for msys folder. in which there should be another folder only bearing a number (in current version it is 1.0). Inside that folder we should get msys.bat. That's all we need. Now we run it. And get a command prompt like window. This is our Unix like environment for windows. The next step here would be to go to the directory where the necessary files for GMP are. Remember, our directory was simply D:/C_Library/gmp-5.0.4. So, on the MSYS window we now write,

cd /d/C_Library/gmp-5.0.4

and press Enter.
Legend says Unix is case sensitive, and turns out that this is certainly true and also there should be no spaces while writing path, but after cd there is a space. So we have to write the directory in proper case. We, should understand that /d/ is equivalent to D:\ in windows, and we have to use forward slash. Now, then we write in MSYS,

./configure --prefix=/d/C_Library/GMP --enable-cxx

and press Enter.
Now, let me shortly explain what it means. First of all you should see that there is a INSTALL.autoconf file the gmp-5.0.4 directory. So, we assume ./configure simply invokes INSTALL.autoconf file that may have the necessary instructions. And --prefix=/d/C_Library/GMP may mean we want the configured thing to be copied into the directory followed after the = sign. and --enable-cxx means it will create a header file for C++ envrinment too...
!! But we are not done yet!
Now we write in MSYS,

make

and press Enter as usual.
We wait for a couple of minutes.... Wait... Wait....
This waiting is getting on our nerves, isn't it? In the mean time let's think over a quote, "Ignorance is bliss as long as it is not fatal."
So, what did you think about? Ok, first let me say what I think. In the field Computer Science ignorance is always bliss, no one will actually die if you are ignorant; of course unless it is a Techno-Thriller Novel/Movie/Comics/Anime bla bla bla. :)
Now, I think we are already done with the above process. The next step is to write

make install

And I give you 0.5 Seconds to say what to press....
we have to wait for several minutes here too... :(
Should I start rambling again? Nah! May be later. It's already finished I guess. :P
Now, that was easy, isn't? We now have several folders in the D:\C_Library\GMP folder. The include folder contains the header files gmp.h and gmpxx.h; and it is believed that for C we should use gmp.h and for C++ we should use gmpxx.h. And the lib folder contains the library files needed by the linker. We write

make check

to see it the header was build accordingly. After the test we should get there were no errors or something like that. ^_^

Using GMP with Code::Blocks

This is fairly easy. Open Code::Blocks. We can assume that we have Code::Blocks and it works with GCC compiler which is MinGW if we are on windows. But, since we now have a fresh MinGW installed in the C:\MinGW folder, we'll use this fresh software with Code::Blocks from now on. Now, we open Code::Blocks ; click on Settings and then Compiler and Debugger....
From the Selected Compiler drop-down menu we choose GNU GCC Compiler.
Then we choose Toolchain Executables and set Compilers Installation Directory to C:\MinGW. I will upload a picture of this. This will make the next steps more clear as well.

So, this should look like this.
Next We select the tab Search directories. It it in the Compiler sub-tab we add the directory of the header files of our GMP library. In our case

D:\C_Library\GMP\include

and in the linker sub-tab we add

D:\C_Library\GMP\lib

Next what we do is we go to Linker Settings tab, and and libgmpxx.a and libgmp.a in the Linker libraries box in the sequence I wrote. This sequencing is important.
And we are done!!! Now, we get out there and make some programs using this library.
An already built GMP can be downloaded from here GMP Library Manual: gmplib.org/manual/

References or the links that helped me in this case a lot

GMP Install Instruction for Windows Platform
Installing GMP on Windows
Using libraries with Code::Blocks

If there are any mistakes, please mention in the comments