In this paper, a new modified version of generalized likelihood ratio test (GLRT) is proposed which improves the detection probability of primary users in cognitive radio (CR) system under colored (or white) noise scenario. At first, in the case of noise variance uncertainty, the effectiveness of the conventional GLRT is compared to eigenvalue grads method (EGM) and energy detection (ED) method. Then, the challenge of spectrum sensing for GLRT algorithm under unknown-variance colored-noise is solved by optimization problem on CVX (MATLAB-based modeling system for convex optimization). It is approved that likelihood ratio (LLR) under colored noise can be converted to the white noise problem using the correction factor on the eigenvalues. This approach implemented on a multi-input multi-output (MIMO) system such that only one of K transmitting antennas which causes the largest eigenvalue is active. Simulation results in white as well as colored noises show higher performance for the modified GLRT respect to the conventional GLRT. Moreover, the required number of samples for the modified GLRT is sufficiently lower than that for the conventional GLRT. Finally, ergodic capacities for conventional and modified versions of GLRT are compared.