WND has the details in "Feds pay $5.8 billion to jail criminal aliens":

...the number of criminal aliens in federal prisons increased from about 42,000 at the end of 2001 to about 49,000 at the end of last year... [that's a 16% increase]

The direct federal costs during the study's time frame were estimated to be $4.2 billion, with federal reimbursements to state and local governments totalling $1.6 billion through the State Criminal Alien Assistance Program, or SCAAP. [The last I heard, Bush wanted to eliminate those SCAAP payments. Even if the status has changed in the intervening year, it's still quite indicative.]

The majority of criminal aliens were identified as citizens of Mexico.

In addition, state prisons in fiscal 2003 housed about 74,000 criminal aliens. About 80 percent were in just five states -- Arizona, California, Florida, New York and Texas.

Also, about 147,000 criminal aliens were jailed in 698 local jurisdictions that received SCAAP reimbursement in 2003.

About 30 percent of those criminals were in five municipal and country jails -- Los Angeles County, California; New York City, New York; Orange County, California; Harris County, Texas; and Maricopa County, Arizona...

The GAO report is available in in this PDF file. UPDATE: Regarding SCAAP, see this:

The Senate Judiciary Committee, on Thursday, March 17, approved Sen. Dianne Feinstein's bill to increase funding for the State Criminal Alien Assistance Program (SCAAP). SCAAP partially reimburses state and local governments for the costs of incarcerating undocumented criminal immigrants. Senator Barbara Boxer is a co-sponsor of the bill along with 10 other Senators from both parties.

S.188 authorizes $750 million for fiscal year 2006; $850 million for FY 2007; and $950 million per fiscal year from FY 2008 through FY 2011. The Committee approved the bill by voice vote.

Last year's FY05 omnibus appropriations bill funded the program at $297 million, down from a high of $585 million several years ago. The Bush Administration has proposed zero-funding SCAAP in its last two budgets. California receives about 40 percent of SCAAP funding - almost $112 million of the $281.6 million total available in FY04.