 The IRS may levy income tax on a portion of your Social Security benefits depending on your combined income from all sources. Your combined income consists of your adjusted gross income, your AGI, on your tax return, plus 50% of your Social Security benefits received during the tax year, plus any other non-taxable income you may have had.