Circumcision is a symbol marking the covenant God has with his people. But as God is interested not only in people's bodies, but also their minds, hearts, desires, souls, etc, so God's covenant concerns not only physical acts but also what is in our minds and hearts:
Deuteronomy 6:4-6 (Judaica)
Hear, O Israel: The Lord is our God; the Lord is one.
And you shall love the Lord, your God, with all your heart and with all your soul, and with all your means.
And these words, which I command you this day, shall be upon your heart.
Circumcision of the heart is a metaphor which appears in Jeremiah. As the prophet rebukes Israel for their many sins, and in particular for their faithlessness as they served idols rather than the true God, he calls them to repentance with the promise that they will escape God's punishment if they circumcise their hearts to the Lord. I think in context it symbolises not only general holy living, but also in particular the rejection of idolatry (שִׁקּוּץ).
Jeremiah 4:1-4 (Judaica)
If you return, O Israel, says the Lord, to Me, you shall return, and if you remove your detestable things (שִׁקּוּצֶ֛יךָ) from My Presence, you shall not wander.
And you will swear, "As the Lord lives," in truth and in justice and in righteousness, nations will bless themselves with him and boast about him.
For so said the Lord to the people of Judah and to Jerusalem: Plow for yourself a furrow, and do not sow upon thorns.
Circumcise yourselves to the Lord and remove the foreskins of your heart, O people of Judah and dwellers of Jerusalem, lest My anger go forth and burn with none to quench it because of the evil of your deeds.
Jeremiah 9:24-25 (Judaica)
Behold, days are coming, says the Lord, when I will punish every circumcised one with his uncircumcision.
Egypt, Judah, Edom, the children of Ammon, Moab, and all those cast off to the corners, who dwell in the desert, for all the nations are uncircumcised, and all the house of Israel are of uncircumcised hearts.