How should we worship God? Does our worship need to be expressed in some physical way, too, or is "inner" worship all that's required? Coming at it from another angle, does physical expression mean anything in itself, without any "inner" component? Consider the example of someone who goes through the motions of worshiping some being other than God; does that mean anything, assuming he is not offering himself to the being?
Augustine considers a question like this in his City of God, X.19:
As to those who think that these visible sacrifices are suitably offered to other gods, but that invisible sacrifices, the graces of purity of mindand holiness of will, should be offered, as greater and better, to the invisible God, Himself greater and better than all others, they must be oblivious that these visible sacrifices are signs of the invisible, as the words we utter are the signs of things. And therefore, as in prayer or praise we direct intelligible words to Him to whom in our heart we offer the very feelings we are expressing, so we are to understand that in sacrifice we offer visible sacrifice only to Him to whom in our heart we ought to present ourselves an invisible sacrifice.
Augustine's point is that outward sacrifice is representative of the inward sacrifice. It is therefore still idolatry of a person were to go through the motions of sacrifice to a god, because that is the nature of sacrificial act: the outside is representative of the inside.
We might understand another corollary of this as follows. Many people want to worship God and live a spiritual life but don't want to go through anything involved with "organized religion." So they don't go to church, don't worship at church, don't belong to a group, etc. Though there are many things to say to such a person, Augustine provides us at least the following line of argument: your worship of God has to manifest itself in some way; and your actual behavior speaks to your inner convictions about God. So if you want nothing to do with God's people, don't seek the friendship and fellowship of persons who love him, don't make regular efforts to offer him any kind of worship or service, how spiritual can you really be?
Our spiritual life manifests itself in our physical life; that is our nature as embodied beings with a spiritual nature or soul. Our worship of God, then, in order to encompass all of ourselves, must have both this "inner" and "outer" aspect. To go through the motions of worshiping a false god is still idolatry, and to worship God inwardly only with no outward manifestations of worship is incomplete.