Hebrew scholars are still undecided whether God sought to kill Moses or his son. Verse 24 does not clearly give the answer, though the pronoun "him" seems to have its antecedent in "Moses" in verse 21.
Moses knew that all Israelites were to be circumcised on the eighth day of their lives (Genesis 17:9-14). However, his wife Zipporah was not an Israelite but a Midianite, who obviously did not follow the Abrahamic covenant of circumcision (though the Midianites were descendants of Abraham through his second wife, Keturah; see Genesis 25:1-2). Apparently, Moses failed to circumcise his son because of her objection to the practice.
Exodus 4 shows that Moses, returning to Egypt to lead God's people Israel to the Promised Land, did not have his own house in subjection (see I Timothy 3:4). God could not allow His direct representative to lead the entire nation of Israel when he had not himself faithfully brought his family under the covenant, of which circumcision was the sign. That God acted to "cut him off" illustrates just how seriously God takes our commitment (or lack thereof) to His covenant (Hebrews 10:26-31).